New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 812419 link

Starred by 2 users

Issue metadata

Status: Verified
Owner:
Last visit > 30 days ago
Closed: Jul 27
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug
Build-Toolchain



Sign in to add a comment

afdo: implement a pipeline to pick up profiles from CWP for chrome

Project Member Reported by laszio@chromium.org, Feb 14 2018

Issue description

To pick up profiles from CWP, we need to

1. Clean up the gcc/gcov pipeline. The gcc/gcov pipeline is no longer active and the corresponding codes needs a cleanup.

2. Automatically pick up profiles for exp1.

3. Point the location for exp1 to where CWP profiles are populated.

4. Add the staleness check.
 
Project Member

Comment 1 by bugdroid1@chromium.org, Feb 17 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/f98e5fcb66b7c8298e6b191a3fc007f644c4a1ce

commit f98e5fcb66b7c8298e6b191a3fc007f644c4a1ce
Author: Ting-Yuan Huang <laszio@chromium.org>
Date: Sat Feb 17 05:41:38 2018

afdo: clean gcc/gcov codes up

gcc/gcov builders are no longer available.

BUG= chromium:812419 
TEST=tryjob master-chrome-pfq; Chrome ebuild is correctly updated.

Change-Id: Idf9b715d2e3c17b7617c9d1a49e3e39ea922439c
Reviewed-on: https://chromium-review.googlesource.com/919540
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Ting-Yuan Huang <laszio@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/f98e5fcb66b7c8298e6b191a3fc007f644c4a1ce/cbuildbot/afdo.py
[modify] https://crrev.com/f98e5fcb66b7c8298e6b191a3fc007f644c4a1ce/cbuildbot/stages/afdo_stages.py

Project Member

Comment 2 by bugdroid1@chromium.org, Feb 20 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/6e5f96a8b7b7571a2a2e1e7b21b86c3825b95190

commit 6e5f96a8b7b7571a2a2e1e7b21b86c3825b95190
Author: Ting-Yuan Huang <laszio@chromium.org>
Date: Tue Feb 20 11:53:39 2018

afdo: update profiles from CWP

TODO: staleness check, adjust path and filename

A pipeline frontend produces AutoFDO profiles from CWP daily.
This is the backend to pickup the profiles and update the ebuilds.

BUG= chromium:812419 
CQ-DEPEND=CL:919540
TEST=afdo_unittest
     cros tryjob: verified that the ebuild is modified as expected.

Change-Id: I06bbae16b256540673fe46785206ae9a85b4547b
Reviewed-on: https://chromium-review.googlesource.com/920624
Commit-Ready: Ting-Yuan Huang <laszio@chromium.org>
Tested-by: Ting-Yuan Huang <laszio@chromium.org>
Reviewed-by: Luis Lozano <llozano@chromium.org>

[modify] https://crrev.com/6e5f96a8b7b7571a2a2e1e7b21b86c3825b95190/cbuildbot/afdo.py
[modify] https://crrev.com/6e5f96a8b7b7571a2a2e1e7b21b86c3825b95190/cbuildbot/afdo_unittest.py
[modify] https://crrev.com/6e5f96a8b7b7571a2a2e1e7b21b86c3825b95190/cbuildbot/stages/afdo_stages.py

Project Member

Comment 3 by bugdroid1@chromium.org, Feb 21 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/e2ef78f2be977674faeb3d2f056a44efb8060725

commit e2ef78f2be977674faeb3d2f056a44efb8060725
Author: Ting-Yuan Huang <laszio@chromium.org>
Date: Wed Feb 21 04:32:01 2018

afdo: clean gcc/gcov codes up, part2.

test_stages.py was missed in the refactoring and resulted in an
attribute / function not found error.

TEST=tryjob chell-chrome-pfq-tryjob
BUG= chromium:812419 

Change-Id: Id77819a22d770cd70ab1ff79be88a0db5442783a
Reviewed-on: https://chromium-review.googlesource.com/926049
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Ting-Yuan Huang <laszio@chromium.org>
Reviewed-by: Luis Lozano <llozano@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/e2ef78f2be977674faeb3d2f056a44efb8060725/cbuildbot/stages/test_stages.py

Comment 4 by laszio@chromium.org, Feb 22 2018

Description: Show this description

Comment 5 by laszio@chromium.org, Feb 22 2018

Summary: afdo: implement a pipeline to pick up profiles from CWP for chrome (was: chromite: clean up legacy codes in afdo)
Project Member

Comment 6 by bugdroid1@chromium.org, Feb 23 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/b5c6f76c0d37c72c1f014c09a47881ef219e7750

commit b5c6f76c0d37c72c1f014c09a47881ef219e7750
Author: Ting-Yuan Huang <laszio@chromium.org>
Date: Fri Feb 23 11:36:22 2018

afdo: update profile location and naming convention for CWP

Changing the profiles from, for example,
gs://chromeos-localmirror/distfiles/afdo/experimental/cwp/chromeos-chrome-amd64-65.0.3299.0_rc-r1.afdo.bz2
to
gs://chromeos-prebuilt/afdo-job/cwp/chrome/R65-3325.65-1519323840.afdo.xz

TEST=afdo_unittest, master-chromium-pfq-tryjob
BUG= chromium:812419 
CQ-DEPEND=CL:932963

Change-Id: I81f85282379c2bf34240b0606aa4d765eb218aa7
Reviewed-on: https://chromium-review.googlesource.com/933087
Commit-Ready: Ting-Yuan Huang <laszio@chromium.org>
Tested-by: Ting-Yuan Huang <laszio@chromium.org>
Reviewed-by: Ting-Yuan Huang <laszio@chromium.org>

[modify] https://crrev.com/b5c6f76c0d37c72c1f014c09a47881ef219e7750/cbuildbot/afdo.py
[modify] https://crrev.com/b5c6f76c0d37c72c1f014c09a47881ef219e7750/cbuildbot/afdo_unittest.py

Project Member

Comment 7 by bugdroid1@chromium.org, Feb 23 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/588841f1355cdd924c6f7c88fde76a15b8521f29

commit 588841f1355cdd924c6f7c88fde76a15b8521f29
Author: Ting-Yuan Huang <laszio@chromium.org>
Date: Fri Feb 23 11:36:21 2018

chromeos-chrome: change location for chrome_afdo_exp1

Previously, it was a place where profiles are manually uploaded.
Now, the profiles are automatically generated.

BUG= chromium:812419 
TEST=falco-release-tryjob

Change-Id: I4f41094b7f7bd8996f8436532a574e6395400b05
Reviewed-on: https://chromium-review.googlesource.com/932963
Commit-Ready: Ting-Yuan Huang <laszio@chromium.org>
Tested-by: Ting-Yuan Huang <laszio@chromium.org>
Reviewed-by: Luis Lozano <llozano@chromium.org>

[modify] https://crrev.com/588841f1355cdd924c6f7c88fde76a15b8521f29/chromeos-base/chromeos-chrome/chromeos-chrome-9999.ebuild
[modify] https://crrev.com/588841f1355cdd924c6f7c88fde76a15b8521f29/chromeos-base/chromeos-chrome/Manifest

Project Member

Comment 8 by bugdroid1@chromium.org, Mar 1 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/38d67bb501bbfe64ff8f6f06584b59e7cfdf8a20

commit 38d67bb501bbfe64ff8f6f06584b59e7cfdf8a20
Author: Ting-Yuan Huang <laszio@chromium.org>
Date: Thu Mar 01 07:54:38 2018

afdo: update the heuristic of profile selection

When a branch just happens, the milestone of master increases by 1.
There will be no profile from that milestone until a dev release is
pushed for a short period of time. Therefore, a profile from previous
branches must be picked instead.

Originally, we search toward root in the branch tree for a profile. Now
we prefer to look at the previous milestone if there's no profile from
current milestone, because:

1. dev channel has few samples. The profile quality is much better from
   beta, which is always in a branch.

2. Master is actually closer to the branch tip than to the branch point,
   assuming that most of the changes on a branch are cherry-picked from
   master.

TEST=afdo_unittest, master-chromium-pfq-tryjob
BUG= chromium:812419 

Change-Id: I80e0c917923820bbceecc2f25569e34ce27a9437
Reviewed-on: https://chromium-review.googlesource.com/935462
Commit-Ready: Ting-Yuan Huang <laszio@chromium.org>
Tested-by: Ting-Yuan Huang <laszio@chromium.org>
Reviewed-by: Ting-Yuan Huang <laszio@chromium.org>

[modify] https://crrev.com/38d67bb501bbfe64ff8f6f06584b59e7cfdf8a20/cbuildbot/afdo.py
[modify] https://crrev.com/38d67bb501bbfe64ff8f6f06584b59e7cfdf8a20/cbuildbot/afdo_unittest.py

Status: Verified (was: Started)

Sign in to add a comment