Investigate AFDO instability on Chrome for Android/Linux |
|
Issue descriptionI've gotten a fair number of bug reports about AFDO rolls causing performance instability on Chrome for Android and Linux. Many of these reports are for blink benchmarks, which are very micro-benchmark-y (especially from the standpoint of AFDO). Others are for profile flakes, and some others are for higher-level benchmarks (speedometer, rasterize_and_record, ...). The goal is to look into these flakes/etc. to see what we can do to stabilize these benchmarks. Some occasional noise is, naturally, expected, but 20% swings in arbitrary benchmarks every week is pretty bad. We'll also randomly see swings in binary size of up to 300KB for Chrome/Linux, which, again, is suboptimal. In parallel, CrOS is trying to flip to using CWP profiles to optimize Chrome. The hope is that this will cause less variance in profile runs, but we'll see. For the moment, we generate profiles from benchmarks that we control, so "please add blink.foo to the profile collection" is also an option :)
,
Jun 7 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/f48146699f1f7ba335710e4a569e59fe0d4f146f commit f48146699f1f7ba335710e4a569e59fe0d4f146f Author: George Burgess IV <gbiv@chromium.org> Date: Thu Jun 07 20:11:34 2018 Stop generating AFDO profiles for non-r1 Chrome versions Roughly, our AFDO file generation logic looks like: - If no chromeos-chrome-amd64-${chrome_version}.perf.data.bz2 exists, generate it. - From chromeos-chrome-amd64-${chrome_version}.perf.data.bz2 and chromeos-chrome-amd64-${chrome_version}_rc-r${chrome_rev}.debug.bz2, create a new AFDO profile named chromeos-chrome-amd64-${chrome_version}_rc-r${chrome_rev}.afdo.bz2. This works perfectly if the perf.data was created on this run. However, if the perf.data already existed, we'll use it as if it was generated from sampling the Chrome at ${chrome_rev}, which isn't what happened, since it's from an earlier revision. Depending on what's changed in Chrome since the original perf.data was collected, this can result in a very messed up profile (e.g. 3437.0-r2). Since we're planning on moving to CWP profiles in the near future, and since -- modulo bugs -- we get a new -r1 daily, it doesn't seem bad to just stop generating AFDO profiles for non-r1 Chromes. The alternative appears to be versioning perf profiles, fixing tools to deal with the versioned profiles, and living with the some-versioned-some-not profiles. BUG=chromium:849881 TEST=./afdo_stages_unittest Change-Id: Ie6ac657760f8925e2770f56face409428277f70c Reviewed-on: https://chromium-review.googlesource.com/1087323 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: George Burgess <gbiv@chromium.org> Reviewed-by: Luis Lozano <llozano@chromium.org> [modify] https://crrev.com/f48146699f1f7ba335710e4a569e59fe0d4f146f/cbuildbot/stages/afdo_stages_unittest.py [modify] https://crrev.com/f48146699f1f7ba335710e4a569e59fe0d4f146f/cbuildbot/stages/afdo_stages.py
,
Jun 8 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/238e290d15417cf449f8463802aa5523f7837d89 commit 238e290d15417cf449f8463802aa5523f7837d89 Author: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Date: Fri Jun 08 00:18:14 2018 Roll src/third_party/chromite 840fa58..531d6f8 (2 commits) https://chromium.googlesource.com/chromiumos/chromite.git/+log/840fa58..531d6f8 git log 840fa58..531d6f8 --date=short --no-merges --format='%ad %ae %s' 2018-06-07 dgarrett@google.com cros tryjob: Use --git-cache-dir for local tryjobs. 2018-06-07 gbiv@chromium.org Stop generating AFDO profiles for non-r1 Chrome versions Created with: gclient setdep -r src/third_party/chromite@531d6f8 The AutoRoll server is located here: https://chromite-chromium-roll.skia.org Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, please contact the current sheriff, who should be CC'd on the roll, and stop the roller if necessary. BUG=chromium:None,chromium:849881 TBR=chrome-os-gardeners@chromium.org Change-Id: I7b435ec35584eee7204c82fb4233e9557a09536d Reviewed-on: https://chromium-review.googlesource.com/1091811 Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Cr-Commit-Position: refs/heads/master@{#565479} [modify] https://crrev.com/238e290d15417cf449f8463802aa5523f7837d89/DEPS
,
Jun 26 2018
Issue 856836 noted a 0.4% size increase on Linux, so the 69.0.3472.0_rc-r1 -> 69.0.3473.0_rc-r1 might be good to look at. No word on whether Android is unhappy with this change.
,
Jun 27 2018
Issue 856834 (Linux) has a few 11%ish swings that are apparently caused by new AFDO profiles, as well.
,
Jun 28 2018
As an update: I'm mostly holding off on this because we plan to switch the source of our profiles from benchmark-based to field-based within the next quarter. The latter is a pretty big change and will probably change the nature of the instability that we're seeing, so it seems a waste to spend substantial amounts of time on this now just to start over in a few weeks...
,
Aug 8
Note to self: seeing some recent flakiness in rasterize_and_record in issue 872165
,
Sep 20
Note to self: the roll to 71.0.3555.0_rc-r1 caused much anger and sadness (read: five perf bugs filed so far. Presumably more to come). All has recovered so far with the 3556 roll. |
|
►
Sign in to add a comment |
|
Comment 1 by g...@chromium.org
, Jun 5 2018