WebRTC FYI bot blamelist is sometimes incorrect
Reported by
phoglund@webrtc.org,
Sep 16 2016
|
||||
Issue descriptionConsider https://build.chromium.org/p/chromium.webrtc.fyi/builders/Android%20Builder%20%28dbg%29/builds/16835. Its blamelist is empty. It has got_webrtc_revision_cp refs/heads/master@{#14241} Then consider https://build.chromium.org/p/chromium.webrtc.fyi/builders/Android%20Builder%20%28dbg%29/builds/16834, the build before. It has got_webrtc_revision_cp refs/heads/master@{#14240} Therefore refs/heads/master@{#14241} should have showed up in the blamelist for build 16835, but it doesn't. It shows up in https://build.chromium.org/p/chromium.webrtc.fyi/builders/Android%20Builder%20%28dbg%29/builds/16836, the build _after_, which has got_webrtc_revision_cp refs/heads/master@{#14242} This is confusing for sheriffs. Not sure if this is a bug in buildbot or in our change-source config.
,
Sep 28 2016
So, this tracks down to that it has historically been impossible to monitor both the src/third_party/libjingle/source/talk and src/third_party/webrtc folders in a good way that translates to the correct revisions being synced and reported to the blame list. All triggered bots have been syncing HEAD always, so there's always been a small window between the revision that triggered the build (revision property) and the point in time when the bot is actually syncing it's checkout (which results in the got_revision property). Now that we've gotten rid of src/third_party/libjingle/source/talk it is however possible to have these bots poll the https://chromium.googlesource.com/external/webrtc/trunk/webrtc/ repo instead and trigger on it.
,
Nov 7 2016
Me and Charu had a really hard time when we created https://bugs.chromium.org/p/chromium/issues/detail?id=662863; the blame lists are all over the place, and I'm not even sure if WebRTC revision gets reported correctly in the builds. Consider https://build.chromium.org/p/chromium.webrtc.fyi/builders/Mac%20Tester/builds/35450 https://build.chromium.org/p/chromium.webrtc.fyi/builders/Mac%20Tester/builds/35451 https://build.chromium.org/p/chromium.webrtc.fyi/builders/Mac%20Tester/builds/35452 https://build.chromium.org/p/chromium.webrtc.fyi/builders/Mac%20Tester/builds/35453 Only 5451 actually appears to move the WebRTC revision. It really looks like the regression is related to a change that got built in build 5451, but the regression is seen clearly first in build 5453. This could be because the graph metric is noisy but it looks really suspicious to me. Are builds really mapping correctly to the dashboard, and are WebRTC revisions really correctly reported in the build pages? Either way, as it stands now it's really hard to accurately triage WebRTC regressions on the FYI bots. We really need either 1) Accurate WebRTC blamelists 2) WebRTC blamelists in the perf dashboard, for the FYI bots. 2) Bisect bots that can bisect into WebRTC or all of the above.
,
Nov 30 2016
,
Nov 30 2016
,
Dec 1 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/depot_tools.git/+/5f5b83bbc24f1dbff484e44b130b182c5bdef4c1 commit 5f5b83bbc24f1dbff484e44b130b182c5bdef4c1 Author: Henrik Kjellander <kjellander@chromium.org> Date: Wed Nov 30 09:48:42 2016 Make Tester bots in chromium.webrtc.fyi sync the right revisions. BUG= chromium:669818 TBR=tandrii@chromium.org Change-Id: Icbefd12bd891c8fdb7b8278083f8b1accb53500c Reviewed-on: https://chromium-review.googlesource.com/414110 Commit-Queue: Henrik Kjellander <kjellander-cc@chromium.org> Reviewed-by: Michael Achenbach <machenbach@chromium.org> [modify] https://crrev.com/5f5b83bbc24f1dbff484e44b130b182c5bdef4c1/recipe_modules/gclient/config.py
,
Dec 1 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/depot_tools.git/+/b6849ac483138b2d479ebe4ed47d3e3890423db9 commit b6849ac483138b2d479ebe4ed47d3e3890423db9 Author: Henrik Kjellander <kjellander@chromium.org> Date: Thu Dec 01 08:26:08 2016 WebRTC: Make got_revision be a WebRTC revision for bots in chromium.webrtc.fyi This is needed in order to be able to use the set_component_rev feature in https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium_tests/api.py?rcl=0&l=169 which is needed for having the testers to sync the same WebRTC revision as the builders. This is a follow-up for https://chromium-review.googlesource.com/c/414110 BUG= chromium:669818 TBR=tandrii@chromium.org Change-Id: I6ca77e739513009e503a3b2a6fedacde8730e9a4 Reviewed-on: https://chromium-review.googlesource.com/415788 Reviewed-by: Henrik Kjellander <kjellander-cc@chromium.org> Reviewed-by: Michael Achenbach <machenbach@chromium.org> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org> Commit-Queue: Henrik Kjellander <kjellander-cc@chromium.org> [modify] https://crrev.com/b6849ac483138b2d479ebe4ed47d3e3890423db9/recipe_modules/gclient/config.py
,
Dec 1 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build.git/+/675b4d9f42e7522fafe88010febf05e42c30136f commit 675b4d9f42e7522fafe88010febf05e42c30136f Author: Henrik Kjellander <kjellander@chromium.org> Date: Wed Nov 30 09:53:06 2016 WebRTC: Poll external/webrtc/trunk/webrtc subtree mirror for chromium.webrtc.fyi bots. This should make it possible to solve the syncing problems, or at least make it easier to understand the revisions that trigger the build. It was not possible to do this earlier since we also had code in src/third_party/libjingle. BUG= 669818 TBR=ehmaldonado@chromium.org Change-Id: I0396e0b1a50ad1c7f7b4fbfaa5f8c12f91d78965 Reviewed-on: https://chromium-review.googlesource.com/414377 Reviewed-by: Michael Achenbach <machenbach@chromium.org> Commit-Queue: Henrik Kjellander <kjellander-cc@chromium.org> [modify] https://crrev.com/675b4d9f42e7522fafe88010febf05e42c30136f/masters/master.chromium.webrtc.fyi/master_source_cfg.py
,
Dec 1 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build.git/+/ccf38c94862cf45884c8e05ee041c2119f4e602f commit ccf38c94862cf45884c8e05ee041c2119f4e602f Author: Henrik Kjellander <kjellander@chromium.org> Date: Thu Dec 01 08:44:06 2016 WebRTC: Use set_component_rev for testers in chromium.webrtc.fyi This depends on https://chromium-review.googlesource.com/c/415788/ to set got_revision properly. It also likely needs https://chromium-review.googlesource.com/c/414377/ to make the revisions set be valid (since we still use a subtree Git mirror for webrtc/): BUG= 491520 , 669818 TBR=tandrii@chromium.org Change-Id: I407a10a8bcfb99c48452907dd8a710f43d25cc8d Reviewed-on: https://chromium-review.googlesource.com/415828 Reviewed-by: Michael Achenbach <machenbach@chromium.org> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org> Commit-Queue: Henrik Kjellander <kjellander-cc@chromium.org> [modify] https://crrev.com/ccf38c94862cf45884c8e05ee041c2119f4e602f/scripts/slave/recipe_modules/chromium_tests/chromium_webrtc_fyi.py [modify] https://crrev.com/ccf38c94862cf45884c8e05ee041c2119f4e602f/scripts/slave/recipes/chromium.expected/full_chromium_webrtc_fyi_Android_Tests__dbg___K_Nexus5_.json [modify] https://crrev.com/ccf38c94862cf45884c8e05ee041c2119f4e602f/scripts/slave/recipes/chromium.expected/full_chromium_webrtc_fyi_Android_Tests__dbg___L_Nexus5_.json [modify] https://crrev.com/ccf38c94862cf45884c8e05ee041c2119f4e602f/scripts/slave/recipes/chromium.expected/full_chromium_webrtc_fyi_Android_Tests__dbg___L_Nexus6_.json [modify] https://crrev.com/ccf38c94862cf45884c8e05ee041c2119f4e602f/scripts/slave/recipes/chromium.expected/full_chromium_webrtc_fyi_Android_Tests__dbg___L_Nexus7_2_.json [modify] https://crrev.com/ccf38c94862cf45884c8e05ee041c2119f4e602f/scripts/slave/recipes/chromium.expected/full_chromium_webrtc_fyi_Android_Tests__dbg___L_Nexus9_.json [modify] https://crrev.com/ccf38c94862cf45884c8e05ee041c2119f4e602f/scripts/slave/recipes/chromium.expected/full_chromium_webrtc_fyi_Linux_Tester.json [modify] https://crrev.com/ccf38c94862cf45884c8e05ee041c2119f4e602f/scripts/slave/recipes/chromium.expected/full_chromium_webrtc_fyi_Mac_Tester.json [modify] https://crrev.com/ccf38c94862cf45884c8e05ee041c2119f4e602f/scripts/slave/recipes/chromium.expected/full_chromium_webrtc_fyi_Win10_Tester.json [modify] https://crrev.com/ccf38c94862cf45884c8e05ee041c2119f4e602f/scripts/slave/recipes/chromium.expected/full_chromium_webrtc_fyi_Win7_Tester.json
,
Dec 2 2016
Closing this as most work is now done (verified in https://build.chromium.org/p/chromium.webrtc.fyi/builders/Linux%20Tester/builds/32833/steps/content_browsertests/logs/stdio). I filed bug 670856 for the last, not yet covered, case: that both builders and testers sync Chromium src@HEAD. It can lead to confusion if one is looking at got_cr_revision for the tester (parent_got_cr_revision will always be correct - the builder's synced revision of src/ but ideally also got_cr_revision should be correct). I will document the limitation in the perf sheriff instructions and point to that bug. |
||||
►
Sign in to add a comment |
||||
Comment 1 by phoglund@webrtc.org
, Sep 16 2016