Migrate android-go-perf to LUCI |
|||||||
Issue description
,
Aug 28
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/d12003b11d80b400e9b82f1abee8682e47b51dd0 commit d12003b11d80b400e9b82f1abee8682e47b51dd0 Author: nednguyen <nednguyen@google.com> Date: Tue Aug 28 15:44:01 2018 Convert android-go-perf to use android-builder-perf as parent builder Bug:878389 Change-Id: I617166271bebd88d4cead26cec18ed1995f8bf73 Reviewed-on: https://chromium-review.googlesource.com/1194302 Reviewed-by: John Budorick <jbudorick@chromium.org> Commit-Queue: Ned Nguyen <nednguyen@google.com> [modify] https://crrev.com/d12003b11d80b400e9b82f1abee8682e47b51dd0/scripts/slave/recipe_modules/chromium_tests/chromium_perf.py
,
Aug 28
+Dtu/Simon: let me know if this is also a problem. I can revert the CL in #2 to switch android-go-perf back to buildbot
,
Aug 28
The upload step is failing due to the lack of service account file. It's baffling me since the android-go_webview-perf builder don't have this problem. John: any insights why /creds/service_accounts/service-account-chromium-perf-histograms.json exist on https://chrome-swarming.appspot.com/bot?id=swarm93-c7&sort_stats=total%3Adesc (android-go_webview-perf builder) but not the swarming bot of android-go-perf? Log:https://logs.chromium.org/logs/chrome/buildbucket/cr-buildbucket.appspot.com/8936935047833572144/+/steps/performance_test_suite_on_Android_device_gobo/0/logs/Merge_script_log/0 Unexpected error while uploading histogram data: Traceback (most recent call last): File "/b/swarming/w/ir/cache/builder/src/tools/perf/core/results_dashboard.py", line 116, in SendResults service_account_file, token_generator_callback) File "/b/swarming/w/ir/cache/builder/src/tools/perf/core/results_dashboard.py", line 473, in _SendHistogramJson service_account_file, DEFAULT_TOKEN_TIMEOUT_IN_MINUTES) File "/b/swarming/w/ir/cache/builder/src/tools/perf/core/results_dashboard.py", line 63, in LuciAuthTokenGeneratorCallback (p.stdout.read(), p.stderr.read())) RuntimeError: Error generating authentication token. Stdout: Stder:[E2018-08-28T10:32:05.906891-07:00 1920 0 service.go:91] Failed to load private key JSON - open /creds/service_accounts/service-account-chromium-perf-histograms.json: no such file or directory [W2018-08-28T10:32:05.906939-07:00 1920 0 auth.go:1032] Failed to read auth token from cache: invalid service account credentials [E2018-08-28T10:32:05.906971-07:00 1920 0 service.go:91] Failed to load private key JSON - open /creds/service_accounts/service-account-chromium-perf-histograms.json: no such file or directory invalid service account credentials
,
Aug 28
,
Aug 28
,
Aug 28
+vadim: who may be able to help with #4
,
Aug 28
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/fa1764d294743c0444ec8fe8a36014861de97609 commit fa1764d294743c0444ec8fe8a36014861de97609 Author: Ned Nguyen <nednguyen@google.com> Date: Tue Aug 28 19:34:28 2018 Revert "Convert android-go-perf to use android-builder-perf as parent builder" This reverts commit d12003b11d80b400e9b82f1abee8682e47b51dd0. Reason for revert: failing the perf dashboard upload step due to /creds/service_accounts/service-account-chromium-perf-histograms.json doesn't exist on the bot Original change's description: > Convert android-go-perf to use android-builder-perf as parent builder > > Bug:878389 > Change-Id: I617166271bebd88d4cead26cec18ed1995f8bf73 > Reviewed-on: https://chromium-review.googlesource.com/1194302 > Reviewed-by: John Budorick <jbudorick@chromium.org> > Commit-Queue: Ned Nguyen <nednguyen@google.com> TBR=nednguyen@google.com,jbudorick@chromium.org Change-Id: I39fcfaa1e4c9d798c19ec75e4fcf8b2139fb2689 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: 878389 Reviewed-on: https://chromium-review.googlesource.com/1194313 Commit-Queue: Ned Nguyen <nednguyen@google.com> Reviewed-by: Ned Nguyen <nednguyen@google.com> [modify] https://crrev.com/fa1764d294743c0444ec8fe8a36014861de97609/scripts/slave/recipe_modules/chromium_tests/chromium_perf.py
,
Aug 28
,
Aug 28
Chatted with Andrii offline, he recommend not relying on a fixed service account file deployed on bot. Instead we should be using the swarming task service account, which is chrome-ci-builder@chops-service-accounts.iam.gserviceaccount.com This would mean I need to update the results_dasbhoard.py to know not passing -service-account-json flag when service_account_file is None. I will try experimenting with switching to use chrome-ci-builder@chops-service-accounts.iam.gserviceaccount.com on android-go_webview-perf builder first, since that builder is only FYI and we don't have too worry much about data loss there.
,
Aug 28
The following revision refers to this bug: https://chromium.googlesource.com/catapult/+/85388f16f3fa3b42cb435f9c8601448f5ce035ea commit 85388f16f3fa3b42cb435f9c8601448f5ce035ea Author: Simon <simonhatch@chromium.org> Date: Tue Aug 28 20:47:41 2018 Dashboard - Whitelist luci oauth client id Bug: chromium:878389 Change-Id: I5d69dabbb7c200ea16d1e09fd64e3a30c33e3313 Reviewed-on: https://chromium-review.googlesource.com/1194708 Reviewed-by: Ned Nguyen <nednguyen@google.com> Commit-Queue: Simon Hatch <simonhatch@chromium.org> [modify] https://crrev.com/85388f16f3fa3b42cb435f9c8601448f5ce035ea/dashboard/dashboard/api/api_auth.py
,
Aug 29
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/20a52dc9fac2202a90f3bc58061119f49c58c361 commit 20a52dc9fac2202a90f3bc58061119f49c58c361 Author: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Date: Wed Aug 29 00:11:49 2018 Roll src/third_party/catapult 18259e73a8b7..85388f16f3fa (2 commits) https://chromium.googlesource.com/catapult.git/+log/18259e73a8b7..85388f16f3fa git log 18259e73a8b7..85388f16f3fa --date=short --no-merges --format='%ad %ae %s' 2018-08-28 simonhatch@chromium.org Dashboard - Whitelist luci oauth client id 2018-08-28 anthonyalridge@google.com Provides callback to link data points in dotplot to their traces. Created with: gclient setdep -r src/third_party/catapult@85388f16f3fa The AutoRoll server is located here: https://autoroll.skia.org/r/catapult-autoroll Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, please contact the current sheriff, who should be CC'd on the roll, and stop the roller if necessary. CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel BUG= chromium:878389 ,chromium:866423 TBR=sullivan@chromium.org Change-Id: I066ecf4c6a0639b87b2f448c4552e857275eba01 Reviewed-on: https://chromium-review.googlesource.com/1194355 Reviewed-by: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Commit-Queue: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Cr-Commit-Position: refs/heads/master@{#586950} [modify] https://crrev.com/20a52dc9fac2202a90f3bc58061119f49c58c361/DEPS
,
Aug 29
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/cb890354f5ff1bada814e1d0fa369e8698179bb9 commit cb890354f5ff1bada814e1d0fa369e8698179bb9 Author: Ned Nguyen <nednguyen@google.com> Date: Wed Aug 29 04:30:24 2018 Set service_account_file of results dashboard upload to None for perf LUCI builder Bug: 878389 , 860677 Cq-Include-Trybots: master.tryserver.chromium.perf:obbs_fyi Change-Id: I5cb85e8a22b1fe3c9bf64a993f3aa3f765a0734e Reviewed-on: https://chromium-review.googlesource.com/1194825 Commit-Queue: Ned Nguyen <nednguyen@google.com> Reviewed-by: Simon Hatch <simonhatch@chromium.org> Cr-Commit-Position: refs/heads/master@{#587012} [modify] https://crrev.com/cb890354f5ff1bada814e1d0fa369e8698179bb9/tools/perf/core/results_dashboard.py [modify] https://crrev.com/cb890354f5ff1bada814e1d0fa369e8698179bb9/tools/perf/core/upload_results_to_perf_dashboard.py [modify] https://crrev.com/cb890354f5ff1bada814e1d0fa369e8698179bb9/tools/perf/process_perf_results.py [modify] https://crrev.com/cb890354f5ff1bada814e1d0fa369e8698179bb9/tools/perf/process_perf_results_unittest.py
,
Aug 29
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/e9fad0c9046b0fbc46b27cdde3449c5094a971d0 commit e9fad0c9046b0fbc46b27cdde3449c5094a971d0 Author: Ned Nguyen <nednguyen@google.com> Date: Wed Aug 29 09:40:17 2018 Only parse 'buildbucket' property if it's string Interestingly, this property is an actual dict in LUCI, and a string JSON in buildbot. LUCI: https://logs.chromium.org/logs/chrome/buildbucket/cr-buildbucket.appspot.com/8936878256900261392/+/steps/performance_webview_test_suite_on_Android_device_gobo/0/stdout Buildbot: https://logs.chromium.org/v/?s=chrome%2Fbb%2Ftryserver.chromium.perf%2Fobbs_fyi%2F485%2F%2B%2Frecipes%2Fsteps%2Ftelemetry_perf_tests_without_chrome_on__none__GPU_on_Linux__with_patch_%2F0%2Fstdout (search for "buildbucket" in both log) Bug:878389 Change-Id: I2aafae6cc51120c83d5a522d501dc726889218ec TBR=simonhatch@chromium.org, jbudorick@chromium.org Change-Id: I2aafae6cc51120c83d5a522d501dc726889218ec Reviewed-on: https://chromium-review.googlesource.com/1195156 Reviewed-by: Ned Nguyen <nednguyen@google.com> Commit-Queue: Ned Nguyen <nednguyen@google.com> Cr-Commit-Position: refs/heads/master@{#587052} [modify] https://crrev.com/e9fad0c9046b0fbc46b27cdde3449c5094a971d0/tools/perf/process_perf_results.py
,
Aug 29
We successfully converted android-go_webview-perf to use LUCI swarming task account: https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/android-go_webview-perf/617 Log: service_account_file is not set. Use LUCI swarming task service account (https://logs.chromium.org/logs/chrome/buildbucket/cr-buildbucket.appspot.com/8936864807863855264/+/steps/performance_webview_test_suite_on_Android_device_gobo/0/logs/Merge_script_log/0) So I am proceeding to convert android-go-perf to LUCI again now
,
Aug 29
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/7a6c1bda2234e23f967d6590d2ed2c366f390591 commit 7a6c1bda2234e23f967d6590d2ed2c366f390591 Author: nednguyen <nednguyen@google.com> Date: Wed Aug 29 11:33:41 2018 Reland "Convert android-go-perf to use android-builder-perf as parent builder" This is a reland of d12003b11d80b400e9b82f1abee8682e47b51dd0 Service account should be ready for uploading to perf dashboard on LUCI (https://bugs.chromium.org/p/chromium/issues/detail?id=878389#c15) Original change's description: > Convert android-go-perf to use android-builder-perf as parent builder > > Bug:878389 > Change-Id: I617166271bebd88d4cead26cec18ed1995f8bf73 > Reviewed-on: https://chromium-review.googlesource.com/1194302 > Reviewed-by: John Budorick <jbudorick@chromium.org> > Commit-Queue: Ned Nguyen <nednguyen@google.com> Bug: 878389 Change-Id: Icd22b5492abf2e4f92dd3db76190b86560d935f3 TBR=jbudorick@chromium.org, dtu@chromium.org Change-Id: Icd22b5492abf2e4f92dd3db76190b86560d935f3 Reviewed-on: https://chromium-review.googlesource.com/1195081 Reviewed-by: Ned Nguyen <nednguyen@google.com> Commit-Queue: Ned Nguyen <nednguyen@google.com> [modify] https://crrev.com/7a6c1bda2234e23f967d6590d2ed2c366f390591/scripts/slave/recipe_modules/chromium_tests/chromium_perf.py
,
Aug 29
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/a80f10eebfe907a95e1425f64b05e932d1009dc5 commit a80f10eebfe907a95e1425f64b05e932d1009dc5 Author: nednguyen <nednguyen@google.com> Date: Wed Aug 29 17:31:33 2018 Remove android-go-perf slave ('slave340-c1') This builder has been converted to LUCI: https://ci.chromium.org/p/chromium/builders/luci.chrome.ci/android-go-perf Bug:878389, 878778 Change-Id: I1fec5a4e8ab3c450b4dffc1c335f6ab981ea8470 Reviewed-on: https://chromium-review.googlesource.com/1195523 Reviewed-by: Dave Tu <dtu@chromium.org> Reviewed-by: John Budorick <jbudorick@chromium.org> Commit-Queue: Ned Nguyen <nednguyen@google.com> [modify] https://crrev.com/a80f10eebfe907a95e1425f64b05e932d1009dc5/masters/master.chromium.perf/slaves.cfg [modify] https://crrev.com/a80f10eebfe907a95e1425f64b05e932d1009dc5/masters/master.chromium.perf/master.cfg
,
Aug 29
,
Aug 29
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/cc8fa1660d52dfe963ed8526026b898195399687 commit cc8fa1660d52dfe963ed8526026b898195399687 Author: nednguyen <nednguyen@google.com> Date: Wed Aug 29 17:52:05 2018 Update milo entry for android-go-perf Bug: 878389 Change-Id: I5c226325dbbfc2f0f044986e03943e08ff4330e3 Reviewed-on: https://chromium-review.googlesource.com/1195672 Reviewed-by: John Budorick <jbudorick@chromium.org> Commit-Queue: Ned Nguyen <nednguyen@google.com> Cr-Commit-Position: refs/heads/master@{#587185} [modify] https://crrev.com/cc8fa1660d52dfe963ed8526026b898195399687/infra/config/global/luci-milo.cfg
,
Aug 29
I think this is a noop for Pinpoint, as long as the recipe and perf-id didn't change. |
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by bugdroid1@chromium.org
, Aug 28