Enable Android Go testing on perf waterfall |
|||||||||||||
Issue descriptionWe will want to add two Android Go builder configs: android-go-perf android-go_webview-perf Since we have 77 android bots in total, I propose we put 35 bots on android-go-perf and 42 bots on android-go_webview-perf.
,
Jul 16
#1: we cannot divide things roughly by half, since the requirement is one host connects with 7 devices. So the number of mobile bots in a pool is almost always divisible by 7. So the best I come up is the 35/42 split, and since webview is always slower, I pick 42 for webview. Pinpoint already have 21 bots in https://chrome-internal.googlesource.com/infradata/config/+/master/configs/chrome-swarming/bots.cfg#564
,
Jul 16
Ok. Thanks for the details!
,
Jul 16
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/config/+/a1563ab8ec4244fbde1a1facdab86c356b9c2878 commit a1563ab8ec4244fbde1a1facdab86c356b9c2878 Author: Nghia Nguyen <nednguyen@google.com> Date: Mon Jul 16 14:23:25 2018
,
Jul 16
The Android Go fyi builder is in https://uberchromegw.corp.google.com/i/chromium.perf.fyi/builders/Android%20Go Proposed next plan: (1) Verify that the new Android Go bots are in the pool (2) Reshard the config. Plan to use 24 shards (70% of capacity). (3) Disable/fix some of the failing tests. (4) Verify that Android Go builder is green --> put it on the main waterfall Repeat (1) --> (4) for the Android Go webview.
,
Jul 16
Sorry, I'm not familiar with any Android Go bots in the Pinpoint pool. Your link points to Nexus 5X bots. Is there another bug in which they were added?
,
Jul 16
Ooops, my bad. The bots were "android-one" and I read 'android-go" somehow. So we would need to divide the bots as follow: pinpoint: 14 bots android-go: 28 bots android-go_webview-perf: 35 bots
,
Jul 16
Do android-go and android-go_webview-perf use the same OS build?
,
Jul 16
(For other devices, we've used google builds for Chrome and aosp builds for WebView)
,
Jul 17
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/config/+/44b715e9343863b38ddc509076cf32057ec549e8 commit 44b715e9343863b38ddc509076cf32057ec549e8 Author: Nghia Nguyen <nednguyen@google.com> Date: Tue Jul 17 08:59:05 2018
,
Jul 17
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/70ef7183f41258454d61368d7f850bb912679d6e commit 70ef7183f41258454d61368d7f850bb912679d6e Author: Ned Nguyen <nednguyen@google.com> Date: Tue Jul 17 09:03:19 2018 Use 19 shards for android-go configuration Bug:863767 Cq-Include-Trybots: master.tryserver.chromium.perf:obbs_fyi Change-Id: Ibe182c3bc7acaf9a0519da2eb365e443df460d68 TBR=perezju@chromium.org Change-Id: Ibe182c3bc7acaf9a0519da2eb365e443df460d68 Reviewed-on: https://chromium-review.googlesource.com/1138407 Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org> Commit-Queue: Ned Nguyen <nednguyen@google.com> Cr-Commit-Position: refs/heads/master@{#575586} [modify] https://crrev.com/70ef7183f41258454d61368d7f850bb912679d6e/testing/buildbot/chromium.perf.fyi.json [delete] https://crrev.com/416cf3c63fd346ce7c841e6b58c18e45fb9de177/tools/perf/core/shard_maps/android_go_14_shard_map.json [add] https://crrev.com/70ef7183f41258454d61368d7f850bb912679d6e/tools/perf/core/shard_maps/android_go_shard_map.json [add] https://crrev.com/70ef7183f41258454d61368d7f850bb912679d6e/tools/perf/core/shard_maps/timing_data/android_go_timing.json
,
Jul 17
John, the new bots build{224..232}-a7 (from https://bugs.chromium.org/p/chromium/issues/detail?id=846026#c9) can not be found in the perf pool. Can you check?
There are ".boto file is missing: /home/chrome-bot/.boto" error if I put in the host id, so not sure if it's related
https://chrome-swarming.appspot.com/bot?id=build224-a7&sort_stats=total%3Adesc
,
Jul 17
Assuming these are meant to be dockered+swarmed? I think they just need a docker config. Let me look into that now. Thanks.
,
Jul 17
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/afff8783623f6c752f2a41f86def00c316578666 commit afff8783623f6c752f2a41f86def00c316578666 Author: Ned Nguyen <nednguyen@google.com> Date: Tue Jul 17 20:47:19 2018 Update entry of 'Android Go' builder in perf_data_generator.py TBR=perezju@chromium.org NOTRY=true # test by PRESUBMIT Bug: 863767 Change-Id: I210c158140e015f0a15ba9d34945fcda8f394d43 Reviewed-on: https://chromium-review.googlesource.com/1140917 Reviewed-by: Ned Nguyen <nednguyen@google.com> Commit-Queue: Ned Nguyen <nednguyen@google.com> Cr-Commit-Position: refs/heads/master@{#575761} [modify] https://crrev.com/afff8783623f6c752f2a41f86def00c316578666/tools/perf/core/perf_data_generator.py
,
Jul 17
The following revision refers to this bug: https://chrome-internal.googlesource.com/infra/puppet/+/ef3ecb0b0108a5cdd0df771f8aa0e4685cd8ea71 commit ef3ecb0b0108a5cdd0df771f8aa0e4685cd8ea71 Author: John Weathersby <johnw@google.com> Date: Tue Jul 17 20:59:46 2018
,
Jul 17
Devices should be represented in swarming now.
,
Jul 17
The build is running now: https://uberchromegw.corp.google.com/i/chromium.perf.fyi/builders/Android%20Go/builds/1756 Thanks John a lot for your super quick turnaround!
,
Jul 17
,
Jul 17
The Pinpoint bots are in Swarming. android-go-perf and android-go_webview-perf have the same dimensions, so both configs have 14 devices available. CL to configure dashboard to send jobs to Pinpoint here: https://chromium-review.googlesource.com/c/catapult/+/1141170
,
Jul 18
,
Jul 18
The Android Go builder is very green in https://ci.chromium.org/buildbot/chromium.perf.fyi/Android%20Go/ If nothing changes, in the next few days, I will bring this to official perf waterfall (and announcing we will use this to replace the Android One builder)
,
Jul 18
The following revision refers to this bug: https://chromium.googlesource.com/catapult/+/941ac6857d71861d20588fc2e92c3be6be46d700 commit 941ac6857d71861d20588fc2e92c3be6be46d700 Author: Dave Tu <dtu@chromium.org> Date: Wed Jul 18 15:02:52 2018 [pinpoint] Add android-go-perf and android-go_webview-perf bots. Bug: chromium:863767 Change-Id: I9d258ecc35c28b79bf5c63aec7a3e04cfa9ffeff Reviewed-on: https://chromium-review.googlesource.com/1141170 Reviewed-by: Simon Hatch <simonhatch@chromium.org> Commit-Queue: Simon Hatch <simonhatch@chromium.org> [modify] https://crrev.com/941ac6857d71861d20588fc2e92c3be6be46d700/dashboard/dashboard/static/pinpoint.html
,
Jul 18
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/f739ba61eedb2cc10d2c489e0974c2cd9af6e818 commit f739ba61eedb2cc10d2c489e0974c2cd9af6e818 Author: nednguyen <nednguyen@google.com> Date: Wed Jul 18 16:07:24 2018 Regenerate timing data & reshard 'Android Go' builder Bug:863767 Cq-Include-Trybots: master.tryserver.chromium.perf:obbs_fyi Change-Id: Id2786b7907c9fd88141391a81c544747843bc017 NOTRY=true # test by PRESUBMIT Change-Id: Id2786b7907c9fd88141391a81c544747843bc017 Reviewed-on: https://chromium-review.googlesource.com/1141928 Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org> Commit-Queue: Ned Nguyen <nednguyen@google.com> Cr-Commit-Position: refs/heads/master@{#576081} [modify] https://crrev.com/f739ba61eedb2cc10d2c489e0974c2cd9af6e818/tools/perf/core/shard_maps/android_go_shard_map.json [modify] https://crrev.com/f739ba61eedb2cc10d2c489e0974c2cd9af6e818/tools/perf/core/shard_maps/timing_data/android_go_timing.json
,
Jul 18
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/ac69bd055c011310cf0ab3d4d84369131589f73b commit ac69bd055c011310cf0ab3d4d84369131589f73b Author: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Date: Wed Jul 18 19:05:33 2018 Roll src/third_party/catapult a824624071cc..941ac6857d71 (1 commits) https://chromium.googlesource.com/catapult.git/+log/a824624071cc..941ac6857d71 git log a824624071cc..941ac6857d71 --date=short --no-merges --format='%ad %ae %s' 2018-07-18 dtu@chromium.org [pinpoint] Add android-go-perf and android-go_webview-perf bots. Created with: gclient setdep -r src/third_party/catapult@941ac6857d71 The AutoRoll server is located here: https://catapult-roll.skia.org Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, please contact the current sheriff, who should be CC'd on the roll, and stop the roller if necessary. CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel BUG=chromium:863767 TBR=sullivan@chromium.org Change-Id: I2e8c37e442665f03348b83ce6d4813d3a73159d1 Reviewed-on: https://chromium-review.googlesource.com/1142005 Reviewed-by: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Commit-Queue: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Cr-Commit-Position: refs/heads/master@{#576152} [modify] https://crrev.com/ac69bd055c011310cf0ab3d4d84369131589f73b/DEPS
,
Jul 18
Perf dashboard and Pinpoint are fully configured. Will you be bringing up a separate FYI bot for WebView?
,
Jul 19
#25: I may just reuse the Android Go FYI builder (maybe rename it) after I move the "Android Go" to perf waterfall
,
Jul 19
Issue 779637 has been merged into this issue.
,
Jul 25
Given that issue 864999 (adding 2 builder slaves) gonna take sometimes to resolve. I will go ahead with putting Android Go config on perf waterfall by reusing the builder in fyi waterfall. If anyone objects this plan, please let me know.
,
Jul 26
,
Jul 26
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/master-manager/+/ebc4b3769736aa478583c85e6c139a2367c078b9 commit ebc4b3769736aa478583c85e6c139a2367c078b9 Author: nednguyen <nednguyen@google.com> Date: Thu Jul 26 10:13:43 2018
,
Jul 26
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/master-manager/+/07f36355bee428690942b0440246f212d188567c commit 07f36355bee428690942b0440246f212d188567c Author: nednguyen <nednguyen@google.com> Date: Thu Jul 26 10:14:53 2018
,
Jul 26
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/e400b39a3fbae0a43e39128f2c12dfdf8c9bd9b2 commit e400b39a3fbae0a43e39128f2c12dfdf8c9bd9b2 Author: Ned Nguyen <nednguyen@google.com> Date: Thu Jul 26 11:07:18 2018 Add android-go-perf builder to perf waterfall TBR=dpranke@chromium.org NOTRY=true # CQ flake Bug: 863767 Cq-Include-Trybots: master.tryserver.chromium.perf:obbs_fyi Change-Id: Iee28cb43d6d5aaf233c9f0a3b67562cba5326c47 Reviewed-on: https://chromium-review.googlesource.com/1150885 Commit-Queue: Ned Nguyen <nednguyen@google.com> Reviewed-by: David Tu <dtu@chromium.org> Reviewed-by: Stephen Martinis <martiniss@chromium.org> Cr-Commit-Position: refs/heads/master@{#578257} [modify] https://crrev.com/e400b39a3fbae0a43e39128f2c12dfdf8c9bd9b2/infra/config/global/luci-milo-dev.cfg [modify] https://crrev.com/e400b39a3fbae0a43e39128f2c12dfdf8c9bd9b2/infra/config/global/luci-milo.cfg [modify] https://crrev.com/e400b39a3fbae0a43e39128f2c12dfdf8c9bd9b2/testing/buildbot/chromium.perf.json [modify] https://crrev.com/e400b39a3fbae0a43e39128f2c12dfdf8c9bd9b2/tools/perf/core/perf_data_generator.py
,
Jul 26
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/config/+/57d952fcdb2760dfbde3d6de5b20d86f29c18a27 commit 57d952fcdb2760dfbde3d6de5b20d86f29c18a27 Author: Nghia Nguyen <nednguyen@google.com> Date: Thu Jul 26 13:14:34 2018
,
Jul 26
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/09628e9ab527745fb29b66fe8cafcdf4201328e2 commit 09628e9ab527745fb29b66fe8cafcdf4201328e2 Author: Ned Nguyen <nednguyen@google.com> Date: Thu Jul 26 13:16:02 2018 Migrate android-go-perf builder to use bots in chrome.tests.perf pool This depends on https://chrome-internal-review.googlesource.com/c/infradata/config/+/655985 NOTRY=true # CQ flake Bug: 863767 Cq-Include-Trybots: master.tryserver.chromium.perf:obbs_fyi Change-Id: I4dd1a3c71ffa03cb4eb4fccbe9660b5f972c2094 Reviewed-on: https://chromium-review.googlesource.com/1150898 Commit-Queue: Ned Nguyen <nednguyen@google.com> Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org> Cr-Commit-Position: refs/heads/master@{#578279} [modify] https://crrev.com/09628e9ab527745fb29b66fe8cafcdf4201328e2/testing/buildbot/chromium.perf.json [modify] https://crrev.com/09628e9ab527745fb29b66fe8cafcdf4201328e2/tools/perf/core/perf_data_generator.py
,
Jul 26
,
Jul 26
android-go builder is now up & running in https://ci.chromium.org/buildbot/chromium.perf/android-go-perf/ The next steps: 1) Migrate test paths ( issue 867908 ) 2) Verify that bisect works for android-go-perf ( issue 867910 ) 3) Enable graph monitoring (not sure if we need this?)
,
Aug 2
,
Aug 3
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/d757d32173f05ba7874c11f0c7fd47a1e9b76c8c commit d757d32173f05ba7874c11f0c7fd47a1e9b76c8c Author: Ned Nguyen <nednguyen@google.com> Date: Fri Aug 03 08:21:44 2018 Reshards android-go-perf builder to include few v8 benchmarks The extra v8 benchmarks we want to schedule on android-go-perf builder are: v8.browsing_mobile speedometer speedometer2 Steps that I did to generate this reshard: 1) Update the android-go-perf's story timing data by running: ./tools/perf/core/retrieve_story_timing.py -c 'android-go-perf' --output-file=tools/perf/core/shard_maps/timing_data/android_go_timing.json 2) Copy the story runtime of v8.browsing_mobile, speedometer, speedometer2 to android_go_timing.json 3) Generate the benchmark shardings with: ./tools/perf/generate_perf_sharding --timing-data tools/perf/core/shard_maps/timing_data/android_go_timing.json --num-shards=19 --output-file tools/perf/core/shard_maps/android_go_shard_map.json --benchmarks=memory.top_10_mobile,system_health.memory_mobile,system_health.common_mobile,power.typical_10_mobile,start_with_url.cold.startup_pages,start_with_url.warm.startup_pages,system_health.webview_startup,v8.browsing_mobile,speedometer,speedometer2 Cq-Include-Trybots: master.tryserver.chromium.perf:obbs_fyi Change-Id: I8a0696dc8b73092f4289b4abca60a40d92458e98 BUG: chromium:863767 Change-Id: I8a0696dc8b73092f4289b4abca60a40d92458e98 Reviewed-on: https://chromium-review.googlesource.com/1160766 Commit-Queue: Ned Nguyen <nednguyen@google.com> Reviewed-by: Ross McIlroy <rmcilroy@chromium.org> Cr-Commit-Position: refs/heads/master@{#580478} [modify] https://crrev.com/d757d32173f05ba7874c11f0c7fd47a1e9b76c8c/tools/perf/core/shard_maps/android_go_shard_map.json [modify] https://crrev.com/d757d32173f05ba7874c11f0c7fd47a1e9b76c8c/tools/perf/core/shard_maps/timing_data/android_go_timing.json
,
Aug 15
,
Nov 12
,
Jan 7
|
|||||||||||||
►
Sign in to add a comment |
|||||||||||||
Comment 1 by perezju@chromium.org
, Jul 16