Disabling a story/benchmark on Nexus 5 also disables it on Nexus 5X |
|||||
Issue descriptionDiscovered this after doing some investigation for panicker@. https://chromium-review.googlesource.com/c/chromium/src/+/1273967, which disabled a story on N5, also seems to have disabled it on N5X. After doing some digging, it looks like we do disabling by platform based on whether the model contains the given string. The model for Nexus 5 contains "Nexus_5". The model for Nexus 5X contains "Nexus_5X". Unsurprisingly, "Nexus_5X" contains the string "Nexus_5", so we disable all N5-disabled stories on 5X. To fix this, we need to do some more precise string matching than just "string contains". I'll look into what the actual models are to see if we can just match on the model names exactly. Beyond this, just fixing this bug will effectively mass-enable a bunch of stories, possibly leading to widespread failures. Our options are to either: 1) Warn the sheriff that these failures may happen and let them deal with it OR 2) Add a bunch of lines to expectations.config, probably linking to this bug, disabling every story on N5X that is currently disabled on N5, allowing us to deal with this at a later date.
,
Nov 15
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/1281acade40000
,
Nov 15
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/15e42ae3e40000
,
Nov 15
📍 Job complete. See results below. https://pinpoint-dot-chromeperf.appspot.com/job/1281acade40000
,
Nov 15
📍 Job complete. See results below. https://pinpoint-dot-chromeperf.appspot.com/job/15e42ae3e40000
,
Nov 16
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/1583cba7e40000
,
Nov 16
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/14c6e89fe40000
,
Nov 16
📍 Job complete. See results below. https://pinpoint-dot-chromeperf.appspot.com/job/14c6e89fe40000
,
Nov 16
📍 Job complete. See results below. https://pinpoint-dot-chromeperf.appspot.com/job/1583cba7e40000
,
Nov 16
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/176fc833e40000
,
Nov 16
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/100c95efe40000
,
Nov 16
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/17dcc8b7e40000
,
Nov 16
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/103cc8b7e40000
,
Nov 16
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/1297a7f7e40000
,
Nov 16
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/1082c41fe40000
,
Nov 16
😿 Pinpoint job stopped with an error. https://pinpoint-dot-chromeperf.appspot.com/job/17dcc8b7e40000 All of the runs failed. The most common error (1/20 runs) was: IOError: [Errno 2] No such file or directory: '/b/swarming/w/it83ld2W/tmp1uKm8Ntelemetry/histograms.json'
,
Nov 16
😿 Pinpoint job stopped with an error. https://pinpoint-dot-chromeperf.appspot.com/job/103cc8b7e40000 All of the runs failed. The most common error (1/20 runs) was: IOError: [Errno 2] No such file or directory: '/b/swarming/w/it_TLr_z/tmp0mDF9jtelemetry/histograms.json'
,
Nov 16
😿 Pinpoint job stopped with an error. https://pinpoint-dot-chromeperf.appspot.com/job/1297a7f7e40000 All of the runs failed. The most common error (1/20 runs) was: IOError: [Errno 2] No such file or directory: '/b/swarming/w/itNtNJMo/tmp0wxUnptelemetry/histograms.json'
,
Nov 16
😿 Pinpoint job stopped with an error. https://pinpoint-dot-chromeperf.appspot.com/job/176fc833e40000 All of the runs failed. The most common error (1/20 runs) was: IOError: [Errno 2] No such file or directory: '/b/s/w/itMsN7aL/tmpnv_TqVtelemetry/histograms.json'
,
Nov 16
😿 Pinpoint job stopped with an error. https://pinpoint-dot-chromeperf.appspot.com/job/100c95efe40000 All of the runs failed. The most common error (1/20 runs) was: IOError: [Errno 2] No such file or directory: '/b/s/w/itE06wXm/tmpWmCI_Vtelemetry/histograms.json'
,
Nov 16
😿 Pinpoint job stopped with an error. https://pinpoint-dot-chromeperf.appspot.com/job/1082c41fe40000 All of the runs failed. The most common error (1/20 runs) was: IOError: [Errno 2] No such file or directory: '/b/swarming/w/it_5Wwn1/tmpNXQTzMtelemetry/histograms.json'
,
Nov 19
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/14d32ecc140000
,
Nov 19
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/17461e47e40000
,
Nov 19
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/10030e5be40000
,
Nov 19
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/175f1affe40000
,
Nov 19
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/16c8f05be40000
,
Nov 19
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/12d32ecc140000
,
Nov 19
😿 Pinpoint job stopped with an error. https://pinpoint-dot-chromeperf.appspot.com/job/16c8f05be40000 All of the runs failed. The most common error (10/20 runs) was: HTTPException: HTTP status code 400: {"error": {"message": "CIPD package path is required. Use \".\" to install to run dir."}}
,
Nov 19
😿 Pinpoint job stopped with an error. https://pinpoint-dot-chromeperf.appspot.com/job/12d32ecc140000 All of the runs failed. The most common error (10/20 runs) was: HTTPException: HTTP status code 400: {"error": {"message": "CIPD package path is required. Use \".\" to install to run dir."}}
,
Nov 19
😿 Pinpoint job stopped with an error. https://pinpoint-dot-chromeperf.appspot.com/job/175f1affe40000 All of the runs failed. The most common error (10/20 runs) was: HTTPException: HTTP status code 400: {"error": {"message": "CIPD package path is required. Use \".\" to install to run dir."}}
,
Nov 19
😿 Pinpoint job stopped with an error. https://pinpoint-dot-chromeperf.appspot.com/job/10030e5be40000 All of the runs failed. The most common error (10/20 runs) was: HTTPException: HTTP status code 400: {"error": {"message": "CIPD package path is required. Use \".\" to install to run dir."}}
,
Nov 19
😿 Pinpoint job stopped with an error. https://pinpoint-dot-chromeperf.appspot.com/job/14d32ecc140000 All of the runs failed. The most common error (10/20 runs) was: HTTPException: HTTP status code 400: {"error": {"message": "CIPD package path is required. Use \".\" to install to run dir."}}
,
Nov 19
😿 Pinpoint job stopped with an error. https://pinpoint-dot-chromeperf.appspot.com/job/17461e47e40000 All of the runs failed. The most common error (10/20 runs) was: HTTPException: HTTP status code 400: {"error": {"message": "CIPD package path is required. Use \".\" to install to run dir."}}
,
Nov 20
dtu@, any idea what happened with that last round of bisects? I've never seen that "CIPD package path is required..." error.
,
Nov 27
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/165a9770140000
,
Nov 27
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/10745880140000
,
Nov 27
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/16bc458fe40000
,
Nov 27
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/13738f4be40000
,
Nov 27
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/15ccfc67e40000
,
Nov 27
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/1389fb38140000
,
Nov 27
Eh - I'm pretty sure this is just some transient error. Restarted all of the bisects.
,
Nov 27
😿 Pinpoint job stopped with an error. https://pinpoint-dot-chromeperf.appspot.com/job/1389fb38140000 All of the runs failed. The most common error (20/20 runs) was: KeyError: 'value'
,
Nov 27
📍 Job complete. See results below. https://pinpoint-dot-chromeperf.appspot.com/job/10745880140000
,
Nov 27
😿 Pinpoint job stopped with an error. https://pinpoint-dot-chromeperf.appspot.com/job/13738f4be40000 All of the runs failed. The most common error (20/20 runs) was: KeyError: 'value'
,
Nov 28
📍 Job complete. See results below. https://pinpoint-dot-chromeperf.appspot.com/job/165a9770140000
,
Nov 28
📍 Job complete. See results below. https://pinpoint-dot-chromeperf.appspot.com/job/15ccfc67e40000
,
Nov 28
📍 Job complete. See results below. https://pinpoint-dot-chromeperf.appspot.com/job/16bc458fe40000
,
Nov 29
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/69b8ea5c6ad8391439cc35b2a742dc98ccbc79a0 commit 69b8ea5c6ad8391439cc35b2a742dc98ccbc79a0 Author: Charlie Andrews <charliea@chromium.org> Date: Thu Nov 29 18:56:18 2018 Add Nexus 5X disables for all tests disabled on Nexus 5 This is necessary in order to fix http://crbug.com/894906 , where all tests disabled on Nexus 5 were accidentally disabled on Nexus 5X "Nexus 5X" contains the string "Nexus 5", and we were targeting the disables based on substring matching. I erred on the side of being redundant: if there was both a "Nexus 5" disable and an "Android" disable, I added a corresponding "Nexus 5X" disable. My idea here was that, if at some later point the "All" bug got fixed and the test got reenabled, I didn't want the person doing that to be confused why the story was failing on Nexus 5X still. Associating the logic bug with the disable explicitly reduces that confusion. Bug: 894906 Change-Id: I601304a979c7b7c6451d1571f8ffce61a60f4ecd Reviewed-on: https://chromium-review.googlesource.com/c/1355482 Reviewed-by: Ned Nguyen <nednguyen@google.com> Reviewed-by: Caleb Rouleau <crouleau@chromium.org> Commit-Queue: Charlie Andrews <charliea@chromium.org> Cr-Commit-Position: refs/heads/master@{#612294} [modify] https://crrev.com/69b8ea5c6ad8391439cc35b2a742dc98ccbc79a0/tools/perf/expectations.config
,
Nov 29
The following revision refers to this bug: https://chromium.googlesource.com/catapult/+/63d69d7792c1ef8f18d8b620ff025de767849514 commit 63d69d7792c1ef8f18d8b620ff025de767849514 Author: Charlie Andrews <charliea@chromium.org> Date: Thu Nov 29 20:08:39 2018 Fix bug where disabling stories on N5 also disabled them on N5X This bug occurred because we were doing substring matching in order to determine whether the current platform matched the disabled platform. "Nexus 5X" contains the string "Nexus 5", so disables targeted at Nexus 5 also affected Nexus 5Xs. Okay to submit now that the accompanying CL (http://crrev.com/c/1355482) is submitted that disables all stories currently disabled on N5 on N5X. Bug: chromium:894906 Change-Id: I19f1a87515edb8e091d63af7dabb0fe631b3c13a Reviewed-on: https://chromium-review.googlesource.com/c/1355419 Commit-Queue: Charlie Andrews <charliea@chromium.org> Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org> Reviewed-by: Caleb Rouleau <crouleau@chromium.org> [modify] https://crrev.com/63d69d7792c1ef8f18d8b620ff025de767849514/telemetry/telemetry/story/expectations_unittest.py [modify] https://crrev.com/63d69d7792c1ef8f18d8b620ff025de767849514/telemetry/telemetry/story/expectations.py
,
Nov 30
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/34cb4eeadbc4ed3e863bef0a6f5a9337143f38bd commit 34cb4eeadbc4ed3e863bef0a6f5a9337143f38bd Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com> Date: Fri Nov 30 01:46:17 2018 Roll src/third_party/catapult ee85992c5c61..63d69d7792c1 (1 commits) https://chromium.googlesource.com/catapult.git/+log/ee85992c5c61..63d69d7792c1 git log ee85992c5c61..63d69d7792c1 --date=short --no-merges --format='%ad %ae %s' 2018-11-29 charliea@chromium.org Fix bug where disabling stories on N5 also disabled them on N5X Created with: gclient setdep -r src/third_party/catapult@63d69d7792c1 The AutoRoll server is located here: https://autoroll.skia.org/r/catapult-autoroll Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, please contact the current sheriff, who should be CC'd on the roll, and stop the roller if necessary. CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel BUG= chromium:894906 TBR=sullivan@chromium.org Change-Id: I82daa446bb0740864fe1e0e433c9c72e680ddb28 Reviewed-on: https://chromium-review.googlesource.com/c/1355560 Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com> Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com> Cr-Commit-Position: refs/heads/master@{#612489} [modify] https://crrev.com/34cb4eeadbc4ed3e863bef0a6f5a9337143f38bd/DEPS
,
Dec 5
,
Jan 16
(6 days ago)
,
Jan 16
(6 days ago)
|
|||||
►
Sign in to add a comment |
|||||
Comment 1 by charliea@chromium.org
, Oct 12