Fix tests in chrome.perf.fyi android bots |
|||||||
Issue description
,
Jul 19
,
Aug 17
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/9cfff7e247fc4eb1d4ffbef2eb8093a496812baa commit 9cfff7e247fc4eb1d4ffbef2eb8093a496812baa Author: Juan Antonio Navarro Perez <perezju@chromium.org> Date: Fri Aug 17 12:16:34 2018 [tools/perf] Disable consistently failing stories on Pixel 2 Disabling the following storie currently showing consistent failures on android-pixel2-perf bot. TBR=nednguyen@google.com NOTRY=true Bug: 865400 Change-Id: I0df6999497e00abfd71bbd00e98a1e274b347aad Reviewed-on: https://chromium-review.googlesource.com/1179751 Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org> Commit-Queue: Juan Antonio Navarro Pérez <perezju@chromium.org> Cr-Commit-Position: refs/heads/master@{#584029} [modify] https://crrev.com/9cfff7e247fc4eb1d4ffbef2eb8093a496812baa/tools/perf/expectations.config
,
Aug 24
Failure logs 1: https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/fe9c2df0-a1b5-11e8-8e17-0242ac110008 Screenshot: https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/ff67408a-a1b5-11e8-8e17-0242ac110008 Stack trace: (ERROR) 2018-08-16 19:59:23,362 story_runner.ProcessError:100 Handleable error. Will try to restart shared state Traceback (most recent call last): File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 125, in _RunStoryAndProcessErrorIfNeeded state.RunStory(results) File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function return func(*args, **kwargs) File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 310, in RunStory self._current_page.Run(self) File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/page/__init__.py", line 99, in Run self.RunPageInteractions(action_runner) File "/b/swarming/w/ir/tools/perf/page_sets/system_health/system_health_story.py", line 114, in RunPageInteractions self._DidLoadDocument(action_runner) File "/b/swarming/w/ir/tools/perf/page_sets/system_health/background_stories.py", line 57, in _DidLoadDocument action_runner.TapElement(selector='.growl-dismiss') File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function return func(*args, **kwargs) File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py", line 305, in TapElement selector=selector, text=text, element_function=element_function)) File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function return func(*args, **kwargs) File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py", line 62, in _RunAction action.RunAction(self._tab) File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function return func(*args, **kwargs) File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/actions/tap.py", line 77, in RunAction element_function=self.element_function) File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/actions/page_action.py", line 130, in EvaluateCallbackWithElement return tab.EvaluateJavaScript(code) File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function return func(*args, **kwargs) File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 215, in EvaluateJavaScript return self._inspector_backend.EvaluateJavaScript(*args, **kwargs) File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function return func(*args, **kwargs) File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 252, in EvaluateJavaScript return self._EvaluateJavaScript(expression, context_id, timeout) File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function return func(*args, **kwargs) File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 38, in Inner return func(inspector_backend, *args, **kwargs) File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 528, in _EvaluateJavaScript return self._runtime.Evaluate(expression, context_id, timeout) File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_runtime.py", line 53, in Evaluate description=details.get('exception', {}).get('description')) EvaluateException: UncaughtError: Error: Tap position is off-screen at TapAction.start (<anonymous>:59:13) at callback (<anonymous>:9:30) at <anonymous>:17:16 at <anonymous>:18:9 Failure logs 2: https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/a2801424-a196-11e8-bd59-0242ac110006 Screenshot: https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/a3682250-a196-11e8-bd59-0242ac110006 Stack trace is same. Looks like the element that we're trying to tap on but timing out is a "growl" bar: action_runner.TapElement(selector='.growl-dismiss') The screenshot doesn't seem to show any such notification bar. Maybe we need to rerecord a special version of this for Pixel 2? I'm not sure. Juan, any ideas?
,
Aug 28
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/87e497fe79e513579770793f2beae24d6b634536 commit 87e497fe79e513579770793f2beae24d6b634536 Author: Ned Nguyen <nednguyen@google.com> Date: Tue Aug 28 00:10:06 2018 Disable failing rasterize_and_record_micro tests Bug:875878, 865400 Change-Id: I99824f79fef1b49f40cdcd575e55251f5a4ab6cc NOTRY=true TBR=chrishtr@chromium.org, sadrul@chromium.org, vmiura@chromium.org Change-Id: I99824f79fef1b49f40cdcd575e55251f5a4ab6cc Reviewed-on: https://chromium-review.googlesource.com/1192354 Reviewed-by: Ned Nguyen <nednguyen@google.com> Commit-Queue: Ned Nguyen <nednguyen@google.com> Cr-Commit-Position: refs/heads/master@{#586512} [modify] https://crrev.com/87e497fe79e513579770793f2beae24d6b634536/tools/perf/expectations.config
,
Nov 20
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/947b1d7e6c8fd786a3c1e358cdc9de24ff9bf3bb commit 947b1d7e6c8fd786a3c1e358cdc9de24ff9bf3bb Author: Juan Antonio Navarro Perez <perezju@chromium.org> Date: Tue Nov 20 14:51:45 2018 [tools/perf] Disable failing blink_perf.bindings stories in Pixel 2 These are the only two stories still often failing on android-pixel2-perf bot. NOTRY=true TBR=nednguyen@google.com Bug: 865400 Change-Id: I3439893506c11b92d0d18a337ec7eef43c55080b Reviewed-on: https://chromium-review.googlesource.com/c/1344133 Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org> Commit-Queue: Juan Antonio Navarro Pérez <perezju@chromium.org> Cr-Commit-Position: refs/heads/master@{#609699} [modify] https://crrev.com/947b1d7e6c8fd786a3c1e358cdc9de24ff9bf3bb/tools/perf/expectations.config
,
Nov 23
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/6efda466841a14a84cf16b224bc5f80c2dece099 commit 6efda466841a14a84cf16b224bc5f80c2dece099 Author: Juan Antonio Navarro Perez <perezju@chromium.org> Date: Fri Nov 23 11:00:52 2018 [tools/perf] Disable flaky browse:shopping:avito Story in v8.browsing_mobile-future is flay on Pixel 2. NOTRY=true TBR=nednguyen@google.com Bug: 865400 Change-Id: I90c2f93787341b19a680a590df370fb978313f32 Reviewed-on: https://chromium-review.googlesource.com/c/1349311 Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org> Commit-Queue: Juan Antonio Navarro Pérez <perezju@chromium.org> Cr-Commit-Position: refs/heads/master@{#610567} [modify] https://crrev.com/6efda466841a14a84cf16b224bc5f80c2dece099/tools/perf/expectations.config
,
Dec 17
Making this bug now about all the android bots we want to promote out of fyi and into the main perf waterfall, namely: https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/android-pixel2-perf https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/android-go_webview-perf https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/android-pixel2_webview-perf Looking at test results (see [1] for links), there do not seem to be too many single stories consistently failing. However, on the webview bots, shard #0 appears to consistently time out. This, I believe, causes results for a bunch of tests to be missed. Maybe some re-sharding is needed? Passing over to you Ned for those sharding issues. [1]: https://docs.google.com/document/d/1cNZWMkgfiK4FYeI4MX1pjkXHusTJuBNCf1JDJN0GFpM/edit?usp=sharing
,
Dec 17
,
Dec 17
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/4b46b1afc60932cdac3de84a98977ffeb9fac107 commit 4b46b1afc60932cdac3de84a98977ffeb9fac107 Author: Ned Nguyen <nednguyen@google.com> Date: Mon Dec 17 15:52:09 2018 Reshards all perf FYI bots Command run: $ ./tools/perf/generate_perf_sharding update -w perf-fyi Bug: 865400 Change-Id: Ifa3c486d02ee1b189cb94c18cca2e92af515a7de Reviewed-on: https://chromium-review.googlesource.com/c/1379395 Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org> Commit-Queue: Ned Nguyen <nednguyen@google.com> Cr-Commit-Position: refs/heads/master@{#617126} [modify] https://crrev.com/4b46b1afc60932cdac3de84a98977ffeb9fac107/tools/perf/core/shard_maps/android-go_webview-perf_map.json [modify] https://crrev.com/4b46b1afc60932cdac3de84a98977ffeb9fac107/tools/perf/core/shard_maps/android-nexus5x-perf-fyi_map.json [modify] https://crrev.com/4b46b1afc60932cdac3de84a98977ffeb9fac107/tools/perf/core/shard_maps/android-pixel2-perf_map.json [modify] https://crrev.com/4b46b1afc60932cdac3de84a98977ffeb9fac107/tools/perf/core/shard_maps/android-pixel2_webview-perf_map.json [modify] https://crrev.com/4b46b1afc60932cdac3de84a98977ffeb9fac107/tools/perf/core/shard_maps/timing_data/android-go_webview-perf_timing.json [modify] https://crrev.com/4b46b1afc60932cdac3de84a98977ffeb9fac107/tools/perf/core/shard_maps/timing_data/android-nexus5x-perf-fyi_timing.json [modify] https://crrev.com/4b46b1afc60932cdac3de84a98977ffeb9fac107/tools/perf/core/shard_maps/timing_data/android-pixel2-perf_timing.json [modify] https://crrev.com/4b46b1afc60932cdac3de84a98977ffeb9fac107/tools/perf/core/shard_maps/timing_data/android-pixel2_webview-perf_timing.json
,
Dec 17
Reassign this bug to Juan
,
Dec 18
On this one now they are shards #0 and #1 the ones often timing out: https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/android-go_webview-perf
,
Dec 18
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/acdc7a42290dd19ae62a97e845167c407100fa9c commit acdc7a42290dd19ae62a97e845167c407100fa9c Author: Juan Antonio Navarro Perez <perezju@chromium.org> Date: Tue Dec 18 13:11:24 2018 [tools/perf] Disable flaky stories on pixel2_webview Disabling loading.mobile which are flaky on new perf.fyi bot. NOTRY=true TBR=crouleau@chromium.org Bug: 865400 Change-Id: I0e7294d655c4021830e191cb98b4a0f634711a4d Reviewed-on: https://chromium-review.googlesource.com/c/1382493 Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org> Commit-Queue: Juan Antonio Navarro Pérez <perezju@chromium.org> Cr-Commit-Position: refs/heads/master@{#617462} [modify] https://crrev.com/acdc7a42290dd19ae62a97e845167c407100fa9c/tools/perf/expectations.config
,
Dec 18
Ned, see #13, can you take another look at that bot?
,
Dec 18
Looks like the problem was because those shards timed out, we don't have complete timing data of the tests being run, hence the sharding configuration isn't good enough ~__~ I will do a manual copy of loading.mobile/* timing data & reshard the two webview bots.
,
Dec 18
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/9c004cfbc80ebf439d2e5666b81b6d7810cb25e7 commit 9c004cfbc80ebf439d2e5666b81b6d7810cb25e7 Author: Ned Nguyen <nednguyen@google.com> Date: Tue Dec 18 17:37:36 2018 Copy timing data of pixel2-webview to android-go-webview and use them to reshard android-go-webview bot Previously, android-go-webview builder was consistenly timed out on some shards. Resharding them using the latest timing data doesn't work well because we were missing accurate story timing data from timed out shard. A quick check: $ wc -l android-go_webview-perf_timing.json 2925 android-go_webview-perf_timing.json $ wc -l android-pixel2_webview-perf_timing.json 3565 android-pixel2_webview-perf_timing.json To address this, we copy the timing data from android-pixel2_webview-perf_timing.json to android-go_webview-perf_timing.json and reshard android-go_webview-perf with such data. It's not perfect, but will give us a better sharding configuration compared with the missing timing data. Command: $ ./tools/perf/generate_perf_sharding update -b android-go_webview-perf --use-old-timing-data Bug: 865400 Change-Id: I83e26ad98e6d0fbba7c0af398dbee018beb7a028 Reviewed-on: https://chromium-review.googlesource.com/c/1382755 Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org> Commit-Queue: Ned Nguyen <nednguyen@google.com> Cr-Commit-Position: refs/heads/master@{#617550} [modify] https://crrev.com/9c004cfbc80ebf439d2e5666b81b6d7810cb25e7/tools/perf/core/shard_maps/android-go_webview-perf_map.json [modify] https://crrev.com/9c004cfbc80ebf439d2e5666b81b6d7810cb25e7/tools/perf/core/shard_maps/timing_data/android-go_webview-perf_timing.json [modify] https://crrev.com/9c004cfbc80ebf439d2e5666b81b6d7810cb25e7/tools/perf/generate_perf_sharding
,
Dec 20
Shard #0 in https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/android-go_webview-perf still timing out :(
,
Dec 20
Ok, so my new plan of tackling this: 1) Update the shard timeout of android-go_webview-perf to 7 hours. Hopefully that this is enough for every shard to finish. 2) Wait for 3-4 days. 3) Reshard the builder again with more accurate test timing data from the last 3 days.
,
Dec 20
Actually I found out that we only run a subset of benchmarks on android-go (go/android-go-perf-waterfall-2018), so we should do the same for android-go_webview CL: https://chromium-review.googlesource.com/c/chromium/src/+/1387145
,
Dec 20
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/e535ffd290e3004fac40dff050b81ec7f618fb62 commit e535ffd290e3004fac40dff050b81ec7f618fb62 Author: Ned Nguyen <nednguyen@google.com> Date: Thu Dec 20 17:29:38 2018 Reduce the tests to run on android-go_webview-perf and reshard the benchmarks Bug: 865400 Change-Id: I82b17058dec8c8358953dbef6b53b3ed6fe81a92 Reviewed-on: https://chromium-review.googlesource.com/c/1387145 Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org> Commit-Queue: Ned Nguyen <nednguyen@google.com> Cr-Commit-Position: refs/heads/master@{#618258} [modify] https://crrev.com/e535ffd290e3004fac40dff050b81ec7f618fb62/tools/perf/core/bot_platforms.py [modify] https://crrev.com/e535ffd290e3004fac40dff050b81ec7f618fb62/tools/perf/core/shard_maps/android-go_webview-perf_map.json
,
Dec 21
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/874463d823ad4cd3d67e4ae867aba2868700e19c commit 874463d823ad4cd3d67e4ae867aba2868700e19c Author: Juan Antonio Navarro Perez <perezju@chromium.org> Date: Fri Dec 21 10:41:07 2018 [tools/perf] Disable loading.mobile/FlipKart_warm_3g Was meant to be disabled by a previous CL, but didn't got disabled due to some bad copy/pasting. NOTRY=true TBR=crouleau@chromium.org Bug: 865400 Change-Id: I44bec00848efbf0eaa41eb9c304a69304fde4bb2 Reviewed-on: https://chromium-review.googlesource.com/c/1388396 Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org> Commit-Queue: Juan Antonio Navarro Pérez <perezju@chromium.org> Cr-Commit-Position: refs/heads/master@{#618495} [modify] https://crrev.com/874463d823ad4cd3d67e4ae867aba2868700e19c/tools/perf/expectations.config
,
Jan 4
As recorded in: https://docs.google.com/document/d/1cNZWMkgfiK4FYeI4MX1pjkXHusTJuBNCf1JDJN0GFpM/edit Test results look mostly good now on these bots. There are no longer any consistently failing stories. Some low flakiness remains, but I think it should be fine to move these out of fyi now. |
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by perezju@chromium.org
, Jul 19