[system-health] Re-enable system_health.memory_mobile.reference on Nexus 5X |
||||||||
Issue descriptionFailing bot: https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5X%20Perf%20%281%29 Sample failing build stdio: https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5X%20Perf%20%281%29/builds/1050/steps/system_health.memory_mobile.reference/logs/stdio Error: [ RUN ] load:tools:gmail (INFO) 2016-05-15 03:09:18,431 cmd_helper._ValidateAndLogCommand:153 [host]> /b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00b2203389f336e2 shell '( /data/local/tmp/profilers/memtrack_helper -d );echo %$?' (INFO) 2016-05-15 03:09:18,533 cmd_helper._ValidateAndLogCommand:153 [host]> /b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00b2203389f336e2 shell '( echo -n '"'"'{"trace_config":{"excluded_categories": ["*"], "included_categories": ["disabled-by-default-memory-infra"], "record_mode": "record-as-much-as-possible"}}'"'"' > /data/local/chrome-trace-config.json );echo %$?' (INFO) 2016-05-15 03:09:18,646 tracing_backend.StartTracing:124 Start Tracing Request: {'params': {'transferMode': 'ReturnAsStream', 'options': 'record-as-much-as-possible', 'categories': 'disabled-by-default-memory-infra,-*'}, 'method': 'Tracing.start'} (INFO) 2016-05-15 03:09:18,657 cache_temperature.EnsurePageCacheTemperature:29 PageCacheTemperature: any (INFO) 2016-05-15 03:10:54,190 cmd_helper._ValidateAndLogCommand:153 [host]> /b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00b2203389f336e2 shell '( logcat -d -t 500 );echo %$?' (ERROR) 2016-05-15 03:10:55,480 exceptions.__init__:71 Problem when trying to gather standard output: 'NoneType' object has no attribute 'splitlines' (INFO) 2016-05-15 03:10:55,485 cmd_helper._ValidateAndLogCommand:153 [host]> /b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00b2203389f336e2 shell '( /system/bin/screencap -p /data/local/tmp/temp_file-997785ca65ca0.png );echo %$?' (INFO) 2016-05-15 03:10:56,747 cmd_helper._ValidateAndLogCommand:153 [host]> /b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00b2203389f336e2 pull /data/local/tmp/temp_file-997785ca65ca0.png /tmp/tmpZMjFoL.png (INFO) 2016-05-15 03:10:56,937 cmd_helper._ValidateAndLogCommand:153 [host]> /b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00b2203389f336e2 shell 'rm -f /data/local/tmp/temp_file-997785ca65ca0.png' Traceback (most recent call last): File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 84, in _RunStoryAndProcessErrorIfNeeded state.RunStory(results) File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 330, in RunStory self._current_page.Run(self) File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/page/__init__.py", line 95, in Run shared_state.page_test.RunNavigateSteps(self, current_tab) File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/page/legacy_page_test.py", line 191, in RunNavigateSteps page.RunNavigateSteps(action_runner) File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/tools/perf/page_sets/system_health_stories.py", line 323, in RunNavigateSteps self._page_spec.login_hook(action_runner, self.credentials_path) File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/tools/perf/page_sets/system_health_stories.py", line 28, in _LogIntoGoogleAccountAndSetUpGmailSession 'https://mail.google.com/mail/mu/mp/872/trigger_redirection_loop') File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py", line 121, in Navigate timeout_in_seconds=timeout_in_seconds)) File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py", line 44, in _RunAction action.RunAction(self._tab) File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/actions/navigate.py", line 23, in RunAction self._timeout_in_seconds) File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 252, in Navigate self._inspector_backend.Navigate(url, script_to_evaluate_on_commit, timeout) File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 35, in inner inspector_backend._ConvertExceptionFromInspectorWebsocket(e) File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 32, in inner return func(inspector_backend, *args, **kwargs) File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 167, in Navigate self._page.Navigate(url, script_to_evaluate_on_commit, timeout) File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_page.py", line 125, in Navigate self.WaitForNavigate(timeout) File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_page.py", line 94, in WaitForNavigate self._inspector_websocket.DispatchNotifications(remaining_time) File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 134, in DispatchNotifications self._Receive(timeout) File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 149, in _Receive data = self._socket.recv() File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/third_party/websocket-client/websocket.py", line 596, in recv opcode, data = self.recv_data() File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/third_party/websocket-client/websocket.py", line 606, in recv_data frame = self.recv_frame() File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/third_party/websocket-client/websocket.py", line 637, in recv_frame self._frame_header = self._recv_strict(2) File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/third_party/websocket-client/websocket.py", line 746, in _recv_strict bytes = self._recv(shortage) File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/third_party/websocket-client/websocket.py", line 732, in _recv raise WebSocketTimeoutException(e.message) TimeoutException: <unprintable TimeoutException object> (INFO) 2016-05-15 03:10:56,984 cmd_helper._ValidateAndLogCommand:153 [host]> /b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00b2203389f336e2 shell '( rm -f /data/local/chrome-trace-config.json );echo %$?' (INFO) 2016-05-15 03:10:57,142 cmd_helper._ValidateAndLogCommand:153 [host]> /b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00b2203389f336e2 shell '( ps | grep -F memtrack_helper; echo "PIPESTATUS: ${PIPESTATUS[@]}" );echo %$?' (INFO) 2016-05-15 03:10:57,256 device_utils.KillAll:911 KillAll('memtrack_helper', ...) attempting to kill the following: (INFO) 2016-05-15 03:10:57,256 device_utils.KillAll:914 31420 /data/local/tmp/profilers/memtrack_helper (INFO) 2016-05-15 03:10:57,257 cmd_helper._ValidateAndLogCommand:153 [host]> /b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00b2203389f336e2 shell '( kill -9 31420 );echo %$?' (INFO) 2016-05-15 03:10:57,317 cmd_helper._ValidateAndLogCommand:153 [host]> /b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00b2203389f336e2 shell '( ps | grep -F memtrack_helper; echo "PIPESTATUS: ${PIPESTATUS[@]}" );echo %$?' (INFO) 2016-05-15 03:10:57,419 timeout_retry.WaitFor:106 condition 'all_pids_killed' met (0.3s) [ FAILED ] load:tools:gmail (99013 ms) Next action: I will disable the benchmark on Nexus 5X reference and investigate the issue locally.
,
May 16 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/b31c89283c32a93b4e9c4a317505283dd7c77651 commit b31c89283c32a93b4e9c4a317505283dd7c77651 Author: petrcermak <petrcermak@chromium.org> Date: Mon May 16 16:38:03 2016 [system-health] Disable system_health.memory_mobile.reference on Nexus 5X Rationale: load:tools:gmail fails consistently on "Android Nexus5X Perf (1)" bot. BUG= 612144 ,589726 CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:winx64_10_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq;tryserver.chromium.perf:linux_perf_cq Review-Url: https://codereview.chromium.org/1985633003 Cr-Commit-Position: refs/heads/master@{#393843} [modify] https://crrev.com/b31c89283c32a93b4e9c4a317505283dd7c77651/tools/perf/benchmarks/system_health.py
,
May 16 2016
The benchmark is now disabled, so I'm reducing the priority.
,
May 16 2016
I'm able to reproduce the issue locally on a Nexus 5X. The benchmark times out even if I force it to use the 32-bit apk.
For some reason, the device is super-slow when loading the URLs, which causes the timeout. The only fix I could find is to increase the timeout:
action_runner.Navigate(
'https://mail.google.com/mail/mu/mp/872/trigger_redirection_loop',
timeout_in_seconds=300)
However, I think it's better to just leave the benchmark disabled on Nexus 5X reference builds for a few weeks until the next stable release. If the problem persists, then we'll need to look deeper into the difference between reference and non-reference builds.
,
May 18 2016
,
Jul 25 2016
This is now blocked on https://github.com/catapult-project/catapult/issues/2533.
,
Aug 3 2016
The reference build has been rolled (https://codereview.chromium.org/2189533003) and then unrolled (https://codereview.chromium.org/2205643003) again, so we still need to wait.
,
Aug 15 2016
Still blocked on https://github.com/catapult-project/catapult/issues/2533.
,
Sep 28 2016
Still blocked :-(
,
Oct 3 2016
,
Oct 3 2016
,
Oct 5 2016
,
Oct 5 2016
,
Feb 3 2017
Just tested on device, seems to be working fine, so since reference versions were incremented in https://codereview.chromium.org/2582303003/patch/1/10001 and blocker https://github.com/catapult-project/catapult/issues/2533 was resolved, I think it's reasonable to just re-enable this test (revert https://codereview.chromium.org/1985633003 ).
,
Feb 6 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/3084cf9e91e970838660a8ea381f5771cf8d54dd commit 3084cf9e91e970838660a8ea381f5771cf8d54dd Author: kraynov <kraynov@chromium.org> Date: Mon Feb 06 11:39:10 2017 [system-health] Re-enable memory_mobile.reference on Nexus 5X. Reference versions were incremented in https://crrev.com/2582303003 and problem seems not being happening anymore. BUG= 612144 Review-Url: https://codereview.chromium.org/2670363002 Cr-Commit-Position: refs/heads/master@{#448233} [modify] https://crrev.com/3084cf9e91e970838660a8ea381f5771cf8d54dd/tools/perf/benchmarks/system_health.py
,
Feb 7 2017
Thanks Grigoriy! |
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by petrcermak@chromium.org
, May 16 2016