New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 634319 link

Starred by 2 users

Issue metadata

Status: WontFix
Owner:
Closed: Sep 2016
Cc:
EstimatedDays: ----
NextAction: ----
OS: Android
Pri: 2
Type: Bug-Regression



Sign in to add a comment

memory.blink_memory_mobile.reference flaky on Android Nexus5X Perf (1) on chromium.perf

Project Member Reported by petrcermak@chromium.org, Aug 4 2016

Issue description

Revision range first seen: Since the bot stopped being purple, ~70 builds ago (https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5X%20Perf%20%281%29/builds/2444)
Link to failing step log: https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5X%20Perf%20%281%29/builds/2511

The failure is always due to tap position being off-screen on the Verge:

Traceback (most recent call last):
  File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 85, in _RunStoryAndProcessErrorIfNeeded
    state.RunStory(results)
  File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 295, in RunStory
    self._current_page.Run(self)
  File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/page/__init__.py", line 99, in Run
    self.RunPageInteractions(action_runner)
  File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/tools/perf/page_sets/blink_memory_mobile.py", line 53, in RunPageInteractions
    selector=TheVergePage.COMMENT_LINK_SELECTOR)
  File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py", line 222, in TapElement
    selector=selector, text=text, element_function=element_function))
  File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py", line 45, in _RunAction
    action.RunAction(self._tab)
  File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/actions/tap.py", line 69, in RunAction
    element_function=self.element_function)
  File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/actions/page_action.py", line 116, in EvaluateCallbackWithElement
    return tab.EvaluateJavaScript(code)
  File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 187, in EvaluateJavaScript
    expr, context_id=None, timeout=timeout)
  File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 215, in EvaluateJavaScriptInContext
    expr, context_id=context_id, timeout=timeout)
  File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 32, in inner
    return func(inspector_backend, *args, **kwargs)
  File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 203, in EvaluateJavaScript
    return self._runtime.Evaluate(expr, context_id, timeout)
  File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_runtime.py", line 52, in Evaluate
    raise exceptions.EvaluateException(res['result']['result']['description'])
EvaluateException: Error: Tap position is off-screen
    at TapAction.start (<anonymous>:59:13)
    at callback (<anonymous>:10:28)
    at <anonymous>:18:16
    at <anonymous>:19:9
    at Object.InjectedScript._evaluateOn (<anonymous>:875:140)
    at Object.InjectedScript._evaluateAndWrap (<anonymous>:808:34)
    at Object.InjectedScript.evaluate (<anonymous>:664:21)

Screenshot: https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/profiler-file-id_2-2016-08-04_04-08-1810797.png

The benchmark failed 6 times in the last 20 runs (more than 1 our of 5 runs), so I'm going to disable it on reference (per sheriffing instructions: https://chromium.googlesource.com/chromium/src/+/master/tools/perf/docs/perf_bot_sheriffing.md).

If the test is disabled, please downgrade to Pri-2.

 
Disabling on Nexus 5X reference: https://codereview.chromium.org/2215783002/
Project Member

Comment 2 by bugdroid1@chromium.org, Aug 4 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3a4f654ac5c0ddd30f9194531add5b108f43d1dd

commit 3a4f654ac5c0ddd30f9194531add5b108f43d1dd
Author: petrcermak <petrcermak@chromium.org>
Date: Thu Aug 04 15:26:47 2016

[perf] Disable memory.blink_memory_mobile.reference on Nexus 5X

Rationale: The benchmark is very flaky on Nexus 5X (6 failures in the
last 20 runs).

BUG= 634319 
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.perf:android_s5_perf_cq;master.tryserver.chromium.perf:linux_perf_cq;master.tryserver.chromium.perf:mac_retina_perf_cq;master.tryserver.chromium.perf:winx64_10_perf_cq
TBR=skyostil

Review-Url: https://codereview.chromium.org/2215783002
Cr-Commit-Position: refs/heads/master@{#409789}

[modify] https://crrev.com/3a4f654ac5c0ddd30f9194531add5b108f43d1dd/tools/perf/benchmarks/memory_infra.py

Cc: petrcermak@chromium.org
Labels: -Pri-1 Pri-2
Owner: bashi@chromium.org
Status: Assigned (was: Untriaged)
The benchmark is now disabled, so I'm lowering the priority and re-assigning to the benchmark owner.
Perf sheriff ping
bashi@

Is there an update on this bug?
bashi: should we close this? are you okay with the test not working on N5X reference?

Comment 7 by bashi@chromium.org, Sep 25 2016

Status: WontFix (was: Assigned)
Yes. Sorry for not responding...

Sign in to add a comment