New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 626174 link

Starred by 2 users
Status: Archived
Owner:
mvstan...@chromium.org
Closed: Dec 27
Cc:
bmeu...@chromium.org
mvstan...@chromium.org
█ nednguyen@chromium.org
EstimatedDays: ----
NextAction: ----
OS: Android
Pri: 2
Type: Bug



Sign in to add a comment

Kraken test is flaky in "telemetry_perf_unittests" on Android

Project Member Reported by brucedaw...@chromium.org, Jul 7 2016 
The Kraken test on "telemetry_perf_unittests on Android" is failing too frequently - about one out of ten builds, which is too frequent. The main cause is unclear - timeout? Incorrect results? The output doesn't seem to say. In one case the failure was because the test device couldn't reach http://krakenbenchmark.mozilla.org/ - are we really depending on an external http URL for our tests?

Details:


On Android Tests builds 28600, 28602, 28614, 28637, and 28641 the "telemetry_perf_unittests on Android" test failed with this unhelpful error:

  Traceback (most recent call last):
    File "/tmp/isolated_runzChA1P/tools/perf/benchmarks/benchmark_smoke_unittest.py", line 77, in BenchmarkSmokeTest
      msg='Failed: %s' % benchmark)
  AssertionError: Failed: <class 'benchmarks.kraken.Kraken'>


Additionally, on build 28647 it failed with a network timeout. URL and output below:
https://build.chromium.org/p/chromium.linux/builders/Android%20Tests/builds/28647/steps/telemetry_perf_unittests%20on%20Android/logs/stdio

  Traceback (most recent call last):
    File "/tmp/isolated_runWG4EK8/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 85, in _RunStoryAndProcessErrorIfNeeded
      state.RunStory(results)
    File "/tmp/isolated_runWG4EK8/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 321, in RunStory
      self._current_page, self._current_tab, results)
    File "/tmp/isolated_runWG4EK8/tools/perf/benchmarks/kraken.py", line 89, in ValidateAndMeasurePage
      'document.title.indexOf("Results") != -1', 700)
    File "/tmp/isolated_runWG4EK8/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 136, in WaitForJavaScriptExpression
      e.message + '\n' + debug_message)
  TimeoutException: Timed out while waiting 700s for IsJavaScriptExpressionTrue.
  Console output:
  (error) http://krakenbenchmark.mozilla.org/favicon.ico:0: Failed to load resource: the server responded with a status of 404 (Not Found)

I'm not sure who to assign this to - I'm hoping the assignee will have some ideas.

Comment 1 by brucedaw...@chromium.org, Jul 7 2016 

I looked back to some older builds and found that build 28462 failed with the usual error:

AssertionError: Failed: <class 'benchmarks.kraken.Kraken'>

So this is not a new problem. The failures significantly hurt the reliability of the commit queue and need to be addressed in some way.

Comment 2 by qyears...@chromium.org, Jul 7 2016

Cc: bmeu...@chromium.org mvstan...@chromium.org nednguyen@chromium.org
Labels: -Pri-3 Pri-2
Summary: Kraken test is flaky (in "telemetry_perf_unittests on Android") (was: Kraken test is flaky)
According to the benchmark owners sheet (https://docs.google.com/spreadsheets/d/1R_1BAOd3xeVtR0jn6wB5HHJ2K25mIbKp3iIRQKkX38o/view#gid=0), mvstanton and bmeurer are the owners of kraken.

AFAIK, isn't the kraken Telemetry benchmark getting the content of krakenbenchmark.mozilla.org from a Web Page Replay recording? Maybe it's not doing that some of the time?

Ned, do you think it's a good idea to add kraken to the blacklist not to be included in benchmark_smoke_unittest.py?

Comment 3 by nedngu...@google.com, Jul 7 2016 

The way to proceed here is adding kraken to the blacklist & assign to benchmark owner to investigate & reenable the smoke coverage.

Comment 4 by qyears...@chromium.org, Jul 7 2016

Cc: qyears...@chromium.org
Owner: mvstan...@chromium.org
Summary: Kraken test is flaky in "telemetry_perf_unittests" on Android (was: Kraken test is flaky (in "telemetry_perf_unittests on Android"))
Sounds reasonable -- https://codereview.chromium.org/2127743003 should add it to the blacklist.

mvstanton or bmeurer could you take this bug? It sounds like there may be multiple causes of flakiness of kraken on Android, and fixing it will involve first looking more closely at the failures on https://build.chromium.org/p/chromium.linux/builders/Android%20Tests.

Possibly the main issue is that kraken is making requests to external sites, when it should be using WPT for everything, including in the Telemetry smoke test.
Project Member

Comment 5 by bugdroid1@chromium.org, Jul 7 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e665af6052f8529a24e2880a09d60066fe9e387a

commit e665af6052f8529a24e2880a09d60066fe9e387a
Author: qyearsley <qyearsley@chromium.org>
Date: Thu Jul 07 04:24:37 2016

Add kraken to the blacklist in benchmark_smoke_unittest.

BUG= 626174 
CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:android_s5_perf_cq

Review-Url: https://codereview.chromium.org/2127743003
Cr-Commit-Position: refs/heads/master@{#404073}

[modify] https://crrev.com/e665af6052f8529a24e2880a09d60066fe9e387a/tools/perf/benchmarks/benchmark_smoke_unittest.py

Comment 6 by brucedaw...@chromium.org, Jul 7 2016 

The only time that I saw a network failure from mozilla.org was for the favicon so it may be that that is the only thing being retrieved remotely. Or, maybe that is coming from a Web Page Replay recording and that failed for some reason.

Thanks for investigating. I was hesitant to disable kraken since it seemed like too important a test, but the failures were a clear distraction.

Comment 7 by qyears...@chromium.org, Aug 13 2016

Cc: -qyears...@chromium.org

Comment 8 by mvstan...@chromium.org, Dec 27

Status: Archived (was: Assigned)
►

Sign in to add a comment