New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 725796 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: May 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: ----



Sign in to add a comment

telemetry_perf_unittests failing on 6 builders (Mac10.1{0,1,2}, Linux Tests, Win 7 Tests {,x64})

Project Member Reported by vitaliii@chromium.org, May 24 2017

Issue description

I had a look at first failing builds on all three builders.
The situation seems the same - benchmarks.benchmark_smoke_unittest.BenchmarkSmokeTest.webrtc fails.
The log does not look helpful:
=== LOG STARTS ===
  Trace sizes in bytes: {'traceEvents': 5435932, 'telemetry': 34426, 'tabIds': 40}
  Try printing formatted exception: None None None
  
  Traceback (most recent call last):
    RunBenchmark at /b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/story_runner.py:389
      benchmark.ValidateExpectations(stories)
    ValidateExpectations at /b/s/w/ir/third_party/catapult/telemetry/telemetry/benchmark.py:273
      self._expectations.ValidateAgainstStorySet(story_set)
    ValidateAgainstStorySet at /b/s/w/ir/third_party/catapult/telemetry/telemetry/story/expectations.py:31
      raise TypeError('Story %s is not in the story set.' % expectation)
  TypeError: Story multiple_peerconnections is not in the story set.
  
  Locals:
    expectation : 'multiple_peerconnections'
    s           : <page_sets.webrtc_cases.GetUserMedia object at 0x10bbca990>
    stories     : ['hd_local_stream_10s']
    story_set   : <page_sets.webrtc_cases.WebrtcPageSet object at 0x10bbcae50>
  
  Traceback (most recent call last):
    File "/b/s/w/ir/tools/perf/benchmarks/benchmark_smoke_unittest.py", line 84, in BenchmarkSmokeTest
      msg='Failed: %s' % benchmark)
  AssertionError: Failed: <class 'benchmarks.webrtc.WebrtcPerfBenchmark'>
Retrying failed tests (attempt #3 of 3)...
[ RUN      ] /b/s/w/itGGHOHx/tmp3DxdSC.html
[       OK ] /b/s/w/itGGHOHx/tmp3DxdSC.html (3565 ms)
[1/1] benchmarks.benchmark_smoke_unittest.BenchmarkSmokeTest.webrtc failed unexpectedly 34.0436s:
=== LOG ENDS ===
I had a look at 2 "sampled" failing builds per each of 3 builds, the failures there are as described above.
Summary: telemetry_perf_unittests failing on 6 builders (Mac10.1{0,1,2}, Linux Tests, Win 7 Tests {,x64}) (was: telemetry_perf_unittests failing on 3 builders (Mac10.1{0,1,2}))
The same test fails on 3 more builders:

https://build.chromium.org/p/chromium.linux/builders/Linux%20Tests
https://build.chromium.org/p/chromium.win/builders/Win%207%20Tests%20x64%20%281%29
https://build.chromium.org/p/chromium.win/builders/Win7%20Tests%20%281%29
On Linux Tests FindIt reports https://codereview.chromium.org/2899123002 as a culprit. 

This CL is also present in the first failing builds. The CL disables "webrtc.multiple_peerconnections story" (see issue 725502).

Cc: rnep...@chromium.org nednguyen@chromium.org
Components: Speed>Telemetry Tests>Telemetry
Hi rnephew@ & nednguyen@,

your CL (https://codereview.chromium.org/2899123002) seems to break benchmarks.benchmark_smoke_unittest.BenchmarkSmokeTest.webrtc.
Owner: ehmaldonado@chromium.org
Status: Assigned (was: Started)
I've contacted people cc'ed in issue 725502 via chat.

ehmaldonado@ confirmed that revert of https://codereview.chromium.org/2899123002 will cause failing of multiple_peerconnections tests.
ehmaldonado@ effectively reverted https://codereview.chromium.org/2899123002, while disabling the multiple_peerconnections test in a different way. The CL is https://codereview.chromium.org/2896253004 and we are landing it. 
Project Member

Comment 8 by bugdroid1@chromium.org, May 24 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a1db6b7d87922f09904037e82447170cc97c0a2e

commit a1db6b7d87922f09904037e82447170cc97c0a2e
Author: ehmaldonado <ehmaldonado@chromium.org>
Date: Wed May 24 07:39:44 2017

[Telemetry] Disable webrtc.multiple_peerconnections story.

It is failing on all platforms.
The previous CL (https://codereview.chromium.org/2899123002) caused builders to break.

BUG=725502, 725796 
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
TBR=rnephew@chromium.org, nednguyen@google.com, vitaliii@chromium.org

Review-Url: https://codereview.chromium.org/2896253004
Cr-Commit-Position: refs/heads/master@{#474198}

[modify] https://crrev.com/a1db6b7d87922f09904037e82447170cc97c0a2e/tools/perf/benchmarks/webrtc.py
[modify] https://crrev.com/a1db6b7d87922f09904037e82447170cc97c0a2e/tools/perf/page_sets/webrtc_cases.py

I forcelanded the CL without waiting for trybots (it is effectively a revert of a recent CL with a trivial fix on top).
Linux Tests builder is green now (https://build.chromium.org/p/chromium.linux/builders/Linux%20Tests/builds/56978). Waiting for other builders to pickup the revert.
They all are green now.
Thanks ehmaldonado@ for the fix.
Status: Fixed (was: Assigned)
Components: Test>Telemetry
Components: -Speed>Telemetry

Sign in to add a comment