New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 628713 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
OOO until 2019-01-24
Closed: Jul 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: All
Pri: 1
Type: Bug

Blocking:
issue 352807
issue 628022



Sign in to add a comment

Individual WebGL 2.0 conformance test failures are causing reports of missing shards

Project Member Reported by kbr@chromium.org, Jul 15 2016

Issue description

In this run:
https://build.chromium.org/p/chromium.gpu.fyi/builders/Mac%20Retina%20Release/builds/5615

apparently only one test failed on one shard (shard #6 -- https://chromium-swarm.appspot.com/user/task/300805d31945aa10 ), but the step was reported as having a missing shard, and which shard's output was missing was not reported. The failing test was also not reported.

Need to get to the bottom of this ASAP.

 

Comment 1 by kbr@chromium.org, Jul 15 2016

Cc: nedngu...@google.com vadimsh@chromium.org eyaich@chromium.org ynovikov@chromium.org dpranke@chromium.org
Status: Started (was: Assigned)
The reason is that shard #9 failed catastrophically:
https://chromium-swarm.appspot.com/user/task/300805d55514a210

Here's the stack trace:

INFO:root:Chose browser: PossibleDesktopBrowser(type=release, executable=/b/swarm_slave/w/ir3p2zyF/out/Release/Chromium.app/Contents/MacOS/Chromium, flash=None)
INFO:root:Requested remote debugging port: 0
INFO:root:Starting Chrome ['/b/swarm_slave/w/ir3p2zyF/out/Release/Chromium.app/Contents/MacOS/Chromium', '--enable-unsafe-es3-apis', '--test-type=gpu', '--disable-domain-blocking-for-3d-apis', '--disable-gesture-requirement-for-media-playback', '--disable-gpu-process-crash-limit', '--disable-accelerated-video-decode', '--enable-experimental-canvas-features', '--js-flags=--expose-gc', '--enable-logging=stderr', '--enable-net-benchmarking', '--metrics-recording-only', '--no-default-browser-check', '--no-first-run', '--enable-gpu-benchmarking', '--disable-background-networking', '--no-proxy-server', '--disable-component-extensions-with-background-pages', '--disable-default-apps', '--remote-debugging-port=0', '--enable-crash-reporter-for-testing', '--window-size=1280,1024', '--user-data-dir=/b/swarm_slave/w/itcxu7zt/tmptJEdsR', 'about:blank']
INFO:root:Discovered ephemeral port 49183
INFO:root:Found crashpad_database_util
INFO:root:No minidump found via crashpad_database_util
INFO:root:Found crashpad_database_util
INFO:root:No minidump found via crashpad_database_util
Can't get standard output with --show-stdout
[344:13319:0715/084420:ERROR:ipc_channel_mojo.cc(332)] Failed to flush pending messages
[320:1287:0715/084420:WARNING:url_request_context_getter.cc(43)] URLRequestContextGetter leaking due to no owning thread.
ERROR:root:Failure while starting browser backend.
Traceback (most recent call last):
  File "/b/swarm_slave/w/ir3p2zyF/third_party/catapult/telemetry/telemetry/internal/browser/browser.py", line 55, in __init__
    self._browser_backend.Start()
  File "/b/swarm_slave/w/ir3p2zyF/third_party/catapult/telemetry/telemetry/internal/backends/chrome/desktop_browser_backend.py", line 291, in Start
    self._WaitForBrowserToComeUp()
  File "/b/swarm_slave/w/ir3p2zyF/third_party/catapult/telemetry/telemetry/internal/backends/chrome/chrome_browser_backend.py", line 174, in _WaitForBrowserToComeUp
    raise exceptions.BrowserConnectionGoneException(self.browser, e)
BrowserConnectionGoneException: Timed out while waiting 60s for HasBrowserFinishedLaunching.
Found Minidump: False
Stack Trace:
********************************************************************************
	No crash dump found.
********************************************************************************
Standard output:
********************************************************************************
********************************************************************************
ERROR

======================================================================
ERROR: setUpClass (gpu_tests.webgl_conformance_integration_test.WebGLConformanceIntegrationTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/b/swarm_slave/w/ir3p2zyF/content/test/gpu/gpu_tests/webgl_conformance_integration_test.py", line 77, in setUpClass
    cls.StartBrowser()
  File "/b/swarm_slave/w/ir3p2zyF/third_party/catapult/telemetry/telemetry/testing/serially_executed_browser_test_case.py", line 83, in StartBrowser
    cls.browser = cls._browser_to_create.Create(cls._browser_options)
  File "/b/swarm_slave/w/ir3p2zyF/third_party/catapult/telemetry/telemetry/internal/backends/chrome/desktop_browser_finder.py", line 68, in Create
    browser_backend, self._platform_backend, self._credentials_path)
  File "/b/swarm_slave/w/ir3p2zyF/third_party/catapult/telemetry/telemetry/internal/browser/browser.py", line 55, in __init__
    self._browser_backend.Start()
  File "/b/swarm_slave/w/ir3p2zyF/third_party/catapult/telemetry/telemetry/internal/backends/chrome/desktop_browser_backend.py", line 291, in Start
    self._WaitForBrowserToComeUp()
  File "/b/swarm_slave/w/ir3p2zyF/third_party/catapult/telemetry/telemetry/internal/backends/chrome/chrome_browser_backend.py", line 174, in _WaitForBrowserToComeUp
    raise exceptions.BrowserConnectionGoneException(self.browser, e)
BrowserConnectionGoneException: Timed out while waiting 60s for HasBrowserFinishedLaunching.
Found Minidump: False
Stack Trace:
********************************************************************************
	No crash dump found.
********************************************************************************
Standard output:
********************************************************************************
********************************************************************************


The browser_test_runner apparently didn't get a chance to write its JSON. It should probably have caught this failure and said that the shard's output was invalid. Still, the collect step needs to be robust to this.

Fixing in https://codereview.chromium.org/2150993005 .

I think we should also fix serially_executed_browser_test_case.py so that browser_test_runner actually put all the test it's supposed to run in the failure list?

Comment 3 by kbr@chromium.org, Jul 15 2016

Blocking: 628022

Comment 4 by kbr@chromium.org, Jul 15 2016

Yes, I agree some fix should be done to serially_executed_browser_test_case.py or browser_test_runner.py to do what you suggest, but I don't know what it'd need to be. Neither of those two classes is in the stack trace above.

Project Member

Comment 5 by bugdroid1@chromium.org, Jul 15 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/41c3f867f5138ee0dd392e161222ebde963c5c35

commit 41c3f867f5138ee0dd392e161222ebde963c5c35
Author: kbr <kbr@chromium.org>
Date: Fri Jul 15 19:26:34 2016

Make _merge_isolated_script_shards robust to harness failures.

Occasionally the harness fails catastrophically and doesn't write any
output.

BUG= 628713 

Review-Url: https://codereview.chromium.org/2150993005

[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipe_modules/swarming/api.py
[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipe_modules/test_utils/test_api.py
[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipes/chromium.expected/dynamic_isolated_script_test_harness_failure_zero_retcode.json
[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipes/chromium.expected/dynamic_swarmed_failed_isolated_script_test.json
[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipes/chromium.expected/dynamic_swarmed_isolated_script_test_linux_gpu.json
[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipes/chromium.expected/dynamic_swarmed_isolated_script_test_mac_gpu.json
[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipes/chromium.expected/dynamic_swarmed_isolated_script_test_win_gpu.json
[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipes/chromium.expected/dynamic_swarmed_isolated_script_test_win_non_gpu.json
[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipes/chromium.expected/dynamic_swarmed_passed_isolated_script_test.json
[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipes/chromium.expected/dynamic_swarmed_passed_isolated_script_test_with_swarming_failure.json
[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipes/chromium.expected/dynamic_swarmed_passed_with_bad_retcode_isolated_script_test.json
[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_failed_isolated_script_test.json
[add] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_isolated_script_test_harness_failure.json
[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_isolated_script_test_missing_shard.json
[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_passed_isolated_script_test.json
[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipes/chromium.py
[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipes/chromium_trybot.expected/dynamic_isolated_script_test_with_args_on_trybot.json

Project Member

Comment 6 by bugdroid1@chromium.org, Jul 15 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/41c3f867f5138ee0dd392e161222ebde963c5c35

commit 41c3f867f5138ee0dd392e161222ebde963c5c35
Author: kbr <kbr@chromium.org>
Date: Fri Jul 15 19:26:34 2016

Make _merge_isolated_script_shards robust to harness failures.

Occasionally the harness fails catastrophically and doesn't write any
output.

BUG= 628713 

Review-Url: https://codereview.chromium.org/2150993005

[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipe_modules/swarming/api.py
[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipe_modules/test_utils/test_api.py
[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipes/chromium.expected/dynamic_isolated_script_test_harness_failure_zero_retcode.json
[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipes/chromium.expected/dynamic_swarmed_failed_isolated_script_test.json
[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipes/chromium.expected/dynamic_swarmed_isolated_script_test_linux_gpu.json
[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipes/chromium.expected/dynamic_swarmed_isolated_script_test_mac_gpu.json
[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipes/chromium.expected/dynamic_swarmed_isolated_script_test_win_gpu.json
[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipes/chromium.expected/dynamic_swarmed_isolated_script_test_win_non_gpu.json
[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipes/chromium.expected/dynamic_swarmed_passed_isolated_script_test.json
[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipes/chromium.expected/dynamic_swarmed_passed_isolated_script_test_with_swarming_failure.json
[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipes/chromium.expected/dynamic_swarmed_passed_with_bad_retcode_isolated_script_test.json
[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_failed_isolated_script_test.json
[add] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_isolated_script_test_harness_failure.json
[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_isolated_script_test_missing_shard.json
[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_passed_isolated_script_test.json
[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipes/chromium.py
[modify] https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35/scripts/slave/recipes/chromium_trybot.expected/dynamic_isolated_script_test_with_args_on_trybot.json

Ken, if setUpClass failed, python's unittest runner will set results._classSetupFailed to True (see " def _handleClassSetUp(self, test, result):" in https://svn.python.org/projects/stackless/trunk/Lib/unittest/suite.py).
browser_test_runner can check for that property to set the failure list.
Project Member

Comment 8 by bugdroid1@chromium.org, Jul 15 2016

Project Member

Comment 9 by bugdroid1@chromium.org, Jul 15 2016

Project Member

Comment 10 by bugdroid1@chromium.org, Jul 15 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra.git/+/17b8270549e3ae13c36b96f970183c2652f4e8a1

commit 17b8270549e3ae13c36b96f970183c2652f4e8a1
Author: recipe-roller <recipe-roller@chromium.org>
Date: Fri Jul 15 19:49:36 2016

Roll recipe dependencies (trivial).

This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug
(or complain)

build:
  https://crrev.com/41c3f867f5138ee0dd392e161222ebde963c5c35 Make _merge_isolated_script_shards robust to harness failures. (kbr@chromium.org)

R=kbr@chromium.org
BUG= 628713 

TBR=martiniss@chromium.org,phajdan.jr@chromium.org

Review-Url: https://codereview.chromium.org/2157583002

[modify] https://crrev.com/17b8270549e3ae13c36b96f970183c2652f4e8a1/infra/config/recipes.cfg

Comment 11 by kbr@chromium.org, Jul 15 2016

Status: Fixed (was: Started)

Sign in to add a comment