PaygenTestBeta test suites time out and are killed. |
||||||
Issue descriptionThe test team are looking at this build today: https://cros-goldeneye.corp.google.com/chromeos/console/viewRelease?releaseName=M62-BETA-CHROMEOS-4 On this build, two boards (sand & enguarde) are missing AU test results seemingly because the builder failed to generate all of the AU tests that were needed to be run as part of their paygen_au_beta suite Enguarde failed log: https://uberchromegw.corp.google.com/i/chromeos_release/builders/sand-release%20release-R62-9901.B/builds/45/steps/PaygenTestBeta/logs/stdio Vs a successful run on the same branch: https://uberchromegw.corp.google.com/i/chromeos_release/builders/sand-release%20release-R62-9901.B/builds/42/steps/PaygenTestBeta/logs/stdio The suite linked to from the builder page shows the missing delta autoupdate_EndToEndTest_paygen_au_beta_delta_9901.49.0: http://cautotest-prod.corp.google.com/afe/#tab_id=view_job&object_id=149768223 But since it is marked as aborted GE does not find test results for it: https://uberchromeos-server38.corp.google.com/new_tko/#tab_id=spreadsheet_view&row=job_name%252Ctest_name%252Cplatform&column=status&show_incomplete=true&show_only_latest=false&show_invalid=true&condition=job_name%2520LIKE%2520%27sand-release/R62-9901.49.0/paygen%255C_au%255C_beta/%2525%27%2520AND%2520test_name%2520%253C%253E%2520%27SERVER_JOB%27%2520AND%2520test_name%2520NOT%2520LIKE%2520%27CLIENT%255C_JOB%2525%27%2520AND%2520platform%2520LIKE%2520%27sand%27
,
Oct 18 2017
In those examples, all tests were generated, but did not finish in time and were aborted. IE: The test suite did not run to completion. This is either a lab performance issue, or a product bug related to poor update performance/stability.
,
Oct 18 2017
,
Oct 18 2017
There are other boards where the suite timed out because the test wasn't scheduled in time. E.g. http://cautotest-prod.corp.google.com/afe/#tab_id=view_job&object_id=149763749 But Goldeneye lists them as failed, where as in the two boards I mentioned in #1, it shows them as Missing
,
Oct 19 2017
The test suite is fully generated it's scheduled. After it's scheduled, the builders don't do anything but wait for the results. However, there are many different kinds of timeouts, and they aren't all reported the same way. Ah... I missed this before the builder hit this timeout on the builder side: 00:22:30: WARNING: No output from <_BackgroundTask(_BackgroundTask-7:6:7:3, started)> for 8610 seconds That is a known error being worked on related to how builders wait build suite results. After that happens, the builder thinks the test failed. They may or may not be canceled in the lab, and that cancellation may be reported... oddly. crbug.com/772985
,
Oct 24 2017
This seems to be causing a large number of 63 release builders to fail, ie, - https://uberchromegw.corp.google.com/i/chromeos_release/builders/asuka-release%20release-R63-10032.B/builds/12 - https://uberchromegw.corp.google.com/i/chromeos_release/builders/banon-release%20release-R63-10032.B/builds/12 - https://uberchromegw.corp.google.com/i/chromeos_release/builders/bob-release%20release-R63-10032.B/builds/12
,
Nov 22 2017
https://uberchromegw.corp.google.com/i/chromeos_release/builders/asuka-release%20release-R63-10032.B has been green, archiving |
||||||
►
Sign in to add a comment |
||||||
Comment 1 by dhadd...@chromium.org
, Oct 18 2017