New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 780503 link

Starred by 1 user

Issue metadata

Status: Duplicate
Closed: Nov 2017
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: ----

Sign in to add a comment

cave-release:1635 failed

Project Member Reported by, Nov 1 2017

Issue description

cave-release:1635 failed

Builders failed on: 
- cave-release:

This one is a little tricky to follow, but we can see that the suite gets aborted:
Only 2 of the tests in the suite run, and this is the last line from the suite log:
11/01 02:05:14.728 DEBUG|             suite:1378| Adding job keyval for autoupdate_EndToEndTest.paygen_au_canary_full=153008096-chromeos-test

Previously, we see a really slow step communicating with MySQL:
11/01 00:56:13.323 DEBUG|             suite:1378| Adding job keyval for autoupdate_EndToEndTest.paygen_au_canary_delta=153008091-chromeos-test
11/01 02:04:46.180 ERROR|                db:0023| 02:04:46 11/01/17: An operational error occurred during a database operation: (2006, 'MySQL server has gone away'); retrying, don't panic yet

Back over in the PaygenTestCanary log:

02:10:06: INFO: Refreshing due to a 401 (attempt 1/2)
02:10:06: INFO: Refreshing access_token
02:13:46: INFO: Refreshing due to a 401 (attempt 1/2)
02:13:46: INFO: Refreshing access_token
02:16:38: INFO: Refreshing due to a 401 (attempt 1/2)
02:16:38: INFO: Refreshing access_token
02:23:44: WARNING: Killing tasks: [<_BackgroundTask(_BackgroundTask-7:6:6:3, started)>]
02:23:44: WARNING: Killing 16475 (sig=24 SIGXCPU)

Have you run into issues like this with MySQL slowness before? This does seem like a really long time.
The PaygenTestCanary failure looks similar to
The MySQL error is a red herring.

All the suite job does is create its child jobs and wait on them.  The MySQL connection it holds can time out (MySQL server has gone away), so it refreshes it.

The "slow step communicating with MySQL" is just the suite job doing nothing waiting for any of its tests to finish.

The error is simply the suite timing out.  I will need to dig to find out why; a blind guess is that there are not enough DUTs or the host scheduler which is responsible for matching tests to DUTs to run has a problem.
Mergedinto: 781682
Status: Duplicate (was: Assigned)

Sign in to add a comment