nocturne-paladin: failing HWTest [bvt-inline] Suite job failure |
||
Issue descriptionIn CQ run: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8927881820845345760 nocturne-paladin: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8927879867806249216 experienced a failing HWTest [bt-inline] Suite job failure: https://luci-logdog.appspot.com/logs/chromeos/buildbucket/cr-buildbucket.appspot.com/8927879867806249216/+/steps/HWTest__bvt-inline_/0/stdout http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/264331464-chromeos-test/ ERROR shows an interesting: 12/06 09:37:19.163 ERROR| db:0043| 09:37:19 12/06/18: An operational error occurred during a database operation: (2006, 'MySQL server has gone away'); retrying, don't panic yet DEBUG shows a long pause between "Adding job keyval" and next output, approximately 50 minutes: 12/06 09:38:12.497 INFO | server_job:0217| END GOOD 264331551-chromeos-test/chromeos6-row5-rack19-host3/cheets_StartAndroid_P.stress cheets_StartAndroid_P.stress timestamp=1544117729 localtime=Dec 06 09:35:29 12/06 09:38:12.497 DEBUG| suite:1289| Adding job keyval for cheets_StartAndroid_P.stress=264331551-chromeos-test 12/06 10:29:14.590 ERROR| logging_manager:0626| Current thread 0x00007f87c7967740: 12/06 10:29:14.591 ERROR| logging_manager:0626| File "/usr/local/autotest/server/cros/dynamic_suite/job_status.py", line 144 in _sleep 12/06 10:29:14.591 ERROR| logging_manager:0626| File "/usr/local/autotest/server/cros/dynamic_suite/job_status.py", line 136 in wait_for_results 12/06 10:29:14.591 DEBUG| autoserv:0376| Received SIGTERM
,
Dec 6
Alex here are some other bugs with the long delay after "Adding job keyval", at least one of them also features the MySQL has gone away warning: https://crbug.com/839621 https://crbug.com/771257 https://crbug.com/735514 https://crbug.com/598517
,
Dec 6
All the tests passed but the suite never exited and timed out. Looks like the suite runner got stuck somewhere inside http://cs/chromeos_public/src/third_party/autotest/files/server/cros/dynamic_suite/dynamic_suite.py?l=609&rcl=55d6a913b8d5615e11b30275da9203f272338e75
,
Dec 6
It looks like the last test finished at 10:27 (see https://storage.cloud.google.com/chromeos-autotest-results/264331742-chromeos-test/chromeos6-row5-rack16-host9/login_RetrieveActiveSessions/debug/login_RetrieveActiveSessions.DEBUG) and then the entire suite timed out at 10:29. My guess is that the results uploader (or some other finalizing process) didn't finish in the two minutes after the last test finished and the suite timed out. So this is a regular suite timeout error that looks strange due to a timing coincidence. As to a long pause between "Adding job keyval" and next output - this is normal (normal given that the tests are taking longer than usual, that is), since the tests are running asynchronously and don't report to the same log.
,
Dec 6
Marking closed, as this is most likely a flake. Please, reopen if this (or some other time out of bvt-inline suite) happens again. |
||
►
Sign in to add a comment |
||
Comment 1 by dburger@chromium.org
, Dec 6