|autoserv logs missing for some timed-out jobs|
|Project Member Reported by email@example.com, Feb 21 2017||Back to list|
No logs are collected for some jobs, making it impossible to do any further debugging. For instance: https://uberchromegw.corp.google.com/i/chromeos/builders/expresso-release/builds/1038 https://uberchromegw.corp.google.com/i/chromeos/builders/expresso-release/builds/1038/steps/AUTest%20%5Bau%5D/logs/stdio Suite job [ PASSED ] platform_Powerwash [ FAILED ] platform_Powerwash ABORT: Timed out, did not run. autoupdate_EndToEndTest.npo_delta [ PASSED ] autoupdate_Rollback [ PASSED ] Suite timings: Downloads started at 2017-02-21 05:33:23 Payload downloads ended at 2017-02-21 05:33:26 Suite started at 2017-02-21 05:34:06 Artifact downloads ended (at latest) at 2017-02-21 05:34:18 Testing started at 2017-02-21 05:37:50 Testing ended at 2017-02-21 05:54:41 Links to test logs: Suite job http://cautotest/tko/retrieve_logs.cgi?job=/results/102542367-chromeos-test/ platform_Powerwash http://cautotest/tko/retrieve_logs.cgi?job=/results/102542367-chromeos-test/ autoupdate_EndToEndTest.npo_delta http://cautotest/tko/retrieve_logs.cgi?job=/results/102542579-chromeos-test/ autoupdate_Rollback http://cautotest/tko/retrieve_logs.cgi?job=/results/102542583-chromeos-test/ (Also, I don't find it very useful to have the platform_Powerwash URL point to the suite job URL) The suite job points to the autoserv.DEBUG here: https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/102542367-chromeos-test/hostless/debug/ which shows the job getting scheduled and assigned a job ID: 02/21 05:34:27.899 DEBUG| suite:0854| Scheduling autoupdate_EndToEndTest_npo_delta_9304.0.0 02/21 05:34:28.339 DEBUG| suite:1206| Adding job keyval for autoupdate_EndToEndTest_npo_delta_9304.0.0=102542579-chromeos-test 02/21 05:34:28.340 DEBUG| suite:0854| Scheduling autoupdate_Rollback 02/21 05:34:28.745 DEBUG| suite:1206| Adding job keyval for autoupdate_Rollback=102542583-chromeos-test 02/21 05:34:28.746 DEBUG| suite:0854| Scheduling platform_Powerwash 02/21 05:34:29.169 DEBUG| suite:1206| Adding job keyval for platform_Powerwash=102542586-chromeos-test 02/21 05:34:29.169 DEBUG| suite:0927| Scheduled 3 tests, writing the total to keyval. 02/21 05:34:29.170 DEBUG| dynamic_suite:0593| Waiting on suite. However, the bucket for that job ID is empty, and we hit a wall.
Feb 21 2017,
A related problem (maybe it should be a separate bug?): Some jobs which are linked to the same bucket as the "suite job", but the job ID in autoserv.DEBUG points to a valid bucket. For instance: https://uberchromegw.corp.google.com/i/chromeos/builders/caroline-release/builds/426 https://uberchromegw.corp.google.com/i/chromeos/builders/caroline-release/builds/426/steps/HWTest%20%5Bbvt-inline%5D/logs/stdio snippets: Suite job [ PASSED ] platform_DMVerityBitCorruption [ FAILED ] platform_DMVerityBitCorruption ABORT: Timed out, did not run. ... Links to test logs: Suite job http://cautotest/tko/retrieve_logs.cgi?job=/results/102544934-chromeos-test/ platform_DMVerityBitCorruption http://cautotest/tko/retrieve_logs.cgi?job=/results/102544934-chromeos-test/ go to the suite bucket: https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/102544934-chromeos-test/ the "hostless" autoserv.DEBUG has this: 02/21 05:48:29.934 INFO | server_job:0184| INFO ---- Start bvt-inline timestamp=1487684909 localtime=Feb 21 05:48:29 02/21 05:48:29.934 DEBUG| suite:0854| Scheduling platform_DMVerityBitCorruption 02/21 05:48:30.273 DEBUG| suite:1206| Adding job keyval for platform_DMVerityBitCorruption=102545130-chromeos-test and this bucket: https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/102545130-chromeos-test/chromeos2-row8-rack1-host4/ is populated normally.
> platform_DMVerityBitCorruption ABORT: Timed out, did not run. This message means what it says. Nothing ran, so there are no logs; this can't realistically be fixed. The suite job logs might have something to say about what was so slow that the job never got scheduled, but it's way too late for that debug.
|► Sign in to add a comment|