Project: chromium Issues People Development process History Sign in
New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 2 users
Status: WontFix
Owner:
Closed: Jun 21
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment
autoserv logs missing for some timed-out jobs
Project Member Reported by semenzato@chromium.org, Feb 21 2017 Back to list
No logs are collected for some jobs, making it impossible to do any further debugging.  For instance:

https://uberchromegw.corp.google.com/i/chromeos/builders/expresso-release/builds/1038

https://uberchromegw.corp.google.com/i/chromeos/builders/expresso-release/builds/1038/steps/AUTest%20%5Bau%5D/logs/stdio

Suite job                           [ PASSED ]
platform_Powerwash                  [ FAILED ]
platform_Powerwash                    ABORT: Timed out, did not run.
autoupdate_EndToEndTest.npo_delta   [ PASSED ]
autoupdate_Rollback                 [ PASSED ]

Suite timings:
Downloads started at 2017-02-21 05:33:23
Payload downloads ended at 2017-02-21 05:33:26
Suite started at 2017-02-21 05:34:06
Artifact downloads ended (at latest) at 2017-02-21 05:34:18
Testing started at 2017-02-21 05:37:50
Testing ended at 2017-02-21 05:54:41


Links to test logs:
Suite job http://cautotest/tko/retrieve_logs.cgi?job=/results/102542367-chromeos-test/
platform_Powerwash http://cautotest/tko/retrieve_logs.cgi?job=/results/102542367-chromeos-test/
autoupdate_EndToEndTest.npo_delta http://cautotest/tko/retrieve_logs.cgi?job=/results/102542579-chromeos-test/
autoupdate_Rollback http://cautotest/tko/retrieve_logs.cgi?job=/results/102542583-chromeos-test/

(Also, I don't find it very useful to have the platform_Powerwash URL point to the suite job URL)

The suite job points to the autoserv.DEBUG here:

https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/102542367-chromeos-test/hostless/debug/

which shows the job getting scheduled and assigned a job ID:

02/21 05:34:27.899 DEBUG|             suite:0854| Scheduling autoupdate_EndToEndTest_npo_delta_9304.0.0
02/21 05:34:28.339 DEBUG|             suite:1206| Adding job keyval for autoupdate_EndToEndTest_npo_delta_9304.0.0=102542579-chromeos-test
02/21 05:34:28.340 DEBUG|             suite:0854| Scheduling autoupdate_Rollback
02/21 05:34:28.745 DEBUG|             suite:1206| Adding job keyval for autoupdate_Rollback=102542583-chromeos-test
02/21 05:34:28.746 DEBUG|             suite:0854| Scheduling platform_Powerwash
02/21 05:34:29.169 DEBUG|             suite:1206| Adding job keyval for platform_Powerwash=102542586-chromeos-test
02/21 05:34:29.169 DEBUG|             suite:0927| Scheduled 3 tests, writing the total to keyval.
02/21 05:34:29.170 DEBUG|     dynamic_suite:0593| Waiting on suite.

However, the bucket for that job ID is empty, and we hit a wall.



 
A related problem (maybe it should be a separate bug?):

Some jobs which are linked to the same bucket as the "suite job", but the job ID in autoserv.DEBUG points to a valid bucket.  For instance:

https://uberchromegw.corp.google.com/i/chromeos/builders/caroline-release/builds/426

https://uberchromegw.corp.google.com/i/chromeos/builders/caroline-release/builds/426/steps/HWTest%20%5Bbvt-inline%5D/logs/stdio

snippets:

Suite job                           [ PASSED ]
  platform_DMVerityBitCorruption      [ FAILED ]
  platform_DMVerityBitCorruption        ABORT: Timed out, did not run.

...

Links to test logs:
  Suite job http://cautotest/tko/retrieve_logs.cgi?job=/results/102544934-chromeos-test/
  platform_DMVerityBitCorruption http://cautotest/tko/retrieve_logs.cgi?job=/results/102544934-chromeos-test/

go to the suite bucket:

https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/102544934-chromeos-test/

the "hostless" autoserv.DEBUG has this:

02/21 05:48:29.934 INFO |        server_job:0184| INFO	----	Start bvt-inline	timestamp=1487684909	localtime=Feb 21 05:48:29	
02/21 05:48:29.934 DEBUG|             suite:0854| Scheduling platform_DMVerityBitCorruption
02/21 05:48:30.273 DEBUG|             suite:1206| Adding job keyval for platform_DMVerityBitCorruption=102545130-chromeos-test

and this bucket:

https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/102545130-chromeos-test/chromeos2-row8-rack1-host4/

is populated normally.



Status: WontFix
>   platform_DMVerityBitCorruption        ABORT: Timed out, did not run.

This message means what it says.  Nothing ran, so there are no
logs; this can't realistically be fixed.  The suite job logs might
have something to say about what was so slow that the job never
got scheduled, but it's way too late for that debug.

Sign in to add a comment