While caroline has been failing for a while. Latest 2 builds are showing paygen failures.


Feb 14 02:06	??	failure	#405	Failed steps failed cbuildbot [caroline-release] failed paygentestdev
Feb 13 18:06	??	failure	#404	Failed steps failed cbuildbot [caroline-release] failed paygentestcanary failed paygentestdev

#404
PaygenTestCanary
https://luci-logdog.appspot.com/v/?s=chromeos%2Fbb%2Fchromeos%2Fcaroline-release%2F404%2F%2B%2Frecipes%2Fsteps%2FPaygenTestCanary%2F0%2Fstdout


host: chromeos2-row8-rack1-host4, status: Ready, locked: False diagnosis: Working
labels: ['board:caroline', 'bluetooth', 'accel:cros-ec', 'arc', 'hw_video_acc_enc_h264', 'os:cros', 'hw_jpeg_acc_dec', 'power:battery', 'ec:cros', 'hw_video_acc_h264', 'servo', 'hw_video_acc_vp8', 'cts_abi_x86', 'cts_abi_arm', 'storage:mmc', 'webcam', 'caroline', 'internal_display', 'phase:DVT', 'touchpad', 'touchscreen', 'variant:caroline', 'sku:caroline_intel_skylake_core_m3_4Gb', 'pool:bvt', 'audio_loopback_dongle', 'fwrw-version:caroline-firmware/R49-7820.263.0', 'fwro-version:caroline-firmware/R49-7820.263.0']
Last 10 jobs within 3:18:00:
101521143 caroline-release/R58-9282.0.0/paygen_au_canary/autoupdate_EndToEndTest_paygen_au_canary_full_9282.0.0 started on: 2017-02-14 00:14:00 status Failed
59977414 Reset started on: 2017-02-14 00:12:42 status PASS
59977398 Cleanup started on: 2017-02-14 00:10:27 status PASS
101513535 caroline-release/R58-9282.0.0/paygen_au_canary/autoupdate_EndToEndTest_paygen_au_canary_full_9282.0.0 started on: 2017-02-13 23:02:17 status Aborted
59976533 Reset started on: 2017-02-13 23:00:06 status PASS
host: chromeos2-row8-rack1-host5, status: Running, locked: False diagnosis: Working
labels: ['board:caroline', 'bluetooth', 'accel:cros-ec', 'arc', 'hw_video_acc_enc_h264', 'os:cros', 'hw_jpeg_acc_dec', 'power:battery', 'ec:cros', 'hw_video_acc_h264', 'servo', 'hw_video_acc_vp8', 'cts_abi_x86', 'cts_abi_arm', 'storage:mmc', 'webcam', 'caroline', 'internal_display', 'pool:bvt', 'audio_loopback_dongle', 'sku:caroline_intel_skylake_core_m3_4Gb', 'phase:DVT', 'touchpad', 'touchscreen', 'variant:caroline']
Last 10 jobs within 3:18:00:
59977404 Reset started on: 2017-02-14 00:11:19 status PASS
101517387 caroline-release/R58-9282.0.0/paygen_au_dev/autoupdate_EndToEndTest_paygen_au_dev_full_9000.77.0 started on: 2017-02-13 23:35:13 status Completed
59976939 Reset started on: 2017-02-13 23:34:16 status PASS

host: chromeos2-row8-rack1-host13, status: Ready, locked: False diagnosis: Working
labels: ['board:caroline', 'bluetooth', 'accel:cros-ec', 'arc', 'hw_video_acc_enc_h264', 'os:cros', 'hw_jpeg_acc_dec', 'power:battery', 'ec:cros', 'hw_video_acc_h264', 'servo', 'hw_video_acc_vp8', 'cts_abi_x86', 'cts_abi_arm', 'storage:mmc', 'webcam', 'caroline', 'internal_display', 'phase:DVT', 'touchpad', 'touchscreen', 'variant:caroline', 'sku:caroline_intel_skylake_core_m3_4Gb', 'audio_loopback_dongle', 'fwrw-version:caroline-firmware/R49-7820.263.0', 'fwro-version:caroline-firmware/R49-7820.263.0', 'pool:bvt']
Last 10 jobs within 3:18:00:
101517390 caroline-release/R58-9282.0.0/paygen_au_dev/autoupdate_EndToEndTest_paygen_au_dev_delta_9000.77.0 started on: 2017-02-14 00:21:08 status Completed
59977511 Reset started on: 2017-02-14 00:19:46 status PASS
59977265 Repair started on: 2017-02-13 23:56:22 status PASS
59977227 Reset started on: 2017-02-13 23:53:01 status FAIL
101513545 caroline-release/R58-9282.0.0/paygen_au_canary/autoupdate_EndToEndTest_paygen_au_canary_delta_9280.0.0 started on: 2017-02-13 23:01:56 status Failed
59976535 Reset started on: 2017-02-13 23:00:07 status PASS

au_canary_delta and au_canary_full failed, but it seems chromeos2-row8-rack1-host4 was considered failure enough:
autoupdate_EndToEndTest.paygen_au_canary_full    [ FAILED ]
autoupdate_EndToEndTest.paygen_au_canary_full      ABORT: Failed to perform stateful update on chromeos2-row8-rack1-host4

I'm not sure why au_canary_delta failure doesn't enough to trigger a failure though. Here's the logs from that run:

https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/101189056-chromeos-test/chromeos2-row8-rack1-host4/debug/ 

In the logs I just see an exception saying something failed, but I don't see any indication as to the reasoning. There were these in the logs:

02/14 00:17:57.388 ERROR|        dev_server:0427| Devserver call failed: "http://100.115.245.200:8082/check_health?", timeout: 2.0 seconds, Error: retry exception (label="get_load"), timeout = 2s
02/14 00:18:48.753 ERROR|        base_utils:0280| [stderr] mux_client_request_session: read from master failed: Broken pipe
02/14 00:18:59.684 ERROR|        dev_server:0427| Devserver call failed: "http://100.115.245.198:46233/check_health?", timeout: 6.0 seconds, Error: retry exception (label="get_load"), timeout = 6s
02/14 00:19:06.692 ERROR|        dev_server:0427| Devserver call failed: "http://100.115.245.198:46233/check_health?", timeout: 6.0 seconds, Error: retry exception (label="get_load"), timeout = 6s
02/14 00:19:09.724 ERROR|        base_utils:0280| [stderr] [0214/001909:INFO:update_engine_client.cc(471)] Forcing an update by setting app_version to ForcedUpdate.
02/14 00:19:09.725 ERROR|        base_utils:0280| [stderr] [0214/001909:INFO:update_engine_client.cc(473)] Initiating update check and install.
02/14 00:19:09.763 ERROR|        base_utils:0280| [stderr] [0214/001909:INFO:update_engine_client.cc(502)] Waiting for update to complete.
02/14 00:25:37.223 ERROR|        base_utils:0280| [stderr] [0214/002537:INFO:update_engine_client.cc(224)] Update succeeded -- reboot needed.
02/14 00:25:37.648 ERROR|        base_utils:0280| [stderr] [0214/002537:INFO:update_engine_client.cc(493)] Querying Update Engine status...
02/14 00:28:20.609 ERROR|        base_utils:0280| [stderr] mux_client_request_session: read from master failed: Broken pipe
02/14 00:29:55.138 ERROR|        dev_server:0427| Devserver call failed: "http://100.115.245.198:36384/check_health?", timeout: 6.0 seconds, Error: retry exception (label="get_load"), timeout = 6s
02/14 00:29:59.818 ERROR|        base_utils:0280| [stderr] [0214/002959:INFO:update_engine_client.cc(473)] Initiating update check and install.
02/14 00:45:19.645 ERROR|           control:0165| Received test error: autoupdate_EndToEndTest.paygen_au_canary_full failed


PaygenTestDev then subsequently fails.
https://luci-logdog.appspot.com/v/?s=chromeos%2Fbb%2Fchromeos%2Fcaroline-release%2F404%2F%2B%2Frecipes%2Fsteps%2FPaygenTestDev%2F0%2Fstdout 
This appears to be a timeout that a job just never completed and autotest server whacked it.

While this is the link to the suite (http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=101185230), I don't see a mention of delta or full in the logs even though the autotest server is listing jobs with that name.


#405 shows same build step error as above (PaygenTestDev), but no other failures beside that.
https://luci-logdog.appspot.com/v/?s=chromeos%2Fbb%2Fchromeos%2Fcaroline-release%2F405%2F%2B%2Frecipes%2Fsteps%2FPaygenTestDev%2F0%2Fstdout

No breadcrumbs but some timeout messages which I assume is a timeout being exceeded. This might be failing logs for that, but I can't really tell.
https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/101268740-chromeos-test/chromeos2-row8-rack1-host5/

02/14 10:44:59.448 ERROR|   logging_manager:0626| STATUS: INFO	----	----	Job aborted by autotest_system on 2017-02-14 10:44:04
02/14 10:44:59.448 ERROR|   logging_manager:0626| Unexpected indent regression, aborting
02/14 10:44:59.449 ERROR|   logging_manager:0626| 
02/14 10:44:59.449 ERROR|   logging_manager:0626| STATUS: END ABORT	autoupdate_EndToEndTest.paygen_au_dev_delta	autoupdate_EndToEndTest.paygen_au_dev_delta	None
02/14 10:44:59.450 ERROR|   logging_manager:0626| parsing test autoupdate_EndToEndTest.paygen_au_dev_delta autoupdate_EndToEndTest.paygen_au_dev_delta
02/14 10:44:59.450 ERROR|   logging_manager:0626| ADD: ABORT
02/14 10:44:59.450 ERROR|   logging_manager:0626| Subdir: autoupdate_EndToEndTest.paygen_au_dev_delta
02/14 10:44:59.451 ERROR|   logging_manager:0626| Testname: autoupdate_EndToEndTest.paygen_au_dev_delta
02/14 10:44:59.451 ERROR|   logging_manager:0626| None
02/14 10:44:59.452 ERROR|   logging_manager:0626| 
02/14 10:44:59.452 ERROR|   logging_manager:0626| STATUS: INFO	----	----	Job aborted by autotest_system on 2017-02-14 10:44:04
02/14 10:44:59.452 ERROR|   logging_manager:0626| parsing test ---- SERVER_JOB

Not sure if that's just a regular timeout messages or not.