09977bd8-b3ea-45e2-b90e-0c763731c634 Builders failed on: - caroline-release: https://luci-milo.appspot.com/buildbot/chromeos/caroline-release/1176 - veyron_tiger-release: https://luci-milo.appspot.com/buildbot/chromeos/veyron_tiger-release/1633
Following the caroline-release URL: https://luci-milo.appspot.com/buildbot/chromeos/caroline-release/1176 and looking at the paygen logs, we arrive here: https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos%2Fcaroline-release%2F1176%2F%2B%2Frecipes%2Fsteps%2FPaygenTestDev%2F0%2Fstdout ..where we can see that all of the tests in the suite passed, but something failed afterward. 11-01-2017 [09:40:13] Start collecting test results and dump them to json. Suite job [ PASSED ] autoupdate_EndToEndTest.paygen_au_dev_full [ PASSED ] autoupdate_EndToEndTest.paygen_au_dev_delta [ PASSED ] autoupdate_EndToEndTest.paygen_au_dev_delta [ PASSED ] autoupdate_EndToEndTest.paygen_au_dev_full [ PASSED ] Suite timings: Downloads started at 2017-11-01 07:02:12 Payload downloads ended at 2017-11-01 07:02:30 Suite started at 2017-11-01 07:02:52 Artifact downloads ended (at latest) at 2017-11-01 07:04:27 Testing started at 2017-11-01 08:50:08 Testing ended at 2017-11-01 09:27:47 Looking through the logs, we see a failed reset: host: chromeos6-row1-rack23-host7, status: Repairing, locked: False diagnosis: Working labels: ['board:caroline', 'bluetooth', 'accel:cros-ec', 'arc', 'os:cros', 'power:battery', 'ec:cros', 'servo', 'cts_abi_x86', 'cts_abi_arm', 'storage:mmc', 'webcam', 'caroline', 'internal_display', 'pool:bvt', 'hw_jpeg_acc_dec', 'hw_video_acc_h264', 'hw_video_acc_enc_h264', 'sku:caroline_intel_skylake_core_m3_4Gb', 'variant:caroline', 'touchpad', 'touchscreen', 'stylus', 'phase:PVT', 'hw_video_acc_vp8', 'model:caroline', 'audio_loopback_dongle', '4k_video_h264', '4k_video_vp8'] Last 10 jobs within 3:18:00: 2312049 Reset started on: 2017-11-01 09:21:59 status FAIL 153479852 caroline-release/R64-10088.0.0/paygen_au_dev/autoupdate_EndToEndTest_paygen_au_dev_delta_10088.0.0 started on: 2017-11-01 08:50:05 status Completed 2311855 Reset started on: 2017-11-01 08:49:12 status PASS 1 I wasn't sure what to do with "2312049", so I ran dut-status for that host, hoping to find the job show up there: teravest@teravest:~/cros/src$ dut-status -d 72 chromeos6-row1-rack23-host7 hostname S last checked URL chromeos6-row1-rack23-host7 OK 2017-11-01 11:53:06 http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos6-row1-rack23-host7/2312940-reset/ This isn't the reset that we're looking for, but it does give us guidance to find the correct URL, which is: https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/hosts/chromeos6-row1-rack23-host7/2312049-reset And from there, we can go to autoserv.DEBUG, and find the underlying reset failure. https://00e9e64bac2741547cb26cf457c4cbb52a08e16dacb9e2cb21-apidata.googleusercontent.com/download/storage/v1/b/chromeos-autotest-results/o/hosts%2Fchromeos6-row1-rack23-host7%2F2312049-reset%2F20170111091924%2Fdebug%2Fautoserv.DEBUG?qk=AD5uMEvfDzEBvCIyJQirNjyVkQ7N7PA9tbDUmiWiKGkwwpRF9ntfiUEDL7gBve3WyUeVKpZh7usn0VpNViAlObvrXX4YrqAmyGsLoAQ-lS8jACUR21s6Tjm6G4dnG9iqaKl-JJJe3CdcX6GqlE7lmztfrPTZXA7nYkGkPgSmq9E__2lyV32_-25rxNALZSpT9SVoZ5RVB_i415OF48YEiMwkkVKWPBFMBUsX9tW8-8CAj8mBYotFVgmNgsGG-nsCRx7VsRHVCgqAnbvtLiLgOFqw5nm6HOn7ZcNboJSvUKmaAQAzeEbPDM-uE-zEnW0snPqqNqgJCSntZSnbWfbROnXLuVlNBUGM7ZCV-xV-jE3JPd1kX2ptYzBJCTrLBJ3gS6NeMNqwA1ppKcMmUS7UYh2wvW9rbOnOMQ1KLcijNVdjFjQAKQ035iIiFGHDX7-Asa9W9fO52wEMuOnQhfMqS1BWRAiYRnwWkQcGJpfHCjgj7IHoRg7xfPdqh0Hz3_AK-n1F49cRpfGUyedZ-xlzeWKGlER3le02GPDJ2opzaZLPtMleSIr2ti6EapCbQBjiBE_CUC79MDeB5TGJECbP60IGAZp38x5ipMvkRTN40snbn7vKVY0tTXNKe6wZfC-po6jZGG_qg4LkvM9g5RUMuKSEDIkKLJ7_XEm_ZsIXUlPeszlmmL8FNVGWE34o3lgHx29HsmrnSXJmYpPOPfIc7yYWHGlev_6T2UW77aiXQtFEDGFyy5R63IXVAFhG-QFNKka8uSX2VDZYeC1PUaILY4-ebF7fAP7qevxVhpP-3ZOgS6fyOUVevDNKX9Wl5-1Pj7sn5arPcxL5p2OuSPDa5IaumsDUdhxCvxAod8EofTebaPQpMfmI-Eg 11/01 09:22:26.308 ERROR| repair:0354| Failed: The most recent AU attempt on this DUT succeeded Traceback (most recent call last): File "/usr/local/autotest/client/common_lib/hosts/repair.py", line 351, in _verify_host self.verify(host) File "/usr/local/autotest/server/hosts/cros_repair.py", line 194, in verify 'Last AU on this DUT failed') AutoservVerifyError: Last AU on this DUT failed 11/01 09:22:26.312 INFO | server_job:0214| FAIL ---- verify.good_au timestamp=1509553346 localtime=Nov 01 09:22:26 Last AU on this DUT failed
What's really strange is that looking at the logs, it seems like a reset just before this succeeded. https://00e9e64bac4ae9f259c517eef899a26fdfa20e0516e006ec8c-apidata.googleusercontent.com/download/storage/v1/b/chromeos-autotest-results/o/hosts%2Fchromeos6-row1-rack23-host7%2F2311855-reset%2F20170111084909%2Fstatus.log?qk=AD5uMEsJZUBqu5PzikLcI4C-XM0tI5R2Rrx9iycR2WJT_9Yu8TnckmbwbV7BIX3RbnOyJbBuERu2WvgtiMqbgNHN9NNjyxIaeTfEHdW8FZu-1ToPKaIyvGxswv_55trb6fErp-eIuwzDcUbJingPX6-iCDxeLqRnjA0SFcP4s_RNtlEF0eDKESmefno1du5ZS1IBBzcscBB1-QZC9M6YDo8SXJGJsxuz3zmJIVHYq3F_cAOEzkjvxCaLjBQNKQBjGE_hIce3s1omBVsCnMX76fp5XSwRBHKHNfqtEj7MPXp9-a39jm8ZQpcYhaeFygddoeY9QTNgnEJLELbGOA3G96q6HfFFuQe5RZE-wM_qwZaBtmbHaL9JU3RIp4ytgiyla4DldqWbmGOMgztY9GaH4lZ6mI32zVHoeY0baV_h-_aLBp7_8FWILljZlw-tM9L3YHHDgm5Ljk2YW2Kdz6YhSx44HH8tWL65J5WEqw7X8eTWlEendOxWlwzZi3yS-wuqMQVeEn_DpkpgRYsraMBgZdBgYI3R3vUiUBsOuM2pf9I3Dj3oYOr0jKf9cJ7j2erlhP5lI80C3yAcFxWvV9VJVNEQELTrStmvNlX3N5OaNw10BJNXthgx67q3_24SoW7el40Bs7fIWYuQGmYA4oWijy6HgRx7_tsk65c5ousDjkGZSL7lppc07rRWlqUInfRWZ9UBpsLrQnnRcNMw2b--As5YP6RrXeH4V8aPN6n-fLlHV1Ou-Ceob0eXXcZw4mUI4irUQcX-SE9soqBtyHdFiW7VT4qi4ZjZdd61o-j9I-3_J50Sevz4qdp4VE8UtpVdldGQfOywWNJ9q2rV0BWitPLI5j_DiVh_mg GOOD ---- verify.good_au timestamp=1509551375 localtime=Nov 01 08:49:35 The closest recent provision job also seems to have worked (and was before that reset): https://00e9e64bac1b6cc8d6a00b9034f060b3066ea9268888b26d58-apidata.googleusercontent.com/download/storage/v1/b/chromeos-autotest-results/o/hosts%2Fchromeos6-row1-rack23-host7%2F2311054-provision%2F20170111065243%2Fstatus.log?qk=AD5uMEvwCr3PFQhq6jjjiq2V-rwrqK94kxLvgAgDx1L5arxWu5WDEsDoJSUxd7NkauH7kIEb1_24qaaT5YvQfAWAhbwetejQ_lIwzBMwbR2zt_ozlmqHjfDjIjLCHYNmnEBixq6URkIYBXs2JPatyvUZ1eI0_9azc8tGR7ZDAhW1-BdX_zqSzcDVKgdUJti5FWs7gYjeOIL6st1P8KXEBwAab8A9fCTKbNceTK5y5EccdgVndeq7LjAPPFjF_aIEt6IZhhT-8u3nT4rSUKSba32TlB-78xeEiYK-HfdmJ21PTPzLcME3kZGiT6lUH5WgKKYpcMQZ1EKdZOlIom_a2VQ4jJM6gLMeCix4jD5VZUBvMCYzhuG3-X_Vjjm78eSkp7keEKfsp6XTydEHuZNPbz7iY4kL-XRk11ovm05wORx_1fHasZWqTRycq0DrfFSJVKPP8hBUt4HAbbqu4pAaGmFLveObYfEH3RhL-6Aeoa6rzyahMEQoNA8PWvIG1brX-Ogky4NXxosTwsAsCq_K3oAQ40OZfAGQQ4oCJDFHpBqHppFjG-oHNg3TgwvaYI8rE7S5OJjhABoVWtm-RX3GggEy_gEFZNWtljyxNn_aXutzjgKwFzh2kcczAlfl3NlZaj_uf3VLaQo881FMkmEt6bP9z1kBboAaUO20ONdEKN4SjoLezBK9fTigQ4moNe_54aJGi3chyMkillRaygPpG27q5gb7IQ4cT73rGpydQRqd8BbgaBE8OYq9eqNuoiqzFJ9Uij3Lb8ON7dhOh_RfGl0nZBPmBNdTKVnO9oFIMvnJHmec5Fv2mme5BLAsy-IUDHv9EQW0NQ_cxh6Kq_h5YqpyVJUFpc2ZsA ayatane, Am I missing some provision jobs that happened here? Do those get rolled into other jobs?
Comment 1 by teravest@chromium.org
, Nov 1 2017Labels: OS-Chrome Pri-2
Owner: teravest@chromium.org
Summary: PaygenTestCanary/PaygenTestDev fail due to Reset failure (was: 09977bd8-b3ea-45e2-b90e-0c763731c634)