uprev chrome to 59.0.3042.0 |
||||
Issue descriptionWe have not upreved in a few days and 59.0.3043.0 was the last revision that was affected by infra issues only (newer Chrome revisions have actual failures).
,
Mar 21 2017
I am hoping to get an uprev today, is there a specific reason we need to update to 59.0.3042.0?
,
Mar 21 2017
This is the candidate build for 59.0.3042.0: https://uberchromegw.corp.google.com/i/chromeos/builders/master-chromium-pfq/builds/4105 It has *5* failures. I am looking at those now.
,
Mar 21 2017
x86-generic and amd64-generic failed in VMTest, do we understand what those failures were? The others do appear to be provisioning related failures.
,
Mar 21 2017
The VMTest logs for x86-generic and amd64-generic both just look like this: bin/ctest --board=x86-generic --type=vm --no_graphics --verbose --target_image=/b/cbuild/internal_master/src/build/images/x86-generic/latest-cbuildbot/chromiumos_test_image.bin --test_results_root=/b/cbuild/internal_master/chroot/tmp/cbuildbotinsU4f/test_harness --only_verify --suite=smoke --ssh_private_key=/b/cbuild/internal_master/src/build/images/x86-generic/latest-cbuildbot/id_rsa exited with code 1 Does anyone know how to interpret that?
,
Mar 21 2017
This is the actual output
Starting a KVM instance
INFO : Launching: /b/cbuild/internal_master/chroot/usr/bin/qemu-system-x86_64 -enable-kvm -m 2G -smp 4 -vga cirrus -pidfile /tmp/kvm.9228 -chardev pipe,id=control_pipe,path=/tmp/kvm.9228.monitor -serial file:/tmp/kvm.9228.serial -mon chardev=control_pipe -daemonize -net nic,model=virtio -display none -net user,hostfwd=tcp:127.0.0.1:9228-:22 -drive file=/tmp/cbuildbot-tmpW151x5/chromiumos_qemu_disk.bin.cJtjBm,index=0,media=disk,cache=unsafe
WARNING: Image format was not specified for '/tmp/cbuildbot-tmpW151x5/chromiumos_qemu_disk.bin.cJtjBm' and probing guessed raw.
Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
Specify the 'raw' format explicitly to remove the restrictions.
INFO : KVM started with pid stored in /tmp/kvm.9228
INFO : Serial output, if available, can be found here in /tmp/kvm.9228.serial
Could not initiate first contact with remote host
Connection timed out during banner exchange
Connection timed out during banner exchange
Failed to connect to virtual machine, retrying ...
,
Mar 21 2017
,
Mar 21 2017
Yeah, so if you look here, we had a bunch of failures like this last week: https://build.chromium.org/p/chromiumos.chromium/builders/x86-generic-tot-chromium-pfq-informational?numbuilds=200 Which appears to have since been fixed. So 3042 is genuinely bad; we definitely do not want to uprev it. The current known issues have been addressed in the 3047 branch, so the next PFQ run (whcih should start in ~20 minutes) will hopefully succeed (minus infra flake). Let's wait for that.
,
Mar 21 2017
haven't done this before, but it looks like what's outlined here <http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium-os#TOC-ssh-failed>, so the next steps according to the "recipe" is to Download the disk and memory images, and then resume the VM using kvm on your workstation. $ tar --use-compress-program=pbzip2 -xf \ failed_SimpleTestUpdateAndVerify_1_update_chromiumos_qemu_disk.bin.8Fet3d.tar $ tar --use-compress-program=pbzip2 -xf \ failed_SimpleTestUpdateAndVerify_1_update_chromiumos_qemu_mem.bin.TgS3dn.tar $ cros_start_vm \ --image_path=chromiumos_qemu_disk.bin.8Fet3d \ --mem_path=chromiumos_qemu_mem.bin.TgS3dn You should now have a VM which has resumed at exactly the point where the test framework determined that it could not connect. Note that, at this time, we don't have an easy way to mount the VM filesystem, without booting it. If you're interested in improving that, please see crbug.com/313484.) For more information about troubleshooting VMs, see how to run Chrome OS image under VMs.
,
Mar 21 2017
iirc, the vm failures are issue 701693 (manifested as broken ssh).
,
Mar 21 2017
I diffed a bunch of builds. - vmtest for peppy and cyan passed. - We are not running generic on any of the other waterfalls (cq, android pfq, toolchain etc.) so no damage to other teams. - Next build doesn't have the generic failures anymore, but a new problem (simple chrome). Overall it may not be worth the effort to investigate this any further just to go up from 3041 to 2042. I am fine with WontFix. That said
,
Mar 21 2017
Ok. If comment 10 is right then this fixed the vmtest https://chromium-review.googlesource.com/#/c/455349/ Which indeed would explain why the next build is clean https://uberchromegw.corp.google.com/i/chromeos/builders/master-chromium-pfq/builds/4106 https://luci-milo.appspot.com/buildbot/chromeos/amd64-generic-chromium-pfq/9846 By that theory it is safe to uprev to 3042.
,
Mar 21 2017
While I think it is safe to uprev the bang for the buck is low at this stage. Sorry for the waste of time. |
||||
►
Sign in to add a comment |
||||
Comment 1 by ihf@chromium.org
, Mar 21 2017