New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 703759 link

Starred by 1 user

Issue metadata

Status: WontFix
Owner:
Closed: Mar 2017
Cc:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 3
Type: Bug



Sign in to add a comment

uprev chrome to 59.0.3042.0

Project Member Reported by ihf@chromium.org, Mar 21 2017

Issue description

We have not upreved in a few days and 59.0.3043.0 was the last revision that was affected by infra issues only (newer Chrome revisions have actual failures).
 

Comment 1 by ihf@chromium.org, Mar 21 2017

Summary: uprev chrome to uprev chrome to 59.0.3042.0 (was: uprev chrome to 59.0.3043.0 )
I need to correct myself. Unfortunately 59.0.3042.0.
I am hoping to get an uprev today, is there a specific reason we need to update to 59.0.3042.0?

This is the candidate build for 59.0.3042.0:
https://uberchromegw.corp.google.com/i/chromeos/builders/master-chromium-pfq/builds/4105

It has *5* failures. I am looking at those now.

x86-generic and amd64-generic failed in VMTest, do we understand what those failures were?

The others do appear to be provisioning related failures.

Cc: achuith@chromium.org
The VMTest logs for x86-generic and amd64-generic both just look like this:

bin/ctest --board=x86-generic --type=vm --no_graphics --verbose --target_image=/b/cbuild/internal_master/src/build/images/x86-generic/latest-cbuildbot/chromiumos_test_image.bin --test_results_root=/b/cbuild/internal_master/chroot/tmp/cbuildbotinsU4f/test_harness --only_verify --suite=smoke --ssh_private_key=/b/cbuild/internal_master/src/build/images/x86-generic/latest-cbuildbot/id_rsa exited with code 1

Does anyone know how to interpret that?

Comment 6 by ihf@chromium.org, Mar 21 2017

This is the actual output

Starting a KVM instance
INFO    : Launching: /b/cbuild/internal_master/chroot/usr/bin/qemu-system-x86_64 -enable-kvm -m 2G -smp 4 -vga cirrus -pidfile /tmp/kvm.9228 -chardev pipe,id=control_pipe,path=/tmp/kvm.9228.monitor -serial file:/tmp/kvm.9228.serial -mon chardev=control_pipe -daemonize -net nic,model=virtio -display none -net user,hostfwd=tcp:127.0.0.1:9228-:22 -drive file=/tmp/cbuildbot-tmpW151x5/chromiumos_qemu_disk.bin.cJtjBm,index=0,media=disk,cache=unsafe
WARNING: Image format was not specified for '/tmp/cbuildbot-tmpW151x5/chromiumos_qemu_disk.bin.cJtjBm' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.
INFO    : KVM started with pid stored in /tmp/kvm.9228
INFO    : Serial output, if available, can be found here in /tmp/kvm.9228.serial
Could not initiate first contact with remote host
Connection timed out during banner exchange
Connection timed out during banner exchange
Failed to connect to virtual machine, retrying ... 

Comment 7 by ihf@chromium.org, Mar 21 2017

Summary: uprev chrome to 59.0.3042.0 (was: uprev chrome to uprev chrome to 59.0.3042.0)
Yeah, so if you look here, we had a bunch of failures like this last week:

https://build.chromium.org/p/chromiumos.chromium/builders/x86-generic-tot-chromium-pfq-informational?numbuilds=200

Which appears to have since been fixed.

So 3042 is genuinely bad; we definitely do not want to uprev it.

The current known issues have been addressed in the 3047 branch, so the next PFQ run (whcih should start in ~20 minutes) will hopefully succeed (minus infra flake). Let's wait for that.

Comment 9 by pbe...@chromium.org, Mar 21 2017

haven't done this before, but it looks like what's outlined here
<http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium-os#TOC-ssh-failed>,
so the next steps according to the "recipe" is to


Download the disk and memory images, and then resume the VM using kvm on
your workstation.
$ tar --use-compress-program=pbzip2 -xf \

failed_SimpleTestUpdateAndVerify_1_update_chromiumos_qemu_disk.bin.8Fet3d.tar

$ tar --use-compress-program=pbzip2 -xf \

failed_SimpleTestUpdateAndVerify_1_update_chromiumos_qemu_mem.bin.TgS3dn.tar

$ cros_start_vm \
    --image_path=chromiumos_qemu_disk.bin.8Fet3d \
    --mem_path=chromiumos_qemu_mem.bin.TgS3dn

You should now have a VM which has resumed at exactly the point where the
test framework determined that it could not connect.

Note that, at this time, we don't have an easy way to mount the VM
filesystem, without booting it. If you're interested in improving that,
please see crbug.com/313484.)

For more information about troubleshooting VMs, see how to run Chrome OS
image under VMs.
iirc, the vm failures are  issue 701693  (manifested as broken ssh).

Comment 11 by ihf@chromium.org, Mar 21 2017

I diffed a bunch of builds.
- vmtest for peppy and cyan passed.
- We are not running generic on any of the other waterfalls (cq, android pfq, toolchain etc.) so no damage to other teams.
- Next build doesn't have the generic failures anymore, but a new problem (simple chrome).

Overall it may not be worth the effort to investigate this any further just to go up from 3041 to 2042. I am fine with WontFix.

That said 

Comment 12 by ihf@chromium.org, Mar 21 2017

Ok. If comment 10 is right then this fixed the vmtest
https://chromium-review.googlesource.com/#/c/455349/

Which indeed would explain why the next build is clean
https://uberchromegw.corp.google.com/i/chromeos/builders/master-chromium-pfq/builds/4106
https://luci-milo.appspot.com/buildbot/chromeos/amd64-generic-chromium-pfq/9846

By that theory it is safe to uprev to 3042.

Comment 13 by ihf@chromium.org, Mar 21 2017

Status: WontFix (was: Started)
While I think it is safe to uprev the bang for the buck is low at this stage. Sorry for the waste of time.

Sign in to add a comment