New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 824956 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Apr 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

moblab-generic-vm-pre-cq provision test failing due to wrong "found build"

Project Member Reported by sha...@chromium.org, Mar 22 2018

Issue description

https://luci-milo.appspot.com/buildbot/chromiumos.tryserver/pre_cq/91251

15:05:53 INFO | autoserv| [stderr] Suite job   [ PASSED ]
15:05:53 INFO | autoserv| [stderr] provision   [ FAILED ]
15:05:53 INFO | autoserv| [stderr] provision     FAIL: Failed to update 192.168.231.100 to build R67-10509.0.0-b2405484; found build 10509.0.2018_03_22_1210 instead
15:05:53 INFO | autoserv| [stderr]
15:05:53 INFO | autoserv| [stderr] Suite timings:
15:05:53 INFO | autoserv| [stderr] Downloads started at 2018-03-22 12:54:58
15:05:53 INFO | autoserv| [stderr] Payload downloads ended at 2018-03-22 12:55:03
15:05:53 INFO | autoserv| [stderr] Suite started at 2018-03-22 12:55:11
15:05:53 INFO | autoserv| [stderr] Artifact downloads ended (at latest) at 2018-03-22 12:55:11
15:05:53 INFO | autoserv| [stderr] Testing started at 2018-03-22 12:55:17
15:05:53 INFO | autoserv| [stderr] Testing ended at 2018-03-22 13:03:55
15:05:53 INFO | autoserv| [stderr]
15:05:53 INFO | autoserv| [stderr]
15:05:53 INFO | autoserv| [stderr] Links to test logs:
15:05:53 INFO | autoserv| [stderr] Suite job http://localhost/tko/retrieve_logs.cgi?job=/results/1-moblab/
15:05:53 INFO | autoserv| [stderr] provision http://localhost/tko/retrieve_logs.cgi?job=/results/2-moblab/


15:05:53 INFO | autoserv| [stderr] provision     FAIL: Failed to update 192.168.231.100 to build R67-10509.0.0-b2405484; found build 10509.0.2018_03_22_1210 instead

So we expect to have a 
 

Comment 1 by sha...@chromium.org, Mar 22 2018

As I was saying, we expect to have a numbered version release #, instead we get a timestamp appended.

We have seen similar occurrences in the past (but no clear resolution path), eg. issue 601625.

Comment 2 by sha...@chromium.org, Mar 22 2018

Cc: yueherngl@chromium.org
Nevermind, it's not really similar to issue 601625.

I recall Yueherngl@ had an issue recently with a firmware builder he setup, let me take a closer look.
Cc: pprabhu@chromium.org dgarr...@chromium.org
This is caused by a bug between Autotest and some recent builder
changes, more or less.  I don't understand the mechanism for how
this got into the system, though.

Owner: dgarr...@chromium.org
Status: Assigned (was: Untriaged)
Don, can you take a look?
The logs from the test suite aren't adequate to show it, but basically,
this failure is a consequence of a code change that alters how builds
are named.  dgarrett@ will know which change is involved.

The reason moblab-generic-vm-pre-cq is failing is because there's a
_second_ code path that makes a broken check comparing "build installed"
with "build name".  The second broken check is similar to the first
broken check (the one we fixed), but it's only used by moblab.

What I can't explain is why we're only seeing this change _now_.
I suspect dgarrett@ can enlighten us.

For the record, the broken check is in (you guessed it) cros_host.py
(the root of all evil in the world not caused by a love of money).
Here's the relevant source:
    def _post_update_processing(self, updater, expected_kernel=None):
    # ...
        # Check that we've got the build we meant to install.
        if not updater.check_version_to_confirm_install():
            raise autoupdater.ChromiumOSError(
                'Failed to update %s to build %s; found build '
                '%s instead' % (self.hostname,
                                updater.update_version,
                                self.get_release_version()))

My current inclination is to outright disable the problem code
rather than sort through the immediate fix; the check is redundant
given the current verifier, which I expect _will_ work correctly.

Project Member

Comment 8 by bugdroid1@chromium.org, Mar 23 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/50b69931fdcaa8ed3c42ad4f6741c20df419e8df

commit 50b69931fdcaa8ed3c42ad4f6741c20df419e8df
Author: Richard Barnette <jrbarnette@chromium.org>
Date: Fri Mar 23 00:04:56 2018

[autotest] Delete a problematic check from machine_install().

In CrosHost.machine_install(), there's a sanity check that the
build installed on the DUT is the same as the build that was
requested.  The check operates by comparing strings in
/etc/lsb-release against portions of the URL that named the build;
that check requires various heuristics and special cases.

The check in question is only used by Moblab, not by Autotest in
the full lab.

A recent change to the way builds are named on the builders broke
the check in machine_install(), and now the moblab pre-CQ builder
is failing.  The check is somewhat redundant, so stop the bleeding
by deleting the check.

BUG= chromium:824956 
TEST=run this through the pre-cq

Change-Id: Iccf19779b5f7a4c7a4e8e233e95c8c683cf6e990
Reviewed-on: https://chromium-review.googlesource.com/976644
Tested-by: Richard Barnette <jrbarnette@chromium.org>
Trybot-Ready: Richard Barnette <jrbarnette@chromium.org>
Reviewed-by: Shuqian Zhao <shuqianz@chromium.org>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>

[modify] https://crrev.com/50b69931fdcaa8ed3c42ad4f6741c20df419e8df/server/hosts/cros_host.py

Owner: jrbarnette@chromium.org
Status: Fixed (was: Assigned)
I believe this is now fixed.

Sign in to add a comment