New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 637307 link

Starred by 1 user

Issue metadata

Status: Verified
Owner:
Closed: Aug 2016
Cc:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

M52 branch pfq AFDOUpdate Ebuild stage failing

Project Member Reported by gkihumba@google.com, Aug 12 2016

Issue description

Link to build:
https://uberchromegw.corp.google.com/i/chromeos_release/builders/lumpy-pre-flight-branch%20release-R52-8350.B/builds/194

Link to logs:
https://uberchromegw.corp.google.com/i/chromeos_release/builders/lumpy-pre-flight-branch%20release-R52-8350.B/builds/194/steps/AFDOUpdateEbuild/logs/stdio

@@@BUILD_STEP@AFDOUpdateEbuild@@@
************************************************************
** Start Stage AFDOUpdateEbuild - Fri, 12 Aug 2016 00:00:27 -0700 (PDT)
** 
** Updates the Chrome ebuild with the names of the AFDO profiles.
************************************************************
00:00:33: INFO: RunCommand: /b/cbuild/shared_internal/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' ls -l -- gs://chromeos-prebuilt/afdo-job/canonicals/latest-chromeos-chrome-amd64-52.afdo
00:00:34: INFO: Found latest AFDO info file gs://chromeos-prebuilt/afdo-job/canonicals/latest-chromeos-chrome-amd64-52.afdo but it is too old
00:00:34: INFO: RunCommand: /b/cbuild/shared_internal/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' ls -l -- gs://chromeos-prebuilt/afdo-job/canonicals/latest-chromeos-chrome-amd64-51.afdo
00:00:35: INFO: Found latest AFDO info file gs://chromeos-prebuilt/afdo-job/canonicals/latest-chromeos-chrome-amd64-51.afdo but it is too old

@@@STEP_FAILURE@@@
00:00:35: ERROR: Could not find appropriate AFDO profile
00:00:35: INFO: Running cidb query on pid 32178, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x5530250>
00:00:35: INFO: Running cidb query on pid 32178, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x566d210>
************************************************************
** Finished Stage AFDOUpdateEbuild - Fri, 12 Aug 2016 00:00:35 -0700 (PDT)
************************************************************
00:00:35: INFO: Running cidb query on pid 32178, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x5532490>
00:00:35: INFO: Running cidb query on pid 32178, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f6c988a5d10>
00:00:35: INFO: Running cidb query on pid 32178, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x577f190>



 

Comment 1 by dshi@chromium.org, Aug 12 2016

Cc: takaoka@chromium.org davidri...@chromium.org vprupis@chromium.org
 gsutil ls -l -- gs://chromeos-prebuilt/afdo-job/canonicals/latest-chromeos-chrome-amd64-52.afdo
        46  2016-08-03T03:59:59Z  gs://chromeos-prebuilt/afdo-job/canonicals/latest-chromeos-chrome-amd64-52.afdo

It's now older than 7 days as checked here:
https://cs.corp.google.com/chromeos_public/chromite/cbuildbot/afdo.py?rcl=4f2c557fea7b96d91980d95ab9329825404c06a4&l=360

+sheriffs
Cc: llozano@chromium.org
I'm trying to wrap my head around what this is doing and how it works, but it seems like part of the issue is that the AFDO updates are stale because there haven't been any builds on those branches in almost a week:
https://uberchromegw.corp.google.com/i/chromeos_release/builders/lumpy-pre-flight-branch%20release-R52-8350.B

This combined with the interaction of the changes https://chromium-review.googlesource.com/281195 and https://chromium-review.googlesource.com/285542 which prevent updating the ebuild on the last successful build:
https://uberchromegw.corp.google.com/i/chromeos_release/builders/lumpy-pre-flight-branch%20release-R52-8350.B/builds/192/steps/AFDOUpdateEbuild/logs/stdio
12:01:50: INFO: RunCommand: git status --porcelain -- Manifest chromeos-chrome-52.0.2743.117_rc-r1.ebuild chromeos-chrome-9999.ebuild in /b/cbuild/shared_internal/src/third_party/chromiumos-overlay/chromeos-base/chromeos-chrome
12:01:52: INFO: AFDO info for the Chrome ebuild did not change. Nothing to commit

+llozano@ who implemented the changes to not rev the ebuild.
Project Member

Comment 3 by sheriffbot@chromium.org, Aug 13 2016

Labels: Hotlist-Google
I will take a look at this but why is this a P1? isn't this branch inactive?

Comment 5 by gkihumba@google.com, Aug 19 2016

Comment 4: We were still building off of it as late as last week. We may have one more stable release so it would be good to have the pfq up and running.

Comment 6 by gkihumba@google.com, Aug 19 2016

Cc: xixuan@chromium.org dgarr...@chromium.org
Xixuan/ Don will you please look into this? Thanks

Comment 7 by xixuan@chromium.org, Aug 19 2016

Is 'afdo file is not updated in 7 days' expected? 

If so, release the restriction. If not, the problem becomes to why afdo file is not updated as scheduled.
What builders generate the AFDO profiles?

The one this builder is checking hasn't been updated since the 3rd.


gsutil -o 'Boto:num_retries=10' ls -l -- gs://chromeos-prebuilt/afdo-job/canonicals/latest-chromeos-chrome-amd64-52.afdo
        46  2016-08-03T03:59:59Z  gs://chromeos-prebuilt/afdo-job/canonicals/latest-chromeos-chrome-amd64-52.afdo

Owner: llozano@chromium.org
I will find a workaround for this.
I have a workaround in here for review. https://chromium-review.googlesource.com/#/c/373683/1
The real change is a little bit more complicated. I will need to test before I can put it on ToT.
Project Member

Comment 11 by bugdroid1@chromium.org, Aug 22 2016

Labels: merge-merged-release-R52-8350.B
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/27873487e9b14f56eb7ea9ccdd3e3a5021c53fa3

commit 27873487e9b14f56eb7ea9ccdd3e3a5021c53fa3
Author: Luis Lozano <llozano@chromium.org>
Date: Mon Aug 22 08:35:53 2016

AFDO: increase allowed stale for profile on inactive branch.

R52 is mostly inactive. Chrome is not being uprevved anymore.
But the AFDO process is trying to check if the AFDO profile is
not stale and it is failing because it has not been updated in
the last 7 days. To workaround this, increase the stale limit
to 35 days which should be more than enough.

BUG= chromium:637307 
TEST=none

Change-Id: I3212eaf89c1328113f751d243fae3aece6ccf869
Reviewed-on: https://chromium-review.googlesource.com/373683
Commit-Queue: Luis Lozano <llozano@chromium.org>
Tested-by: Luis Lozano <llozano@chromium.org>
Trybot-Ready: Luis Lozano <llozano@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/27873487e9b14f56eb7ea9ccdd3e3a5021c53fa3/cbuildbot/afdo.py

Status: Fixed (was: Assigned)
Pfq is green, marking this as fixed. Thanks!
Labels: VerifyIn-54
bulk verified
Status: Verified (was: Fixed)

Sign in to add a comment