New issue
Advanced search Search tips

Issue 854769 link

Starred by 1 user
Status: Assigned
Owner:
athilenius@chromium.org
Cc:
jclinton@chromium.org
chromeos-continuous-integration-fire-hose@google.com
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

Buildroots being heavily wiped.

Project Member Reported by dgarr...@chromium.org, Jun 20 2018 
While looking into why some builds appear to be taking a long time to start, I found that InitialSync is often taking up 45 minutes, and that the Clean step before it is seeing "Unknown layout: Wiping buildroot".

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8943179840228975120

This graph shows how often we see that error, which has suddenly spiked.

http://shortn/_TS64ELy8EC

PS: That error is only expected on brand new builders, or when we explicitly change the build directory layout, which hasn't happened.

Comment 1 by dgarr...@chromium.org, Jun 20 2018 

This problem suggests that something is wiping out the file:
  /b/c/cbuild/.cbuildbot_build_state.json

However, that file is a directory level up from cbuildbot's buildroot, and shouldn't be touched by anything other than cbuildbot_launch.

Comment 2 by dgarr...@chromium.org, Jun 20 2018

Cc: jclinton@chromium.org

Comment 3 by dgarr...@chromium.org, Jun 20 2018 

PS: This behavior could cause a huge spike in GoB usage.

Comment 4 by dgarr...@chromium.org, Jun 20 2018

Labels: -Pri-3 Pri-1

Comment 5 by athilenius@chromium.org, Jun 20 2018 

Is this related to  crbug.com/854278 ?

Comment 6 by dgarr...@chromium.org, Jun 20 2018 

I don't think it's related to candy provision problems. It seems super related to  https://crbug.com/854652 , but that the timing doesn't match up.

Comment 7 by ayatane@chromium.org, Jul 25

Components: -Infra>Client>ChromeOS>Test Infra>Client>ChromeOS>CI

Comment 8 by athilenius@chromium.org, Jul 30

Status: WontFix (was: Untriaged)
Is this still happening? I'm going to close it for now, please re-open if it's not just a side effect of  crbug.com/854652 .

Comment 9 by dgarr...@chromium.org, Jul 31 

It's still happening more than I would expect. Looks like 12 times an hour.

http://shortn/_Tzrks8JRWN


This is expected when a new builder is brought up (not in weeks), we update the build directory layout (hasn't happened in about a year), or the 'cbuild' named cache is wiped between builds.

Maybe our named cache is being wiped?

Each time it happens, the initial sync becomes very slow. ~ 30 minutes added to the build time.

Comment 10 by jclinton@chromium.org, Aug 1

Status: Assigned (was: WontFix)

Comment 11 by athilenius@chromium.org, Aug 1

Labels: -Pri-1 Pri-2
►

Sign in to add a comment