CC'ing other sheriffs per https://uberchromegw.corp.google.com/i/chromium.linux/console

This appears to be a problem with the builder recipe or perhaps the filesystem particular buildbot instance.

The stability sheriff rotation is for triaging Chrome crashes, not for diagnosing build infrastructure issues.  Instructions for contacting the infrastructure troopers for various issues are here:

https://chromium.googlesource.com/infra/infra/+/master/doc/users/contacting_troopers.md

The functional error is:
/mnt/data/b/build_internal/scripts/slave/.recipe_deps/depot_tools/ninja -w dupbuild=err -C /mnt/data/b/build/slave/linux-chromeos-buildspec-tests/build/src/out/Release browser_tests chrome interactive_ui_tests -j50
ninja: Entering directory `/mnt/data/b/build/slave/linux-chromeos-buildspec-tests/build/src/out/Release'
ninja: fatal: chdir to '/mnt/data/b/build/slave/linux-chromeos-buildspec-tests/build/src/out/Release' - No such file or directory


This isn't an infrastructure failure. That directory legitimately doesn't exist. Can someone more familiar with the Chromium build system chime in here? What is supposed to create ".../src/out/Release"? "gclient runhooks"? "gn"? "ninja"? Whatever it is, it's not happening for R54 builds.

This is also neither a Goma nor a BuildBot error (see #4).

I don't know what "mb" is, but maybe similar to this: https://bugs.chromium.org/p/chromium/issues/detail?id=630934 ?

+dpranke

Dirk, you know these layers pretty well, any idea what might be happening?

Still failing. See https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/Linux%20ChromeOS%20Buildspec%20Tests/builds/831

I'm largely on vacation and haven't had any time to look at it.

It does sound like there's something misconfigured in the recipe.

+phadjan.jr@, who has been involved with other Chromium build issues.

Still happening. See:
- https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/Linux%20ChromeOS%20Buildspec%20Tests/builds/857
- https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/Linux%20ChromeOS%20Buildspec%20Tests/builds/858
- https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/Linux%20ChromeOS%20Buildspec%20Tests/builds/859

Please fix soon! Needed for browsertests run in lieu of manual testing on Beta/Stable release builds. Assigned to oncall trooper, per https://chromium-build.appspot.com/p/chromium/console

Started https://groups.google.com/a/google.com/d/msg/chrome-infrastructure-team/g6-IXcY1GBI/MakmPKfeEgAJ .

iannucci@ is no longer trooper. Actually marking as a trooper issue.

+stevenjb ...

So ... problem #1: This builder was never touched as part of the GYP->GN migration, so when we stopped running gyp_chromium by default, it probably broke.

Problem #2: If this is a buildspec-based builder that is building things on different branches, it can't handle flipping between GYP and GN.

The answer to both is to have the builder run MB and add the appropriate entries to //tools/mb/mb_config.pyl that get backported to the needed branches. 

I'll see if I can post some patches to do this, but clearly there's a problem in that no one is really claiming ownership of this builder.

+kerz ...

I also don't understand why this builder is on this waterfall at all if it's just doing a regular chromium build. 

Perhaps this should be doing an actual official build and be on one of the official.desktop waterfalls?

Or we should shut it off since clearly it's not the end of the world if it's broken for more than two weeks?

> So ... problem #1: This builder was never touched as part of the GYP->GN migration, so when we 
> stopped running gyp_chromium by default, it probably broke.

Sounds about right on the timeline.

> Problem #2: If this is a buildspec-based builder that is building things on different branches,
> it can't handle flipping between GYP and GN.

It clobbers in between builds for this and other similar  reasons.

> The answer to both is to have the builder run MB and add the appropriate entries to 
> //tools/mb/mb_config.pyl that get backported to the needed branches. 

This only tracks whatever branches ChromeOS is pinning. You can see which is currently pinned by looking in CrOS's release branch at the ebuild filename:

https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/release-R54-8743.B/chromeos-base/chromeos-chrome/

ATM it's "54.0.2840.26".

Any fixes to "src" would only need to target ToT, and whatever current M54 branch CrOS is pinning. Previous milestones work, so no backport necessary.

> I'll see if I can post some patches to do this, but clearly there's a problem in that no one is 
> really claiming ownership of this builder.

I'm not sure that's actually a problem. scunningham@ and I have been working on this builder for a while. I punted to phajden@ and you because, after identifying the nature of the problem, it seemed more reasonable to let someone who actually knows what changed weigh in. 

The fact that it took 12 days to get feedback is the result of vacation, I think, but this builder was definitely missed in the interim.

> I also don't understand why this builder is on this waterfall at all if it's just doing a
> regular chromium build. 

It's doing a Chrome-for-Linux-with-CrOS-flags-enabled build on specifically the milestones that CrOS pins.

> Perhaps this should be doing an actual official build and be on one of the official.desktop waterfalls?

The appeal of this builder is that it specifically targets CrOS milestone pins. If you think it's easier to have three builders building a superset of those pins on each milestone, that's probably fine. I don't see why that is more appealing than just fixing this builder, though.

> Or we should shut it off since clearly it's not the end of the world if it's broken for more than two weeks?

Fixing it is important, hence this P1 bug.

Per #14: This builder is used by the Chrome OS Test team to verify that certain browsertests are run and passing on each release candidate. Part of the CrOS release qualification criteria is that specific browertests pass on the RC. Without these tests, dozens of complicated, time-consuming manual tests would have to be run in their place on each RC, which would slow the release process. So, this builder is needed to maintain release velocity.

@scunningham - thanks for the info. It sounds like I was wrong about a few things, and I'm glad to hear that this is being actively used.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3fd8518b896e46d246be2e9cb8a64bc86063474d

commit 3fd8518b896e46d246be2e9cb8a64bc86063474d
Author: dpranke <dpranke@chromium.org>
Date: Fri Sep 16 17:50:55 2016

Add an MB entry for a missing CrOS builder.

R=stevenjb@chromium.org, phajdan.jr@chromium.org
BUG= 643238 

Review-Url: https://codereview.chromium.org/2344093002
Cr-Commit-Position: refs/heads/master@{#419210}

[modify] https://crrev.com/3fd8518b896e46d246be2e9cb8a64bc86063474d/tools/mb/mb_config.pyl

[Automated comment] There appears to be on-going work (i.e. bugroid changes), needs manual review.

[Automated comment] There appears to be on-going work (i.e. bugroid changes), needs manual review.

This change meets the bar and is approved for M54

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/19afede0b02be9794a154e24b23aa9a595e0ca77

commit 19afede0b02be9794a154e24b23aa9a595e0ca77
Author: Dirk Pranke <dpranke@chromium.org>
Date: Mon Sep 19 20:59:56 2016

Add an MB entry for a missing CrOS builder to M54.

R=phajdan.jr@chromium.org, stevenjb@chromium.org
BUG= 643238 

Review URL: https://codereview.chromium.org/2348763002 .

Cr-Commit-Position: refs/branch-heads/2840@{#418}
Cr-Branched-From: 1ae106dbab4bddd85132d5b75c670794311f4c57-refs/heads/master@{#414607}

[modify] https://crrev.com/19afede0b02be9794a154e24b23aa9a595e0ca77/tools/mb/mb_config.pyl

Still failing: https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/Linux%20ChromeOS%20Buildspec%20Tests/builds/872

Hmm. Do I need to merge this onto some other branch? Or does the buildspec need to roll forward on the branch to pick it up?

reassigning; feel free to punt back to me if/when there's something I can do.

Compile still failing: https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/Linux%20ChromeOS%20Buildspec%20Tests/builds/883

Sorry, Dirk; I don't know how to fix builders, nor to whom this should be assigned. Is there a new trooper on call this week?

This is not the sort of issue a trooper knows how to fix. The infrastructure is working, and is building what it's supposed to build. The problem here is that the thing that is being built is broken and needs to be fixed by someone familiar with the build system.

Argh, apparently I used the name of the tester, not the name of the builder. Fixing ...

i.e., this is actually the relevant failure:

https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/Chrome4CROS%20Packages/builds/2015/steps/generate_build_files/logs/stdio

Okay, I landed a change that will hopefully get picked up in the next roll forward of the buildspecs ...

https://chromium.googlesource.com/chromium/src/+/ebf277c839e039b0ede30124b6fc8b9ee2f5ffc1

If this doesn't work, please let me know.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9703ad9c86932bbd61ba874445d494d52ff8eb70

commit 9703ad9c86932bbd61ba874445d494d52ff8eb70
Author: dpranke <dpranke@chromium.org>
Date: Tue Sep 27 23:54:39 2016

Fix a CrOS buildername in mb_config.pyl.

This is cherry-picking https://codereview.chromium.org/2374843002/
back to the main branch.

TBR=dnj@chromium.org
BUG= 643238 

Review-Url: https://codereview.chromium.org/2372003005
Cr-Commit-Position: refs/heads/master@{#421389}

[modify] https://crrev.com/9703ad9c86932bbd61ba874445d494d52ff8eb70/tools/mb/mb_config.pyl

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ebf277c839e039b0ede30124b6fc8b9ee2f5ffc1

commit ebf277c839e039b0ede30124b6fc8b9ee2f5ffc1
Author: Dirk Pranke <dpranke@chromium.org>
Date: Tue Sep 27 21:51:34 2016

Fix the builder name in mb_config.pyl.

TBR=dnj@chromium.org
BUG= 643238 

Review URL: https://codereview.chromium.org/2374843002 .

Cr-Commit-Position: refs/branch-heads/2840@{#556}
Cr-Branched-From: 1ae106dbab4bddd85132d5b75c670794311f4c57-refs/heads/master@{#414607}

[modify] https://crrev.com/ebf277c839e039b0ede30124b6fc8b9ee2f5ffc1/tools/mb/mb_config.pyl

Well, we got closer:

https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/Chrome4CROS%20Packages/builds/2020/steps/generate_build_files/logs/stdio

It looks like dominicc@ landed a change on the branch deleting that file, but forgot to  remove the entry from core.gypi, so we're still trying to build it.

Reassigning to dominicc@ .

 Issue 651558  has been merged into this issue.

Still failing: https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/Linux%20ChromeOS%20Buildspec%20Tests/builds/906

Still failing: 
- https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/Linux%20ChromeOS%20Buildspec%20Tests/builds/911
- https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/Linux%20ChromeOS%20Buildspec%20Tests/builds/908

The lack of browsertest coverage on release candidate builds increases risk for Beta and Stable pushes. Please get these builds working asap.

Oh, that's confusing. I guess there are two different builders, both of which have issues.

On "Chrome4CROS Packages", on M54, the current problem is that dominicc@'s change was merged back from trunk incorrectly, so `generate_build_files` is failing.

On M55/trunk, the compile is failing because the "linux_packages_all" is not correctly defined in the GN files (and isn't being pulled in in this configuration), I think.

If dominicc@'s change is fixed, I expect M54 will hit the same problem with linux_packages_all.

*Separately*, "Linux ChromeOS Buildspec Tests" also needs the MB entry (that I added initially and then reverted), and a recipe-side change to enable MB.

I can try to look at some of these things today.

Update: I've posted CLs for trunk/M54/M53 that I *think* will fix the current set of issues. I'm waiting for the trunk CL to land (and stick) before I merge back the others.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b0a20f37160c224e1a27f92d859879f56a500eda

commit b0a20f37160c224e1a27f92d859879f56a500eda
Author: dpranke <dpranke@chromium.org>
Date: Thu Oct 06 23:40:45 2016

Add MB config and installer target for an official CrOS builder.

This adds an MB config for 'Linux ChromeOS Buildspec Tests' builder, and makes the 'linux_packages_all' target be defined for CrOS builds as well as desktop Linux, though I have no idea if this'll actually build in that config.

TBR=dnj@chromium.org, brettw@chromium.org
BUG= 643238 

Review-Url: https://codereview.chromium.org/2399883003
Cr-Commit-Position: refs/heads/master@{#423739}

[modify] https://crrev.com/b0a20f37160c224e1a27f92d859879f56a500eda/chrome/installer/BUILD.gn
[modify] https://crrev.com/b0a20f37160c224e1a27f92d859879f56a500eda/tools/mb/mb_config.pyl

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/56f3211e09709239d185e4846332a0e1632e89ed

commit 56f3211e09709239d185e4846332a0e1632e89ed
Author: Dirk Pranke <dpranke@chromium.org>
Date: Fri Oct 07 21:33:53 2016

Merge MB, installer build changes for official CrOS builders to M53.

This backports the changes needed for the 'Chrome4CROS Packages'
and 'Linux ChromeOS Buildspec Tests' builders back to M53 (branch 2785).

R=brettw@chromium.org
TBR=brettw@chromium.org, dnj@chromium.org
BUG= 643238 

Review URL: https://codereview.chromium.org/2387333007 .

Cr-Commit-Position: refs/branch-heads/2785@{#948}
Cr-Branched-From: 68623971be0cfc492a2cb0427d7f478e7b214c24-refs/heads/master@{#403382}

[modify] https://crrev.com/56f3211e09709239d185e4846332a0e1632e89ed/chrome/installer/BUILD.gn
[modify] https://crrev.com/56f3211e09709239d185e4846332a0e1632e89ed/tools/mb/mb_config.pyl

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7f2dd2e161fcec3ba14173402c71dd8f0c1d5a96

commit 7f2dd2e161fcec3ba14173402c71dd8f0c1d5a96
Author: Dirk Pranke <dpranke@chromium.org>
Date: Fri Oct 07 22:03:16 2016

Merge fixes for chromeos.chrome builders back to M54.

R=brettw@chromium.org
TBR=brettw@chromium.org, dnj@chromium.org
BUG= 643238 

Review URL: https://codereview.chromium.org/2400013002 .

Cr-Commit-Position: refs/branch-heads/2840@{#688}
Cr-Branched-From: 1ae106dbab4bddd85132d5b75c670794311f4c57-refs/heads/master@{#414607}

[modify] https://crrev.com/7f2dd2e161fcec3ba14173402c71dd8f0c1d5a96/chrome/installer/BUILD.gn
[modify] https://crrev.com/7f2dd2e161fcec3ba14173402c71dd8f0c1d5a96/tools/mb/mb_config.pyl

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/25ebc753012ec7ea27bc6f2f6442e527947635ea

commit 25ebc753012ec7ea27bc6f2f6442e527947635ea
Author: dpranke <dpranke@google.com>
Date: Fri Oct 07 22:32:30 2016

Okay, all of the changes I was planning to land have landed. Let's see if things green up.

Builds are succeeding! Thank you for your persistence, Dirk.

Now, I need to deal with the backlog of failing/flaky Browsertests that have accumulated in M54 over the last few weeks :-(

Marking this bug as fixed. Entered bug 654561 for failing browsertests. 

I'm glad things are working now, and I'm sorry for the delay.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b0a20f37160c224e1a27f92d859879f56a500eda

commit b0a20f37160c224e1a27f92d859879f56a500eda
Author: dpranke <dpranke@chromium.org>
Date: Thu Oct 06 23:40:45 2016

Add MB config and installer target for an official CrOS builder.

This adds an MB config for 'Linux ChromeOS Buildspec Tests' builder, and makes the 'linux_packages_all' target be defined for CrOS builds as well as desktop Linux, though I have no idea if this'll actually build in that config.

TBR=dnj@chromium.org, brettw@chromium.org
BUG= 643238 

Review-Url: https://codereview.chromium.org/2399883003
Cr-Commit-Position: refs/heads/master@{#423739}

[modify] https://crrev.com/b0a20f37160c224e1a27f92d859879f56a500eda/chrome/installer/BUILD.gn
[modify] https://crrev.com/b0a20f37160c224e1a27f92d859879f56a500eda/tools/mb/mb_config.pyl

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/19afede0b02be9794a154e24b23aa9a595e0ca77

commit 19afede0b02be9794a154e24b23aa9a595e0ca77
Author: Dirk Pranke <dpranke@chromium.org>
Date: Mon Sep 19 20:59:56 2016

Add an MB entry for a missing CrOS builder to M54.

R=phajdan.jr@chromium.org, stevenjb@chromium.org
BUG= 643238 

Review URL: https://codereview.chromium.org/2348763002 .

Cr-Commit-Position: refs/branch-heads/2840@{#418}
Cr-Branched-From: 1ae106dbab4bddd85132d5b75c670794311f4c57-refs/heads/master@{#414607}

[modify] https://crrev.com/19afede0b02be9794a154e24b23aa9a595e0ca77/tools/mb/mb_config.pyl

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ebf277c839e039b0ede30124b6fc8b9ee2f5ffc1

commit ebf277c839e039b0ede30124b6fc8b9ee2f5ffc1
Author: Dirk Pranke <dpranke@chromium.org>
Date: Tue Sep 27 21:51:34 2016

Fix the builder name in mb_config.pyl.

TBR=dnj@chromium.org
BUG= 643238 

Review URL: https://codereview.chromium.org/2374843002 .

Cr-Commit-Position: refs/branch-heads/2840@{#556}
Cr-Branched-From: 1ae106dbab4bddd85132d5b75c670794311f4c57-refs/heads/master@{#414607}

[modify] https://crrev.com/ebf277c839e039b0ede30124b6fc8b9ee2f5ffc1/tools/mb/mb_config.pyl

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7f2dd2e161fcec3ba14173402c71dd8f0c1d5a96

commit 7f2dd2e161fcec3ba14173402c71dd8f0c1d5a96
Author: Dirk Pranke <dpranke@chromium.org>
Date: Fri Oct 07 22:03:16 2016

Merge fixes for chromeos.chrome builders back to M54.

R=brettw@chromium.org
TBR=brettw@chromium.org, dnj@chromium.org
BUG= 643238 

Review URL: https://codereview.chromium.org/2400013002 .

Cr-Commit-Position: refs/branch-heads/2840@{#688}
Cr-Branched-From: 1ae106dbab4bddd85132d5b75c670794311f4c57-refs/heads/master@{#414607}

[modify] https://crrev.com/7f2dd2e161fcec3ba14173402c71dd8f0c1d5a96/chrome/installer/BUILD.gn
[modify] https://crrev.com/7f2dd2e161fcec3ba14173402c71dd8f0c1d5a96/tools/mb/mb_config.pyl

Builds have been successfully compiling since Oct 07. So, marking this verified.