Fixed by revert/

gsutil seems running into emerge sandbox error.  Looking at Chrome's setup, it seems Chrome does not use the system depot_tools, instead Chrome has another copy of depot_tools in src/third_party/depot_tools.  That's likely how Chrome worked around the problem. 

BTW: I am not able to repro locally, so this must be some buildbot specialty.

https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos%2Feve-arcnext-chrome-pfq%2F257%2F%2B%2Frecipes%2Fsteps%2FBuildPackages%2F0%2Fstdout

chromeos-libassistant-0.0.119921-r1: Failed to fetch file gs://chromium-gn/c5665261bd0dba0898c50e609837f46c26d10083 for buildtools/linux64/gn. [Err:  * ACCESS DENIED:  mkdir:        /home/chrome-bot/depot_tools/external_bin/gsutil/gsutil_pyZ0rrv2
chromeos-libassistant-0.0.119921-r1:  * ISE:write_logfile: unable to append logfile: /var/log/sandbox/sandbox-10413.log
chromeos-libassistant-0.0.119921-r1:  * /var/tmp/portage/sys-apps/sandbox-2.11-r2/work/sandbox-2.11/libsandbox/libsandbox.c:check_syscall():990: failure (Bad file descriptor):
chromeos-libassistant-0.0.119921-r1:  * ISE: mkdir(/home/chrome-bot/depot_tools/external_bin/gsutil/gsutil_pyZ0rrv2)
chromeos-libassistant-0.0.119921-r1: 	abs_path: /home/chrome-bot/depot_tools/external_bin/gsutil/gsutil_pyZ0rrv2
chromeos-libassistant-0.0.119921-r1: 	res_path: /mnt/host/depot_tools/external_bin/gsutil/gsutil_pyZ0rrv2

guessing setting DEPOT_TOOLS_UPDATE=0 doesn't help

i think we addressed this in Chromium by adding a call to `ensure_bootstrap` in src/scripts/.  not entirely clear.

+khmel +malaykeshav

This has showed up on the last two builds:

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8947114785961456560
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8947099005671974448

This is a new failure, not the same as before.  I upreved the libassistant client headers yesterday but somehow the builder doesn't seem to see it.  Looking into what's going on.

Chrome HEAD still builds fine for me locally.  Does anyone know how the buildbot syncs the Chrome tree for PFQ builds?  Maybe it's syncing to a particular version that's bad.

re #5 - Sorry, i didn't notice this one doesnt say libassistant. I'll file a new bug for this.

re #6 - Im not quite sure.

Re #6: https://yaqs.googleplex.com/eng/q/6635888470130688

The tag can be seen in SyncChrome Step in builder.

re #7, The error still seems to be caused by my uprev of libassistant as I mentioned in #5.  https://chrome-internal-review.googlesource.com/c/chrome/src-internal/+/622019

The uprev is used by https://chromium-review.googlesource.com/c/chromium/src/+/1048121

It appears the PFQ picked up the public cl, but not the internal cl.  Both cls submitted a couple minutes apart and hours before the PFQ run.

Not sure why this happened. 

The yaqs link in #8 talked about the src but not src-internal.  I think we need to manually make 3424's src and src-internal in sync somehow (if buildbot does not do it automatically)

+some people who might know src-internal for chrome pfq

+mmoss for his input

Think the problem is that src-internal/DEPS change not visible to the created buildspec that is used by PFQ bots. Maybe we should trigger a new buildspec using http://chromepmo.appspot.com/trigger ?

https://chrome-internal.googlesource.com/chrome/tools/buildspec.git/+/ed3ee9a27447b092924d8e87ccb5137625bf6583/releases/68.0.3424.1/DEPS

I think this is the full DEPS for the branch build.  It does not contain the internal cl.  Should we manually edit it?

First of all, NEVER edit a buildspec. Ever.

Second, the issue is that the preferred way to get src-internal nowadays, and the way the buildspecs have been doing it for a while now, is to use the entry that is in src/DEPS, i.e.:

https://crrev.com/7396470c0bfbc3231f4c088ea7520d97332089e2/DEPS#1004

That link is the DEPS as it existed at the time of your src change in https://crrev.com/c/1048121. Note that it has not yet been rolled to include your src-internal change (i.e. 9ad6814d5f1e026662cac01f07248d671c48b4da). So the problem is, your src change should have waited until the src-internal change was autorolled into src/DEPS.

This has happened now, so a trunk build should be fine, but at the point the branch and buildspec was created (https://crrev.com/dc678d64d713b6267bc1fe7802510b53d29cbe0f/DEPS#1004), it hadn't happened yet.

The src-internal cl and src cl are depending on each other.  If the src-internal roll is merged first, the build would also break (maybe only for engineers not buildbots IIUC your comments in #14).

I assume what you are saying is merging src-internal first.  Then in the src CL, include both code change and the src/DEPS roll on src-internal.  Then the build server would always see a consistent view of the repo.

Is there a way for engineers to also sync src-internal based on src/DEPS?  I have src-internal in .gclient file and it seems to sync it to HEAD whenever I run gclient sync.  

Is there something similar in Chromium like CQ-DEPEND in ChromiumOS that would be honored by these various build tools? 

What's a good path to fix the buildspec in 3424?

To checkout src-internal from src/DEPS, 'checkout_src_internal' (the condition on that entry) needs to be True. Remove the "src-internal" entry from your .gclient and add this to your "src" entry:

    "custom_vars": {
      'checkout_src_internal': True,
    }

To fix branch 3424, you would need to cherry-pick your src-internal change to the branched src-internal/DEPS:
https://chrome-internal.googlesource.com/chrome/src-internal.git/+/chromium/3424/DEPS

The branch src/DEPS points to that src-internal branch (https://chromium.googlesource.com/chromium/src/+/refs/branch-heads/3424/DEPS#1005), so it will see the cherry-pick automatically. (At some point it will have an autoroller, just like trunk, so there would be a bit of a delay, but for now you don't have to wait (you also don't have any presubmit checks or try builds or such, so be extra careful to make sure the cherry-pick applies correctly).)

https://chrome-internal-review.googlesource.com/c/chrome/src-internal/+/623030
Cherry picked cleanly.

Should we update the developer instruction to the new way of checking out src-internal?
https://companydoc.corp.google.com/company/teams/chrome/linux_build_instructions.md?cl=head

Ugh, yes please. I didn't realize that had never been updated (note that the bots are already building with 'checkout_src_internal: True', so devs should probably be doing the same, otherwise they're not really building/testing the same things).

And by the bots, I mean things like chromium.chrome and the official.* bots. Obviously that didn't include the bot that broke here :(

 Issue 840899  has been merged into this issue.

Updated the chrome build instruction doc.