looks like a flake on the crash side, not the isolate side.  this doesn't block the build, and you can recover the symbols by manually running upload_symbols and pointing the --failed-list option at the uploaded failed list, and specify the breakpad tarball.  both should be in the gs bucket for the specific build.

I believe there is an IP address whitelist that needs to be updated to include:

104.154.112/24

We have 256 new GCE swarm bots in that IP range. If I'm right, then builds which happen to land on a builder with one of the new IPs will fail, but builds that land on a builder with an old IP will pass.

It's hard to identify which IP address range a builder is in from the name, only via looking up in the name in pantheon.

When we figure out where/how to update the IP whitelist, we should document that in go/cros-builder-address-whitelisting

I didn't look through all examples, but the failing build linked above is from the new IP address range while the passing build is not.

We started failing around 10840.0.0 entirely. 

It looks like these got into this state with some working around 10853.0.0 on 7/6.

Is this something we need to talk to the crash folks about then?

Normally I would expect no debug symbols to block the release itself, we want to be able to see crashes and such right? We can recover it in this case, but it sounds like it may be  painful to run upload_symbols by hand on every one of these boards, but that may be worth it in this case.

The IP whitelisting sounds like the issue, my comment is outdated. 

> Normally I would expect no debug symbols to block the release itself

TPMs are the ones who designed this policy because of random symbol upload flakes, and because it's trivial to recover.  you have to run the upload_symbols command once per failing bot.  we prob could even add a logging line so people can copy & paste the command.

> we want to be able to see crashes

you still get crashes, you just won't get symbols until the symbol files are uploaded.  and crashes don't happen until the images are pushed, so you've got time.

Something in this files needs updating:

https://chrome-internal.googlesource.com/infradata/config/+/master/configs/chrome-infra-auth/ip_whitelist.cfg

I just don't see what.

i don't think that's where the whitelist is maintained.  i think you want to file a b/ here:
  https://b.corp.google.com/issues/new?component=25229

That makes sense. Isolate was working, but I was looking at isolate whitelists.

See http://b/111548287

This should be fixed. Please ping if it keeps happening.

I updated our docs as well.

if you need help recovering the symbols for those builds rather than try to kick off another set, let us know

I think this particular build is dead for a separate blocking bug, so we are probably ok without any manual symbol intervention here.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/f85b60ef1e734ec4af62eb1b93f7ed700b3e9a03

commit f85b60ef1e734ec4af62eb1b93f7ed700b3e9a03
Author: Mike Frysinger <vapier@chromium.org>
Date: Thu Jul 26 22:52:07 2018

DebugSymbolsStage: fix & improve handling of failed uploads

It's possible to use upload_symbols with the failed symbol list to
recover a partial symbol upload failure.  However, the exact way to
run that command is not obvious, especially across multiple builds.
Add a logging line so users can just copy & paste the right command.

While adding unittests for this, I noticed that while we intended to
upload the failed file list when the upload command failed, we threw
the exception too early for that.  This was compounded by the unittest
also throwing the wrong exception, so while we overall made sure the
stage continued to pass in face of failures, we swallowed too many.
So change UploadSymbols to delay throwing DebugSymbolsUploadException
until the end, and fix the unittest that checks this behavior to throw
failures_lib.BuildScriptFailure.

BUG= chromium:864590 
TEST=unittests pass

Change-Id: I6df08e3be7c3df08badab7cf9c15a900fcbdec8f
Reviewed-on: https://chromium-review.googlesource.com/1142595
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/f85b60ef1e734ec4af62eb1b93f7ed700b3e9a03/cbuildbot/stages/artifact_stages_unittest.py
[modify] https://crrev.com/f85b60ef1e734ec4af62eb1b93f7ed700b3e9a03/cbuildbot/stages/artifact_stages.py