+vapier

06:40:04: INFO: Uploading symbol_file: chrome/D856317FD6E47CED70C06D5B267CEAA90/chrome.sym
07:37:05: WARNING: could not upload: chrome.sym: HTTP 400: Bad Request

you should try the retry commands suggested at the end:
07:46:47: NOTICE: upload_symbols --failed-list gs://chromeos-image-archive/coral-release/R71-11123.0.0/failed_upload_symbols.list gs://chromeos-image-archive/coral-release/R71-11123.0.0/debug_breakpad.tar.xz

if that passes, it was a crash server flake.  if that still fails, the sym file might be too big or something.

Manually running that now.

Manual run is still running but it also failed to upload chrome.sym in the same place. In both cases (the builder and locally), the chrome.sym upload runs suspiciously close to 1 hour and then the HTTP 400 response is sent. So, it's likely either a file size upload limit or a timeout. File size limit seems suspect:

$ ls -lh chrome/D856317FD6E47CED70C06D5B267CEAA90/chrome.sym
-rw-r--r-- 1 jclinton primarygroup 601M Oct  3 07:34 chrome/D856317FD6E47CED70C06D5B267CEAA90/chrome.sym

Who is responsible for the Crash Server and its associated server-side limits?

iirc, the limit should be just under 700MiB per-sym file, so that should be working

crash/ is tracked via b/ and the "Misc. Services > Crash" component.  that should go to the internal crash team who manages this service.

Yes, the plot thickens: coral-release's last successful build uploaded a chrome.sym that was larger: https://luci-logdog.appspot.com/v/?s=chromeos/buildbucket/cr-buildbucket.appspot.com/8934660381024680288/+/steps/DebugSymbols/0/stdout .

14:34:58: INFO: Uploading symbol_file: chrome/BE83950D41D0C80613480927ADABFA0F0/chrome.sym
14:39:01: INFO: upload of  627569931 bytes took 0:04:03.019468

And it did it in just 4 minutes. So, the most likely explanation is that Crash Server is crashing and the GFE is eventually kicking out the request with a 400 response.

Will file a b/ bug.

Filed https://b.corp.google.com/issues/117235960

jclinton@, can you please share chrome.sym that you used for reproducing the problem (comment #4). You should probably compress it before attaching.

In the meantime I'll search our Crash Collector logs for crashes.

The file that's crashing the server is too large to upload to crbug. Here's the file uploaded to Google Drive: https://drive.google.com/file/d/1ZtT3n411YkJnaSiH6tWgX39BW5yaIpVc/view

FYI, still problematic on a later build:
11125.0.0, 71.0.3567.0
https://cros-goldeneye.corp.google.com/chromeos/console/listBuild?milestone=71#/

Still problematic for 11128.0.0 / 71.0.3567.0.

Jason, should the be reassigned per #9, etc.?

Thanks,

the issue is already assigned to the right team in b/117235960.  there isn't much we can do about it as it isn't our infra that's failing here.

Ah, thanks.. I missed the buganizer reference; I'll track there.

Quick update: we just pushed a release with a fix for this issue.

It's still failing this morning but with a new error code (502): https://luci-logdog.appspot.com/v/?s=chromeos/buildbucket/cr-buildbucket.appspot.com/8933271624790986576/+/steps/DebugSymbols/0/stdout

Over to new oncall to track resolution.

Looks like leading theory for 502 errors is OOM in their jobs: b/117349015

Hi, the latest M71 is still failing to upload debug symbols.  I'm treating this as a DEV blocker since we're getting reports of crashes and this compromises the ability to debug.  

Possible to escalate since it's a DEV blocker?

feel free to comment on the b/ directly

b/117349015 was closed as fixed yet the latest build still isn't getting symbols...

https://luci-logdog.appspot.com/v/?s=chromeos/buildbucket/cr-buildbucket.appspot.com/8933149476442563920/+/steps/DebugSymbols/0/stdout

I think vapier means b/117235960, which isn't closed. I am very unfamiliar with this, is there someone more competent that can spearhead this?

All that needs to be done is to look at the logs, find out what the HTTP response code is for chrome.sym upload, and report that to the Google Crash server team on that bug.

Done.

The Crash server issue was resolved (b/117235960) and chrome.sym from https://stainless.corp.google.com/browse/chromeos-image-archive/coral-release/R71-11123.0.0 was uploaded manually in order to validate that the fix was working.

11143.0.0 / 71.0.3572.0 has symbols; awaiting the next build to check there too.  Thanks for everyone's work on this!

do we need to go through and recover/re-upload the chrome symbols for previous builds, or are you OK with just moving forward with new builds ?

#29: Depends on the workload involved.  Preferred so we can capture crash signatures and metrics to determine their impact as heavy / light hitters.

#30: let me rephrase.  are there any releases before R71-11143.0.0 that you care about ?  if not, i'm not going to bother uploading the old symbols.

Yes, this release was pushed:

M71-DEV-CHROMEOS-4 (11125.0.0, 71.0.3567.0)


That's it though.  Thanks!

about half way through restoring them and they're passing.  i assume they'll finish by tomorrow.

[Auto-generated comment by a script] We noticed that this issue is targeted for M-71; it appears the fix may have landed after branch point, meaning a merge might be required. Please confirm if a merge is required here - if so add Merge-Request-71 label, otherwise remove Merge-TBD label. Thanks.

Thanks again for #34.  Most appreciated.

+1 on the thank you Vapier, nice work.

Re-opening since we appear to be missing debug symbols for the past two M71 builds:

2018-10-24 00:51	11151.11.0	71.0.3578.21
2018-10-23 00:30	11151.10.0	71.0.3578.18

Also note that it's not failing for all boards this time, but for most.

Also happening for ToT/M-72 Builds

This is a different issue. Please open a new bug.

[Auto-generated comment by a script] We noticed that this issue is targeted for M-71; it appears the fix may have landed after branch point, meaning a merge might be required. Please confirm if a merge is required here - if so add Merge-Request-71 label, otherwise remove Merge-TBD label. Thanks.