The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/bc8e21e8a47633a3b9ccdca075365cce489d66d4

commit bc8e21e8a47633a3b9ccdca075365cce489d66d4
Author: Don Garrett <dgarrett@google.com>
Date: Thu May 19 01:17:54 2016

commands: upload_symbols with --dedupe

Adjust UploadSymbols to pass --dedupe to upload_symbols. This enables
swarming support to avoid uploading symbol files that have already been
uploaded, and substantially reduces our uploads to the symbol server.

This may not be stable, but I'm enabling it as a possible solution to
current load issues.

BUG= chromium:610902 
TEST=run_tests + tryjob of lumpy-release

Change-Id: Ied8ab3f2d3875fac81d71c42ae03941c83945ce1
Reviewed-on: https://chromium-review.googlesource.com/345943
Commit-Ready: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Kevin Cheng <kevcheng@chromium.org>

[modify] https://crrev.com/bc8e21e8a47633a3b9ccdca075365cce489d66d4/cbuildbot/commands.py

We'll watch and see if this CL reduces load enough to make things more stable.

reproducible in daisy spring:
https://crash.corp.google.com/browse?stbtiq=3ab1111a00000000#3

That example is from a release branch, which does NOT contain the --dedupe change. If --dedupe proves effective, it can be merged to that branch.

However, checking our release builders, I see --dedupe to be working well for physical builders, but GCE builders are getting errors like this:


12:33:27: WARNING: Authentication is required for https://isolateserver.appspot.com/_ah/api/isolateservice/v1/preupload on attempt 0.
403 Client Error: Forbidden for url: https://isolateserver.appspot.com/_ah/api/isolateservice/v1/preupload
12:33:27: ERROR: Unable to authenticate to https://isolateserver.appspot.com (403 Client Error: Forbidden for url: https://isolateserver.appspot.com/_ah/api/isolateservice/v1/preupload).
12:33:27: WARNING: talking to dedupe server failed
Traceback (most recent call last):
  File "/b/cbuild/internal_master/chromite/scripts/upload_symbols.py", line 467, in SymbolDeduplicator
    items = storage.contains(items)
  File "/b/cbuild/internal_master/chromite/third_party/swarming.client/isolateserver.py", line 1119, in contains
    'Failed to execute preupload query')
MappingError: Failed to execute preupload query

We either need to get additional permissions for the service account used, or need to update the authentication process used when connecting to isolate.

Reproducible in Ultima - CrOS 8350.8.0/52.0.2743.0 
https://crash.corp.google.com/browse?stbtiq=02ebcb2a00000000

An example build link for full logs:

https://uberchromegw.corp.google.com/i/chromeos/builders/parrot_ivb-release/builds/64/steps/DebugSymbols/logs/stdio

Vapier, who owns the debug server? I'd like to push them again to add a batch upload API.

swarming-eng@google.com and maruel@ and vadimsh@

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/23eb707d63b8d1cb1601f9be5ef510ed4422a8ef

commit 23eb707d63b8d1cb1601f9be5ef510ed4422a8ef
Author: Don Garrett <dgarrett@chromium.org>
Date: Wed Jun 01 16:50:42 2016

Revert "commands: upload_symbols with --dedupe"

This reverts commit bc8e21e8a47633a3b9ccdca075365cce489d66d4.

BUG= chromium:610902 

Change-Id: Iecde953eeb3e28771bc3978739ede2e15e6f2f3d
Reviewed-on: https://chromium-review.googlesource.com/348466
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Commit-Queue: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/23eb707d63b8d1cb1601f9be5ef510ed4422a8ef/cbuildbot/commands.py

Re #44: Isolate client doesn't use any real authentication currently and relies on IP whitelist on the server. Golo is already whitelisted.

If CrOS GCE slaves have stable IP subnet assigned to them, we can whitelist it. Otherwise we'd need to implement some real form of an authentication.

Our current GCE builders all are in the address block 104.196.64.0/24.

Moving this nonessential bug to the next milestone.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Re #51: Could we whitelist the CrOS GCE slaves and start using --dedupe? This is currently our biggest crash with M52 and we go beta this week.

Still able to reproduce this issue on build  8350.21.0

Crash report links:

Peach_pi: https://crash.corp.google.com/browse?stbtiq=862675363c504a7fac6d72504cdd67e4

nyan_blaze: https://crash.corp.google.com/browse?stbtiq=a4d9a5aa47254033bd9adee7286693f1

sentry: https://crash.corp.google.com/browse?stbtiq=099c8d2cb6414360a76215afdc6cd95a

Lumpy: https://crash.corp.google.com/browse?stbtiq=4db342f9dd294ea88e2b27577311376d

I've filed  crbug.com/618520  to cover the whitelisting problem.

Can I get some more information here to help quantify the importance of this bug? Something that would be useful when trying to get the crash server team to prioritize the improvements we want to fix this?

We get Chrome crashes and user reports for which we don't get stack traces. Also, we don't get magic signatures for clustering top crashers when this happens. Because of the holes in symbols the crash server is of limited use to ChromeOS right now. This is pretty bad, as we are trying to ship ARC++?

Maybe Zel, Albert or Elijah can explain that better to the crash server team than I do.

If anyone would like to comment here, that would be appreciated.

b/11007866

Just to pile on here, CC'ing folks so they can see why we have no Chrome symbols on recent and current dev channel elm images.

Still see the issue in 8350.21.1 on Lars:
https://crash.corp.google.com/browse?stbtiq=03fc6bdc00000000#3

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/8bdea02b87149e842e1dc179e75586fe940025d9

commit 8bdea02b87149e842e1dc179e75586fe940025d9
Author: Don Garrett <dgarrett@google.com>
Date: Thu May 19 01:17:54 2016

commands: upload_symbols with --dedupe

Adjust UploadSymbols to pass --dedupe to upload_symbols. This enables
swarming support to avoid uploading symbol files that have already been
uploaded, and substantially reduces our uploads to the symbol server.

This may not be stable, but I'm enabling it as a possible solution to
current load issues.

This has been committed, and reverted because of permission errors on
GCE instances. Those permission errors should now have been addressed,
so trying again.

BUG= chromium:610902 
TEST=run_tests + tryjob of lumpy-release

Change-Id: I3e963d3891ecb26171158dae8535ffeaa012d7c3
Reviewed-on: https://chromium-review.googlesource.com/351631
Commit-Ready: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/8bdea02b87149e842e1dc179e75586fe940025d9/cbuildbot/commands.py

I see a very different error that would have the same symptoms. A background process was silent for over 600 seconds and so killed because it looked like it was hung.

This might be related to enabling --dedupe, maybe not.

https://uberchromegw.corp.google.com/i/chromeos/builders/auron_yuna-release/builds/106/steps/DebugSymbols/logs/stdio

Comment 65: This is the same log we see when the debug upload stage fails in the builders

As a very temporary workaround, we can manually upload the symbols for a given build.

Follow the report stage "Artifacts" link to the build artficacts (because of crbug.com/622140 click on the directory above where the link goes), download the file ending with debug.tgz.

Then, using a recent ChromeOS checkout, issue a command like:

bin/upload_symbols --official_build ~/Desktop/nyan_big-release%2FR53-8481.0.0%2Fdebug.tgz

This will take about 90 minutes to run, but should upload the symbols for that build.

Symbols for induced crashes still missing in M52-BETA-6 (8350.38.0) 52.0.2743.49 : 
https://crash.corp.google.com/browse?stbtiq=00301f2a6583447e9af574d8598ed91c

Yes they are. I've been manually uploading them for some boards and will continue tomorrow.

issue can reproduce on below steps:
1. Go to www.hulu.com and select a video to play
2. close/open lid

Crash ID386af41600000000

feedback:
Report ID: 10205042242

Ah, yes. There is a chrome crash on hulu, but no symbols. I filed issue 623220 with details.

Issue 621215 has been merged into this issue.

 Issue 620404  has been merged into this issue.

Symbols are still missing for M52 beta build 8350.46.0
https://crash.corp.google.com/browse?stbtiq=1ca9c9fa83f545afb7b5bea4d20a5995

sontis@: symbols will be missing until we fix the issue. There's people working on it

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/a28be6d6b070abc1a9e1fbccb6ce3d274a14781c

commit a28be6d6b070abc1a9e1fbccb6ce3d274a14781c
Author: Don Garrett <dgarrett@google.com>
Date: Fri Jun 17 01:09:35 2016

upload_symbols: General refactor to remove multiprocess.

After difficulty debuging upload_symbols, refactor it to try and
simplify things.

BUG= chromium:610902 
TEST=Unitests + local manual runs.

Change-Id: Ida6e66b81a48a13498fe82d114981ede9c7491b8
Reviewed-on: https://chromium-review.googlesource.com/354121
Commit-Ready: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/a28be6d6b070abc1a9e1fbccb6ce3d274a14781c/scripts/upload_symbols_unittest.py
[modify] https://crrev.com/a28be6d6b070abc1a9e1fbccb6ce3d274a14781c/scripts/upload_symbols.py

@#68 - I tried to follow the instructions to manually upload symbols for elm-cheets-release R53-8529.0.0 to debug a Chrome crash on a recent elm-cheets canary channel release.

The steps I used were the following, in cros_sdk, using the gs:// link from goldenye.:
gsutil cp gs://chromeos-releases/canary-channel/elm-cheets/8529.0.0/debug-elm-cheets.tgz .
upload_symbols --official_build ./debug-elm-cheets.tgz 

However, it didn't seem to work, since the backtrace [0] is still not symbolized:
[0] https://crash.corp.google.com/browse?q=ReportID%3D%275c7c8efc00000000%27

Also, I see "debug-elm-cheets" symbols [1]
[1] https://pantheon.corp.google.com/m/cloudstorage/b/chromeos-releases/o/canary-channel/elm-cheets/8529.0.0/debug-elm-cheets.tgz

And I "debug.tgz" symbols [2]
[2] https://pantheon.corp.google.com/m/cloudstorage/b/chromeos-image-archive/o/elm-cheets-release/R53-8529.0.0/debug.tgz

Are these different?
Also, neither of these 'debug.tgz' files match the name format of "nyan_big-release%2FR53-8481.0.0%2Fdebug.tgz".

You can display the stack trace locally with minidump_stackwalk.
(https://sites.google.com/a/google.com/crash/users/how-to/how-to-process-crash-reports-from-the-command-line)
minidump_stackwalk can be built by running "ninja -C out/Release minidump_stackwalk" in a Chrome checkout, and the .sym files can be found in the debug tgz file.

FWIW, the steps I used to get the stack trace with minidump_stackwalk:

1. Get the debug.tgz from the same version of chromeos that the minidump was generated on. Untar it and there's a debug/breakpad directory which contains the .sym files you need.
2. Run `minidump_stackwalk <path to minidump> <path to untar'ed debug.tgz>/breakpad`

After landing the updated upload script, the dedupe crash is gone. However we are still overloading the crash server before uploading all symbols.

I'm thinking that I'll do three things.

1) Improve the logging a little to give better stats.
2) Upload the largest symbol files first since they are probably the most important. I think I lost this logic during the rewrite.
3) Slow down the upload rate to try and avoid DDoS'ing the crash server.

Digging a little further, there are some builds for which the upload is working, and some for which it's failing. The dedupe seems to be significantly reducing the number of symbols we actually upload.

I still plan to make the changes mentioned above.

Example success:

https://uberchromegw.corp.google.com/i/chromeos/builders/peach_pit-release/builds/1643

Since this is now working for some of the builders, I'm declaring the emergency part of this over. I still hope to make the symbol upload process much more reliable over all.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/1bc1e10750013f0781096612e65fbe696e9cf0cc

commit 1bc1e10750013f0781096612e65fbe696e9cf0cc
Author: Don Garrett <dgarrett@google.com>
Date: Thu Jul 07 00:06:10 2016

upload_symbols: Sort larger files first.

This restores functionality in the upload_symbols script that was lost
in the rewrite. Upload the larger files first, since they are probably
more important to have uploaded.

BUG= chromium:610902 
TEST=Unittests

Change-Id: Iafef4d436738634f7ae8f3ed981f63959cc36d28
Reviewed-on: https://chromium-review.googlesource.com/358622
Commit-Ready: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/1bc1e10750013f0781096612e65fbe696e9cf0cc/scripts/upload_symbols.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/7a79309d63ec7a3995d10aa091472b668a689320

commit 7a79309d63ec7a3995d10aa091472b668a689320
Author: Don Garrett <dgarrett@google.com>
Date: Wed Jul 06 23:50:27 2016

upload_symbols: Slow the upload rate.

We had a 200 ms minimum delay between upload attempts. Switch to 500ms
delay to avoid DoSing the crash server. Without deduping, and about 4000
symbols, this takes us from ~13 minutes wait time to ~33 minutes wait
time per upload. Or an increase of roughly 20 minutes.

BUG= chromium:610902 
TEST=Unittest.

Change-Id: I03897a246a0fcbedfa31ec1a0bb60253ec5ebaf5
Reviewed-on: https://chromium-review.googlesource.com/358650
Commit-Ready: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/7a79309d63ec7a3995d10aa091472b668a689320/scripts/upload_symbols.py

You should now have symbols for most recent builds, meaning that our current behavior is very similar to how it's been for years.

I still have the bug open, because I intend to try and make the uploads more reliable than they've ever been before, but this will involve working with the crash server team, which will likely take time. See b/30042269 and b/11007866.

Just generated a fake crash (chrome://crash) on my Chromebox and got another un-symbolized crash: 78caddb600000000 - still looks like the Chrome binary has two executable segments.  Have updated issue 601854.

These three changes have had over a week of soak time on TOT. They don't solve all of our problems, but do help. I'd like to merge them to all active branches.

https://chromium-review.googlesource.com/354121
https://chromium-review.googlesource.com/358650
https://chromium-review.googlesource.com/358622

[Automated comment] Request affecting a post-stable build (M51), manual review required.

Your change meets the bar and is auto-approved for M53 (branch: 2785)

[Automated comment] Less than 2 weeks to go before stable on M52, manual review required.

Consider merge approved for 51, however we should theoretically be done with builds on 51.

Merged to R51 and R53 (not yet to R52).

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/2395b58972d3d7c77bde59d4962a616226f9cf00

commit 2395b58972d3d7c77bde59d4962a616226f9cf00
Author: Don Garrett <dgarrett@google.com>
Date: Fri Jun 17 01:09:35 2016

upload_symbols: General refactor to remove multiprocess.

After difficulty debuging upload_symbols, refactor it to try and
simplify things.

BUG= chromium:610902 
TEST=Unitests + local manual runs.

Change-Id: Ida6e66b81a48a13498fe82d114981ede9c7491b8
Previous-Reviewed-on: https://chromium-review.googlesource.com/354121
(cherry picked from commit 2d9e36b4f268dd338a334360e24bf2e73c175670)
Reviewed-on: https://chromium-review.googlesource.com/360334
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Commit-Queue: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/2395b58972d3d7c77bde59d4962a616226f9cf00/scripts/upload_symbols_unittest.py
[modify] https://crrev.com/2395b58972d3d7c77bde59d4962a616226f9cf00/scripts/upload_symbols.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/c6ca043da9f2b42b530e50fb06ec31cdde126aa8

commit c6ca043da9f2b42b530e50fb06ec31cdde126aa8
Author: Don Garrett <dgarrett@google.com>
Date: Wed Jul 06 23:50:27 2016

upload_symbols: Slow the upload rate.

We had a 200 ms minimum delay between upload attempts. Switch to 500ms
delay to avoid DoSing the crash server. Without deduping, and about 4000
symbols, this takes us from ~13 minutes wait time to ~33 minutes wait
time per upload. Or an increase of roughly 20 minutes.

BUG= chromium:610902 
TEST=Unittest.

Change-Id: I03897a246a0fcbedfa31ec1a0bb60253ec5ebaf5
Previous-Reviewed-on: https://chromium-review.googlesource.com/358650
(cherry picked from commit 9382af468d95d88f5e0ff5e45671fd69caab3ef8)
Reviewed-on: https://chromium-review.googlesource.com/359989
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Commit-Queue: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/c6ca043da9f2b42b530e50fb06ec31cdde126aa8/scripts/upload_symbols.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/2c418a79ee6f940d28d2ae99d926f8c29f1e1e19

commit 2c418a79ee6f940d28d2ae99d926f8c29f1e1e19
Author: Don Garrett <dgarrett@google.com>
Date: Thu Jul 07 00:06:10 2016

upload_symbols: Sort larger files first.

This restores functionality in the upload_symbols script that was lost
in the rewrite. Upload the larger files first, since they are probably
more important to have uploaded.

BUG= chromium:610902 
TEST=Unittests

Change-Id: Iafef4d436738634f7ae8f3ed981f63959cc36d28
Previous-Reviewed-on: https://chromium-review.googlesource.com/358622
(cherry picked from commit 8c02731e46f74ee56cd666896557709911462d95)
Reviewed-on: https://chromium-review.googlesource.com/360225
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Commit-Queue: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/2c418a79ee6f940d28d2ae99d926f8c29f1e1e19/scripts/upload_symbols.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/4edc0b447ef4d94db0dd9ad50b4edeadc76cb32d

commit 4edc0b447ef4d94db0dd9ad50b4edeadc76cb32d
Author: Don Garrett <dgarrett@google.com>
Date: Fri Jun 17 01:09:35 2016

upload_symbols: General refactor to remove multiprocess.

After difficulty debuging upload_symbols, refactor it to try and
simplify things.

BUG= chromium:610902 
TEST=Unitests + local manual runs.

Change-Id: Ida6e66b81a48a13498fe82d114981ede9c7491b8
Previous-Reviewed-on: https://chromium-review.googlesource.com/354121
(cherry picked from commit 8f2ebb6360d7dd7fa4370b471588cd8b95dec04e)
Reviewed-on: https://chromium-review.googlesource.com/360243
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Commit-Queue: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/4edc0b447ef4d94db0dd9ad50b4edeadc76cb32d/scripts/upload_symbols_unittest.py
[modify] https://crrev.com/4edc0b447ef4d94db0dd9ad50b4edeadc76cb32d/scripts/upload_symbols.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/aef072165fd26612f4c9d896a2165e8e09670aa1

commit aef072165fd26612f4c9d896a2165e8e09670aa1
Author: Don Garrett <dgarrett@google.com>
Date: Wed Jul 06 23:50:27 2016

upload_symbols: Slow the upload rate.

We had a 200 ms minimum delay between upload attempts. Switch to 500ms
delay to avoid DoSing the crash server. Without deduping, and about 4000
symbols, this takes us from ~13 minutes wait time to ~33 minutes wait
time per upload. Or an increase of roughly 20 minutes.

BUG= chromium:610902 
TEST=Unittest.

Change-Id: I03897a246a0fcbedfa31ec1a0bb60253ec5ebaf5
Previous-Reviewed-on: https://chromium-review.googlesource.com/358650
(cherry picked from commit 595b7c14e205ce5ca74ce4a3b8ea6ad30d68259d)
Reviewed-on: https://chromium-review.googlesource.com/359929
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Commit-Queue: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/aef072165fd26612f4c9d896a2165e8e09670aa1/scripts/upload_symbols.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/2b2317d4eaedba5ba8c0390caef2cb690cf5833a

commit 2b2317d4eaedba5ba8c0390caef2cb690cf5833a
Author: Don Garrett <dgarrett@google.com>
Date: Thu Jul 07 00:06:10 2016

upload_symbols: Sort larger files first.

This restores functionality in the upload_symbols script that was lost
in the rewrite. Upload the larger files first, since they are probably
more important to have uploaded.

BUG= chromium:610902 
TEST=Unittests

Change-Id: Iafef4d436738634f7ae8f3ed981f63959cc36d28
Previous-Reviewed-on: https://chromium-review.googlesource.com/358622
(cherry picked from commit 940827def1f9bc11dd1bd4c87ba2ce3e96f20af1)
Reviewed-on: https://chromium-review.googlesource.com/360333
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Commit-Queue: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/2b2317d4eaedba5ba8c0390caef2cb690cf5833a/scripts/upload_symbols.py

dgarrett - 
Even after the above patches, elm-release builder is still failing to upload symbols:

https://uberchromegw.corp.google.com/i/chromeos/builders/elm-release/builds/207/steps/DebugSymbols/logs/stdio

I see two interesting things:
dump_syms crashed with SIGSEGV

12:43:50: INFO: RunCommand: /b/cbuild/internal_master/chromite/bin/upload_symbols --yes --dedupe --root /b/cbuild/internal_master/chroot --board elm --official_build --failed-list /b/cbuild/internal_master/buildbot_archive/elm-release/R54-8584.0.0/failed_upload_symbols.list in /b/cbuild/internal_master
12:44:01: INFO: Starting new HTTPS connection (1): isolateserver.appspot.com
12:44:02: INFO: Queried 100 files, 1 cache hit
12:44:03: INFO: Uploading symbol_file: nacl_helper/31FE501AA8FDF11880A92E6876BFB81D0/nacl_helper.sym
12:47:29: WARNING: could not upload: nacl_helper.sym: HTTP 403: Forbidden
... 29 more HTTP 403: Forbidden errors ...

STEP_WARNINGS
13:28:24: WARNING: 30 non-recoverable upload errors caused 3777 skipped uploads.
13:28:24: ERROR: encountered 30 problem(s)


Are these the same issues tracked in this issue, or should I file a new one?

I'm seeing this on crashes from samus devices now too.

Is there a workaround for this? There are a number of crashes we're trying to debug on 53 that are not giving us useful stack traces due to this.

I know we can pull down the dump from the crash report. Is there any way to get our hands on the symbols so we can analyze locally? They're not in goldeneye.

I'm out next week, but I meet with the crash server team when I get back to see about a better solution.

In the mean time, there is a manual work around that gkihumba@ can teach you about.

Issue 620247 has been merged into this issue.

This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible!

If all merges have been completed, please remove any remaining Merge-Approved labels from this issue.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Still reproducible in ChromeOS 8530.21.0 / 53.0.2785.20 : 
https://crash.corp.google.com/browse?stbtiq=d6d5900e00000000

This crash has 2 chrome modules, in other words it is fixed in issue 628040 and needs Chrome to rev once more to be gone.

This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible!

If all merges have been completed, please remove any remaining Merge-Approved labels from this issue.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/e5195c5a89d9f869176ef6e799b02617e7d6c78f

commit e5195c5a89d9f869176ef6e799b02617e7d6c78f
Author: Don Garrett <dgarrett@google.com>
Date: Fri Jun 17 01:09:35 2016

upload_symbols: General refactor to remove multiprocess.

After difficulty debuging upload_symbols, refactor it to try and
simplify things.

BUG= chromium:610902 
TEST=Unitests + local manual runs.

Change-Id: Ida6e66b81a48a13498fe82d114981ede9c7491b8
Previous-Reviewed-on: https://chromium-review.googlesource.com/354121
(cherry picked from commit fab51ac6868df7c84683da3d8d759db44853aab4)
Reviewed-on: https://chromium-review.googlesource.com/363063
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/e5195c5a89d9f869176ef6e799b02617e7d6c78f/scripts/upload_symbols_unittest.py
[modify] https://crrev.com/e5195c5a89d9f869176ef6e799b02617e7d6c78f/scripts/upload_symbols.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/03128e688e75d799e9d85a9a96bb723e62b0cd28

commit 03128e688e75d799e9d85a9a96bb723e62b0cd28
Author: Don Garrett <dgarrett@google.com>
Date: Wed Jul 06 23:50:27 2016

upload_symbols: Slow the upload rate.

We had a 200 ms minimum delay between upload attempts. Switch to 500ms
delay to avoid DoSing the crash server. Without deduping, and about 4000
symbols, this takes us from ~13 minutes wait time to ~33 minutes wait
time per upload. Or an increase of roughly 20 minutes.

BUG= chromium:610902 
TEST=Unittest.

Change-Id: I03897a246a0fcbedfa31ec1a0bb60253ec5ebaf5
Previous-Reviewed-on: https://chromium-review.googlesource.com/358650
(cherry picked from commit 4559d4b74f22700116b08ccd2c0a72e6c6af1ed1)
Reviewed-on: https://chromium-review.googlesource.com/363064
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/03128e688e75d799e9d85a9a96bb723e62b0cd28/scripts/upload_symbols.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/fe30980de48feabd84021fdb26a1e76e7a445845

commit fe30980de48feabd84021fdb26a1e76e7a445845
Author: Don Garrett <dgarrett@google.com>
Date: Thu Jul 07 00:06:10 2016

upload_symbols: Sort larger files first.

This restores functionality in the upload_symbols script that was lost
in the rewrite. Upload the larger files first, since they are probably
more important to have uploaded.

BUG= chromium:610902 
TEST=Unittests

Change-Id: Iafef4d436738634f7ae8f3ed981f63959cc36d28
Previous-Reviewed-on: https://chromium-review.googlesource.com/358622
(cherry picked from commit 606a82adff6aa6150242b0c18b1e99d2412e97af)
Reviewed-on: https://chromium-review.googlesource.com/363054
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/fe30980de48feabd84021fdb26a1e76e7a445845/scripts/upload_symbols.py

For the time being, this seems to be fixed. I don't think the current history would be be useful dealing with future issues, so closing.

Issue 629301 has been merged into this issue.

the sorting logic used to be better when things were run in parallel.  as i described in comment 29 in part (4), it was sorted by size & interleaved so you'd eat the queue from both ends.  this lead to an overall faster upload.

while upload_symbols itself didn't implement that logic, cros_generate_breakpad would produce the mix of large/small files that upload_symbols would consume in order.

We could put that logic back easily enough. I don't think I was able to understand what it was doing before.

i totally understand the desire to try and simplify and that it probably wasn't easy to understand if you hadn't looked at it before :)

i think your removal of most of the multiprocessing/parallel usage is what incidentally fixed the hangs with swarming.  i think swarming is based on threading, and they didn't really test at all running under multiprocessing.

Danny this may still be present in 55 beta: https://crash.corp.google.com/browse?stbtiq=c9a009ef00000000

Only few symbols displayed:
https://crash.corp.google.com/browse?stbtiq=4dc40bd880000000#0 

Verified.
https://crash.corp.google.com/browse?stbtiq=3183fec480000000

Reopening this. Symbols missing in M63 e.g. edgar.

https://crash.corp.google.com/browse?q=product.name%3D%27Chrome_ChromeOS%27%20AND%20product.version%3D%2763.0.3239.26%27%20AND%20custom_data.ChromeCrashProto.ptype%3D%27renderer%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27start_thread%27&sql_dialect=dremelsql&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D&stbtiq=&reportid=39a5e736da0ae7a0&index=0#5

Is there an indication on GoldenEye this is failing?
DbgSym column on RC candidate show all good - https://screenshot.googleplex.com/XFugpWDLyLC
URL: https://cros-goldeneye.corp.google.com/chromeos/console/qaRelease?releaseName=M63-BETA-CHROMEOS-1

RBS  issue 779840  was filed yesterday.

Added comment to   issue 779840 

yeah, this bug is resolved.  please don't re-open it in the future simply "when symbols aren't found".  that wasn't the issue here.

dropping r-v-g label as there's nothing secret here.

Did this get merged into M64?  #3 crash for M64-stable at 6.16% / count = 369.

Do we have the merge CL so I can check?

Signature:  https://crash.corp.google.com/browse?q=product.name%3D%27ChromeOS%27%20AND%20product.Version%3D%2710176.68.0%27%20AND%20stable_signature%3D%27raise-35452697%27&stbtiq=&reportid=&index=0#0

this bug was fixed long before M64 was even started.  as mentioned in comment 133, whatever issues you're having now are unrelated to this bug, so please start a new one.