New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 691889 link

Starred by 5 users

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug

Blocked on:
issue 692976



Sign in to add a comment

When the size of Chrome symbol file exceeds the limit, the builder should become red.

Project Member Reported by hashimoto@chromium.org, Feb 14 2017

Issue description

The crash server has a size limit for symbol files, and to avoid hitting this, currently we are stripping CFI lines.
https://chromium.googlesource.com/chromiumos/chromite/+/master/scripts/upload_symbols.py#287

This behavior is problematic because symbol files without CFI lines are totally useless to restore the stack trace.
Also, because we are emitting no errors when strip happens, it's hard to notice and our crash reporting system gets broken silently.

To avoid this, we should make it more visible by letting the builder fail (i.e. displayed with red result) when the size limit is exceeded.
 
Cc: josa...@chromium.org
Josafat said we can let DbgSym stage fail when this happens, and it will be visible on the goldeneye UI.
Blockedon: 692976
Cc: dgarr...@chromium.org
Don said we once let DebugSymbols fail, but it was undone because of flakiness.
Fixing  issue 692976  will make it less robust, and hopefully we'll be able to let it fail again.
Fixing  issue 692976  should make it more robust, less flaky, sorry.

Comment 5 by vapier@chromium.org, Feb 16 2017

to be clear, the issue with DebugSymbols was general upload flakiness.  it has nothing to do with the proposal here: allow it to go red iff we strip down a symbol file.  i think we can improve the code to allow it to go red for some reasons, but orange for others.
Oops. I guess I'm skimming way too many issues.
Should we stop stripping CFI lines, no matter the file size?

If we do that, the crash server will reject files that are too large. Then ensure that will cause the stage to fail (I think that's true today, but I'm not certain).

Comment 8 by vapier@chromium.org, Feb 16 2017

if we get rejected by the symbol server, we don't know why.  it does not return an error like "file too big", it just gives us a HTTP code.

so our option is "you have no symbols", or "you have some symbols, but the result is less useful".  i don't think the stripping+reupload is bad.  but we should have visibility into that.

Comment 9 by vapier@chromium.org, Feb 16 2017

Cc: hashimoto@chromium.org gkihumba@chromium.org
 Issue 693193  has been merged into this issue.
IMO "no symbol + red build" is much better than "useless symbol + green build" which is our current state.
With no symbol on the crash server and red builds on the waterfall, it's really easy to notice the problem as early as possible.
it's not binary, nor is that what anyone suggested.  we're talking about doing reduced symbol => red build.  the issue of no symbol (because of upload failure) will stay green for now due to the past flakiness we've seen.
Everyone has agreed that stripped symbol is useless.
Why do we need to continue stripping?

Flakiness is addressed in  issue 692976 .
I took a look at recent builds, and all errors were caused by timeout.
There was no 403 which we've observed in past times.
Project Member

Comment 13 by bugdroid1@chromium.org, Mar 17 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/45273f06fe1d25260da18893c6797ded33e73a2b

commit 45273f06fe1d25260da18893c6797ded33e73a2b
Author: Ryo Hashimoto <hashimoto@google.com>
Date: Fri Mar 17 05:28:21 2017

upload_symbols: Print exception type when upload fails

This change is needed to make upload_symbols more reliable.

We are no longer seeing time out errors, but still it fails for unknown reason,
and the log is not helpful:

  06:53:09: INFO: Uploading symbol_file: executor/D6DF86DCC24B723ACCBC709A8B89EB270/executor.sym
  06:59:30: WARNING: could not upload: executor.sym: ''

BUG=chromium:691889
TEST=./scripts/upload_symbols_unittest

Change-Id: Id719fc5b718719176fd6fad4be6fc30f65dfb875
Reviewed-on: https://chromium-review.googlesource.com/456182
Commit-Ready: Ryo Hashimoto <hashimoto@chromium.org>
Tested-by: Ryo Hashimoto <hashimoto@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/45273f06fe1d25260da18893c6797ded33e73a2b/scripts/upload_symbols.py

Project Member

Comment 14 by bugdroid1@chromium.org, Mar 23 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/4ddf3f057364fba8f7d66e2ed093f3b4a4f617f8

commit 4ddf3f057364fba8f7d66e2ed093f3b4a4f617f8
Author: Ryo Hashimoto <hashimoto@google.com>
Date: Thu Mar 23 03:05:56 2017

upload_symbols: Retry upload on all network errors

After CL:171601, a few exception types have been added to the except clause.
Retry upload on these exception types too.

With this change, we'll be able to mitigate BadStatusLine (a subclass of
httplib.HTTPException) errors which are obsereved recently.
 22:36:41: INFO: Uploading symbol_file:
cros_unittest_sample/842EF9A00F079948372EBD6F4BE9D7380/cros_unittest_sample.sym
 22:40:41: WARNING: could not upload: cros_unittest_sample.sym:
BadStatusLine ''

BUG=chromium:691889
TEST=./scripts/upload_symbols_unittest

Change-Id: I0004ecf1381337a4d871b47ccd8b2a23e077c210
Reviewed-on: https://chromium-review.googlesource.com/458178
Commit-Ready: Ryo Hashimoto <hashimoto@chromium.org>
Tested-by: Ryo Hashimoto <hashimoto@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/4ddf3f057364fba8f7d66e2ed093f3b4a4f617f8/scripts/upload_symbols.py

Components: -Infra>Platform>Buildbot Infra>Client>ChromeOS

Sign in to add a comment