When the size of Chrome symbol file exceeds the limit, the builder should become red. |
||||
Issue descriptionThe crash server has a size limit for symbol files, and to avoid hitting this, currently we are stripping CFI lines. https://chromium.googlesource.com/chromiumos/chromite/+/master/scripts/upload_symbols.py#287 This behavior is problematic because symbol files without CFI lines are totally useless to restore the stack trace. Also, because we are emitting no errors when strip happens, it's hard to notice and our crash reporting system gets broken silently. To avoid this, we should make it more visible by letting the builder fail (i.e. displayed with red result) when the size limit is exceeded.
,
Feb 16 2017
,
Feb 16 2017
Don said we once let DebugSymbols fail, but it was undone because of flakiness. Fixing issue 692976 will make it less robust, and hopefully we'll be able to let it fail again.
,
Feb 16 2017
Fixing issue 692976 should make it more robust, less flaky, sorry.
,
Feb 16 2017
to be clear, the issue with DebugSymbols was general upload flakiness. it has nothing to do with the proposal here: allow it to go red iff we strip down a symbol file. i think we can improve the code to allow it to go red for some reasons, but orange for others.
,
Feb 16 2017
Oops. I guess I'm skimming way too many issues.
,
Feb 16 2017
Should we stop stripping CFI lines, no matter the file size? If we do that, the crash server will reject files that are too large. Then ensure that will cause the stage to fail (I think that's true today, but I'm not certain).
,
Feb 16 2017
if we get rejected by the symbol server, we don't know why. it does not return an error like "file too big", it just gives us a HTTP code. so our option is "you have no symbols", or "you have some symbols, but the result is less useful". i don't think the stripping+reupload is bad. but we should have visibility into that.
,
Feb 16 2017
,
Feb 17 2017
IMO "no symbol + red build" is much better than "useless symbol + green build" which is our current state. With no symbol on the crash server and red builds on the waterfall, it's really easy to notice the problem as early as possible.
,
Feb 17 2017
it's not binary, nor is that what anyone suggested. we're talking about doing reduced symbol => red build. the issue of no symbol (because of upload failure) will stay green for now due to the past flakiness we've seen.
,
Feb 18 2017
Everyone has agreed that stripped symbol is useless. Why do we need to continue stripping? Flakiness is addressed in issue 692976 . I took a look at recent builds, and all errors were caused by timeout. There was no 403 which we've observed in past times.
,
Mar 17 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/45273f06fe1d25260da18893c6797ded33e73a2b commit 45273f06fe1d25260da18893c6797ded33e73a2b Author: Ryo Hashimoto <hashimoto@google.com> Date: Fri Mar 17 05:28:21 2017 upload_symbols: Print exception type when upload fails This change is needed to make upload_symbols more reliable. We are no longer seeing time out errors, but still it fails for unknown reason, and the log is not helpful: 06:53:09: INFO: Uploading symbol_file: executor/D6DF86DCC24B723ACCBC709A8B89EB270/executor.sym 06:59:30: WARNING: could not upload: executor.sym: '' BUG=chromium:691889 TEST=./scripts/upload_symbols_unittest Change-Id: Id719fc5b718719176fd6fad4be6fc30f65dfb875 Reviewed-on: https://chromium-review.googlesource.com/456182 Commit-Ready: Ryo Hashimoto <hashimoto@chromium.org> Tested-by: Ryo Hashimoto <hashimoto@chromium.org> Reviewed-by: Don Garrett <dgarrett@chromium.org> [modify] https://crrev.com/45273f06fe1d25260da18893c6797ded33e73a2b/scripts/upload_symbols.py
,
Mar 23 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/4ddf3f057364fba8f7d66e2ed093f3b4a4f617f8 commit 4ddf3f057364fba8f7d66e2ed093f3b4a4f617f8 Author: Ryo Hashimoto <hashimoto@google.com> Date: Thu Mar 23 03:05:56 2017 upload_symbols: Retry upload on all network errors After CL:171601, a few exception types have been added to the except clause. Retry upload on these exception types too. With this change, we'll be able to mitigate BadStatusLine (a subclass of httplib.HTTPException) errors which are obsereved recently. 22:36:41: INFO: Uploading symbol_file: cros_unittest_sample/842EF9A00F079948372EBD6F4BE9D7380/cros_unittest_sample.sym 22:40:41: WARNING: could not upload: cros_unittest_sample.sym: BadStatusLine '' BUG=chromium:691889 TEST=./scripts/upload_symbols_unittest Change-Id: I0004ecf1381337a4d871b47ccd8b2a23e077c210 Reviewed-on: https://chromium-review.googlesource.com/458178 Commit-Ready: Ryo Hashimoto <hashimoto@chromium.org> Tested-by: Ryo Hashimoto <hashimoto@chromium.org> Reviewed-by: Don Garrett <dgarrett@chromium.org> [modify] https://crrev.com/4ddf3f057364fba8f7d66e2ed093f3b4a4f617f8/scripts/upload_symbols.py
,
Jun 2 2018
|
||||
►
Sign in to add a comment |
||||
Comment 1 by hashimoto@chromium.org
, Feb 15 2017