Have we made any clang changes this time?

The Windows one was very Windows-specific (issue with cdb.exe binary). +mark, +rsesek who might have a guess why CrashIntentionally() wouldn't show up (sometimes only?), and in the presence of otherwise valid looking stacks.

It doesn’t have symbols for the module that CrashIntentionally lives in. For example:

0x10f67e000 - 0x116f9cfff  Content Shell Framework  0.0.0.0  (WARNING: No symbols, Content Shell Framework, 95BB48765F493ED498937DACF2098BD00)

It looks like components/crash/content/tools/generate_breakpad_symbols.py is responsible for doing all of the dump_syms. Eyeballing it, I’d say that if there’s anything wrong, it’d probably be in GetSharedLibraryDependenciesMac. It’s either not understanding what “otool -l” or “otool -L” have to say, or probably more likely, otool isn’t installed on the bot that’s running this. otool is part of the developer tools (either Xcode or Command Line Tools), and in the swarming world, we may now be deploying bots without those tools.

Thanks Mark for your analysis.

+maruel, dba and iannucci because of the possible configuration problem with the Mac Swarming bots.

+jochen as one of the original OWNERs of this test.

Who can drive this to completion? Was otool guaranteed to be present on the pre-10.13 OS images?

[+erikchen, +justincohen, +sergeyberezin - CC'ing those familiar with the Xcode hermetic efforts]

In #3 mark@ is correct wrt swarming bots without those tools. We've been moving away from a system-wide Xcode. It was my understanding that things should be including the hermetic Xcode toolchain if they need those bits on their own now.

how can one include the hermetic Xcode toolchain? I.e. what target does the gn rule need to depend on?

Re: otool

I thought that swarming bots were deployed with CommandLineTools [needed for git, etc.] which should also have otool. 

The bug this is blocking is RBB, so marking this as such.

Swarming (testing) bots normally don't need git or other similar tools, since it doesn't do any checkouts. Having said that, at least for iOS tests, we are deploying Xcode through CIPD pipeline (https://chromium.googlesource.com/chromium/src/+/master/ios/build/bots/chromium.mac/ios-simulator.json#7), pretty much the entire Xcode.app as released by Apple or go/xcode. I haven't checked if otool is included in the packages that Xcode installs (with xcodebuild -runFirstLaunch), but that should be relatively easy to check.

If otool is needed and is not included, it's probably best to find a way to install it through CIPD by requesting it in the swarming task.

Re #7:  http://crbug.com/652469#c3  -- it was desired with hermetic builds to move that binary out of the way so that it didn't cause an unwanted fallback. otool isn't required for our bootstrap bits, this is relating to an Xcode run.

Re #11: otool comes with Xcode.app, so when you select a valid Xcode installation (or CLT) path, it'll use that.

We can change the deploy so that otool is not moved out of the way if that's the desired way to fix this, as we do need Command Line Tools to bootstrap the bot.

Typo in #12:  "..this is relating to a _swarming task run_" is what I meant (Xcode on the brain).

Pls apply appropriate OSs label. Thank you.

No need to block beta on this

dba@, can you please add otool?

#17 - I can to unblock this, but i'd like to see a plan to stop relying on the system versions, as in #12 I linked to a bug where we've been told to move those out of the way quite some time ago on the desktop builders, and it's rather sad the testing is relying on these tools instead of pulling them in themselves.

Cool, thx. Please assign the bug back to me once otool is in the bot - I'll work on a longer term solution

jochen@: For the longer term solution, please consider reusing go/hermetic-xcode. It's already deployed fleet-wide to ios-* builders, both upstream and downstream, and is a matter of porting it to the 'chromium' recipe.

otool should now be on the bots, retried the task in #0 with success: https://chromium-swarm.appspot.com/task?id=3bd22a14d87da910

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome-golo/chrome-golo/+/893e280cd455b95b450d9ca318bc5e1f54cb9edd

commit 893e280cd455b95b450d9ca318bc5e1f54cb9edd
Author: Bryce Albritton <dba@google.com>
Date: Wed Feb 21 17:27:56 2018

jochen@ this is blocking issue 805475 which is critical and RBB for M66 so this should still be RBB. 

Also why was iOS added as the OS? I see no references as that an impacted OS outside of Sergey referencing how the iOS team uses cipd for xcode deployments ont he bots.

If this isn't blocking issue 805475 please feel free to update why it does not block the upgrade of the CQ and then this won't have to be considered as RBB and can also be removed as blocking 805475 in general.

why are CQ issues release blockers?

But yes, the dependency is now resolved..

Chrome for Mac no longer supports 10.9 after M65, so the 66 milestone needs to be stabilized before branch with a CQ running the needed OS configuration (which will now be 10.13). Tests not passing 10.13 need to start so that the CQ won't be blocked.

After chatting with Sergey, I'm marking this bug as blocked on hermetic Xcode being available in the chromium recipe.

nit: issue 475693 is for ios recipe only.

I filed issue 814426 specifically for swarming tasks triggered by chromium recipe, and switched the dependency to that.

I'm not comfortable with this being P3. This is repeatedly blowing up my mac_chromium_rel_ng trybots and I'm having trouble landing CLs that are otherwise green.

Can you post a link to a failing build? Comment 22 was supposed to fix this

https://crrev.com/c/928828 had:

https://ci.chromium.org/buildbot/tryserver.chromium.mac/mac_chromium_rel_ng/655837
https://ci.chromium.org/buildbot/tryserver.chromium.mac/mac_chromium_rel_ng/656908

I looked at one particular failing task: https://chromium-swarm.appspot.com/task?id=3bd7bca9ca79f710

and logged in to the bot:

chrome-bot@build677-m4:(Mac 10.12.6):~$ which otool
/usr/bin/otool
chrome-bot@build677-m4:(Mac 10.12.6):~$ otool -version
xcrun: error: active developer path ("/b/s/w/ir/Xcode.app/Contents/Developer") does not exist
Use `sudo xcode-select --switch path/to/Xcode.app` to specify the Xcode that you wish to use for command line developer tools, or use `xcode-select --install` to install the standalone command line developer tools.
See `man xcode-select` for more details.

This is a fall-out from issue 475693, specifically, Xcode is installed in a named cache inside a task, and 'xcode-select -s <path>' is run to point the system to that cache. However, when a task does *not* request the named cache and doesn't install Xcode, this path is non-existent (by design). Hence, otool is still missing, even though the system has a wrapper script for it.

So, to fix it properly, we should either install Xcode (or the relevant part) in swarming tasks, or set 'xcode-select -s' to an existing system-wide Xcode.

I'm upping the priority, since it's actually an active problem that was not fully resolved by installing system-wide otool.

> So, to fix it properly, we should either install Xcode (or the relevant part) in swarming tasks, or set 'xcode-select -s' to an existing system-wide Xcode.

FWIW, the 10.13.3 bots do have a valid /Library/Developer/CommandLineTools installation (that's what we use to bring the bot up), so the latter could be done against that path.

Raising to P1. Perhaps we should disable this test completely on macOS until a solution is expected to be working.

Handing this one back to Sergey as he's also owning issue 475693

Another flake: https://ci.chromium.org/buildbot/tryserver.chromium.mac/mac_chromium_rel_ng/659617

Swarming: https://chromium-swarm.appspot.com/task?id=3bedad46c6bb8710&refresh=10&show_raw=1

[42594:775:0226/172406.914590:ERROR:render_frame_impl.cc(868)] Intentionally crashing (with null pointer dereference) because user navigated to chrome://crash/
Received signal 11 SEGV_MAPERR 000000000000
0   Content Shell Framework             0x000000010d6ff83c base::debug::StackTrace::StackTrace(unsigned long) + 28
1   Content Shell Framework             0x000000010d6ff671 base::debug::(anonymous namespace)::StackDumpSignalHandler(int, __siginfo*, void*) + 2401
2   libsystem_platform.dylib            0x00007fff79e04f5a _sigtramp + 26
3   ???                                 0x76e07e95fb451453 0x0 + 8565985673889518675
4   Content Shell Framework             0x00000001118c0f72 content::HandleChromeDebugURL(GURL const&) + 834
5   Content Shell Framework             0x00000001118da4c8 content::RenderFrameImpl::HandleRendererDebugURL(GURL const&) + 264
6   Content Shell Framework             0x000000010aec70f2 content::mojom::FrameNavigationControlStubDispatch::Accept(content::mojom::FrameNavigationControl*, mojo::Message*) + 2674
...

Re: #c39: The testsuite is now run in experimental mode: https://chromium.googlesource.com/chromium/src/+/master/testing/buildbot/test_suite_exceptions.pyl#1204
meaning that it will not cause the bot to go red if it fails. This particular build failed for other reasons.

One of my builds where this failed: https://ci.chromium.org/buildbot/tryserver.chromium.mac/mac_chromium_rel_ng/664331

I've noticed a bunch of failures while looking for other (legitimate) failures. Given the (based on personal perception, may be biased) failure rate, I don't think this test should be on the CQ. Even if failures don't make the bot go red, they're a cognitive tax -- every engineer that has a failing test looks through the list and notices this red block, and then has to learn to ignore it.

Can you please look into a solution that gets you the information you want but doesn't show a red block to others? For example, I think the Network Service team has a build bot that only signals failures to them.

the test is not flaky. What is flaky is the CQ (whether or not a working Xcode is deployed on the bot).

In fact, the test catches regressions when it's executed correctly, so I'd rather push on getting back to state where the bots work.

FYI, content_shell_crash_test (experimental) is very consistently failing on the Mac10.13 Tests builder:

https://ci.chromium.org/buildbot/chromium.mac/Mac10.13%20Tests/?limit=200

The error message is the same as reported in #c39. Adding the Sheriff label, so that other Sheriffs are aware of this.

nit: the test intentionally crashes (by design), so the trace in #c39 is WAI:

...Intentionally crashing (with null pointer dereference) because user navigated to chrome://crash/
...

However, this is not WAI:

FAIL: Could not find reference to CrashIntentionally in stack.

which I believe is caused by the missing Xcode reference.

Based on #c3, since components/crash/content/tools/generate_breakpad_symbols.py is the direct user of otool on Mac, I'm thinking of instrumenting it with ensuring that Xcode exists within the task.

As a quick fix, from #c36, DEVELOPER_DIR=/Library/Developer/CommandLineTools seems to do the trick for otool. I'll add that to the script's logic.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/29607010825e977c57661e288fac955cd874dcb0

commit 29607010825e977c57661e288fac955cd874dcb0
Author: Sergey Berezin <sergeyberezin@google.com>
Date: Wed Mar 07 23:18:12 2018

generate_breakpad_symbols: ensure otool path exists on Mac

BUG=813163
TBR=jochen@chromium.org

Change-Id: I1fb5565b978fb38b39fb2ddf9f726f9a3c5bd794
Reviewed-on: https://chromium-review.googlesource.com/953591
Reviewed-by: Sergey Berezin <sergeyberezin@chromium.org>
Commit-Queue: Sergey Berezin <sergeyberezin@chromium.org>
Cr-Commit-Position: refs/heads/master@{#541636}
[modify] https://crrev.com/29607010825e977c57661e288fac955cd874dcb0/components/crash/content/tools/generate_breakpad_symbols.py

The immediate problem is now resolved: the task list shows an abrupt change from mostly red to mostly succeeding tasks: https://chromium-swarm.appspot.com/tasklist?c=name&c=state&c=created_ts&c=duration&c=pending_time&c=pool&c=bot&c=os&et=1520472540000&f=name%3Acontent_shell_crash_test&f=os%3AMac-10.13&l=50&n=true&s=created_ts%3Adesc&st=1520386140000

Lowering the priority for the follow-up longer term fix.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/df5de24758870daf5b214b3480e0c2ff73cd6efa

commit df5de24758870daf5b214b3480e0c2ff73cd6efa
Author: Trent Apted <tapted@chromium.org>
Date: Mon Mar 12 00:46:19 2018

Revert "generate_breakpad_symbols: ensure otool path exists on Mac"

This reverts commit 29607010825e977c57661e288fac955cd874dcb0.

Reason for revert: causes telemetry_perf_tests to fail on mac 10.12 since https://uberchromegw.corp.google.com/i/chromium.mac/builders/Mac10.12%20Tests/builds/10960

error like

Failed to execute "/b/s/w/ir/.swarming_module_cache/vpython/aa03a4/bin/python /b/s/w/ir/components/crash/content/tools/generate_breakpad_symbols.py --binary=/b/s/w/ir/out/Release/Chromium.app/Contents/Versions/67.0.3365.0/Chromium Helper.app/Contents/MacOS/Chromium Helper --symbols-dir=/b/s/w/itcCWvow/tmpsqdLGL/symbols --build-dir=/b/s/w/ir/out/Release"

causing

Unexpected Failures:
* core.stacktrace_unittest.TabStackTraceTest.testCrashMinimalSymbols
* core.stacktrace_unittest.TabStackTraceTest.testCrashSymbols

Original change's description:
> generate_breakpad_symbols: ensure otool path exists on Mac
> 
> BUG=813163
> TBR=jochen@chromium.org
> 
> Change-Id: I1fb5565b978fb38b39fb2ddf9f726f9a3c5bd794
> Reviewed-on: https://chromium-review.googlesource.com/953591
> Reviewed-by: Sergey Berezin <sergeyberezin@chromium.org>
> Commit-Queue: Sergey Berezin <sergeyberezin@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#541636}

TBR=sergeyberezin@chromium.org,jochen@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug: 813163
Change-Id: If21d6769c514e5a037fb3e1d8ed2a20c43f01482
Reviewed-on: https://chromium-review.googlesource.com/958363
Reviewed-by: Trent Apted <tapted@chromium.org>
Commit-Queue: Trent Apted <tapted@chromium.org>
Cr-Commit-Position: refs/heads/master@{#542412}
[modify] https://crrev.com/df5de24758870daf5b214b3480e0c2ff73cd6efa/components/crash/content/tools/generate_breakpad_symbols.py

Thanks for finding this!

I think need to add more candidate paths for a valid Xcode installation. Will try this next.

Meanwhile, I actually expect the revert to make things worse, not better, since if it fails on Mac 10.12, it means there is no Xcode at `xcode-select -p` (likely for the same reason as in the original issue), and /Library/Developer/CommandLineTools is not installed on 10.12 Macs. The original code simply expects Xcode path to be already set, effectively reducing the choices to just `xcode-select -p`.

Raising the priority again, as I'm sure it'll start breaking the other tasks as before.

...and yet, in practice the revert fixed the broken builds:
last broken build: 2018-03-11 5:39 PM (PDT) https://ci.chromium.org/buildbot/chromium.mac/Mac10.12%20Tests/11099
The revert landed 2018-03-11 5:46 PM (PDT)
first good build: 2018-03-11 6:14 PM (PDT)
https://ci.chromium.org/buildbot/chromium.mac/Mac10.12%20Tests/11100

How does that work??.. And why did it actually break then? I wish it printed the actual error...

Actually, it printed an error, yay logging!

"WARNING: no value found for DEVELOPER_DIR. Some commands may fail."

So, yeah, Mac 10.12 needs more candidate paths. I'm still stumped as to how the revert actually fixed anything... Maybe 10.12 can use otool even if Xcode path is invalid??

A sample 10.12 bot (vm1274-m4) has the following two paths:

/Applications/Xcode8.0.app
/Applications/Xcode9.0.app

I'll add those, and /Applications/Xcode.app just for good measure.

The original error is, of course, back - see attachment.

Landing a fix to both now: https://crrev.com/c/958036

Deleted: Screen Shot 2018-03-12 at 3.01.36 PM.png 543 KB

	Screen Shot 2018-03-12 at 3.01.36 PM.png 543 KB View Download

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e818f293daff2ced41f655e18b6154f57166d80b

commit e818f293daff2ced41f655e18b6154f57166d80b
Author: Sergey Berezin <sergeyberezin@google.com>
Date: Mon Mar 12 22:39:14 2018

generate_breakpad_symbols: ensure otool path exists on Mac - take 2

This is a reland of https://crrev.com/c/953591 with a fix.

BUG=813163
TBR=jochen@chromium.org

Reviewed-on: https://chromium-review.googlesource.com/953591
Reviewed-by: Sergey Berezin <sergeyberezin@chromium.org>
Commit-Queue: Sergey Berezin <sergeyberezin@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#541636}
Change-Id: I2683369ef68c198a954488b2aaa7d66eb63c9b2f
Reviewed-on: https://chromium-review.googlesource.com/958036
Cr-Commit-Position: refs/heads/master@{#542632}
[modify] https://crrev.com/e818f293daff2ced41f655e18b6154f57166d80b/components/crash/content/tools/generate_breakpad_symbols.py

Confirmed for both: 

https://ci.chromium.org/buildbot/chromium.mac/Mac10.12%20Tests/11143
https://ci.chromium.org/buildbot/chromium.mac/Mac10.13%20Tests/375

Lowering the priority again for a longer term fix.

This is happening again apparently for every run on mac_chromium_rel_ng which runs content_shell_crash_tests. See:

https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/28191
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/28186
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/28175
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/28174
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/28173
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/28166
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/28164
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/28163

I haven't triaged these new failures. Marking Sheriff-Chromium. Could someone please take a look?

Indeed, there was a huge backlog of mac_chromium_rel_ng during last 5 hours.

On the main waterfall: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac10.13%20Tests?limit=200

We have https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac10.13%20Tests/1757 as the last good run and https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac10.13%20Tests/1758 as the first bad run.

# Run content_shell and make it crash.
./Content Shell.app/Contents/MacOS/Content Shell --run-layout-test chrome://crash --enable-crash-reporter --crash-dumps-dir=/b/s/w/itscGfxl/tmpLCiLao
# Retrieve crash dump.
FAIL: Expected 1 crash dump, found 0.

In the blamelist, there is a Crashpad update - r551183.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/48d065f6b4b21a4c06185f114103eb2556acd1eb

commit 48d065f6b4b21a4c06185f114103eb2556acd1eb
Author: Joshua Peraza <jperaza@chromium.org>
Date: Wed Apr 25 15:22:46 2018

Retrieve crash dumps from pending directory

Crash dumps that haven't been uploaded are now kept in pending, rather
than marking them as completed.
https://chromium-review.googlesource.com/c/crashpad/crashpad/+/1010970

Bug: 813163
Change-Id: Ib77479f6a3f85a3dc6a5685c9f97dd1b8e5540a4
Reviewed-on: https://chromium-review.googlesource.com/1024512
Reviewed-by: Mike West <mkwst@chromium.org>
Commit-Queue: Joshua Peraza <jperaza@chromium.org>
Cr-Commit-Position: refs/heads/master@{#553563}
[modify] https://crrev.com/48d065f6b4b21a4c06185f114103eb2556acd1eb/content/shell/tools/breakpad_integration_test.py

I'm not actually working on this directly. The longer term fix is tracked in issue 814426.

Is there some way to see if the test is working again? I picked a few builds at random on https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac10.13%20Tests?limit=200 and it was fine there.

https://chromium-swarm.appspot.com/tasklist?c=name&c=state&c=created_ts&c=duration&c=pending_time&c=pool&c=bot&c=os&et=1520472540000&f=name%3Acontent_shell_crash_test&f=os%3AMac-10.13&l=50&n=true&s=created_ts%3Adesc&st=1527812940000 shows 1252 successes and only 15 failures in the past week. You can narrow the search further to drill into the failures on that page.

Hey, neat! How did you come up with this URL?

Got it from #c47 :-)

But from scratch, go to chromium-swarm.appspot.com, select "Task List", and then add the constraints from the list in the box, and select the start/end time.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/924c8465fff86ce13f8a4ca4d352807e0616f0b1

commit 924c8465fff86ce13f8a4ca4d352807e0616f0b1
Author: Nico Weber <thakis@chromium.org>
Date: Tue Jun 05 16:30:45 2018

Make generate_breakpad_symbols.py not silently ignore subprocess errors.

GetCommandOuput() used to pipe stderr to /dev/null, and it ignored
the command's return code. Use check_output() to check the return
code, and keep stderr attached to parent's stderr.

Also make breakpad_integration_test.py a bit simpler (this part is
supposed to be behavior-preserving.)

Bug: 813163
Change-Id: I8c55d3da9fff3b944111c3e868121ac34bf65c17
Reviewed-on: https://chromium-review.googlesource.com/1086981
Reviewed-by: Jochen Eisinger <jochen@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#564531}
[modify] https://crrev.com/924c8465fff86ce13f8a4ca4d352807e0616f0b1/components/crash/content/tools/generate_breakpad_symbols.py
[modify] https://crrev.com/924c8465fff86ce13f8a4ca4d352807e0616f0b1/content/shell/tools/breakpad_integration_test.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5a7973799e44d0313bf5a6df2f11f6fb2ef1474a

commit 5a7973799e44d0313bf5a6df2f11f6fb2ef1474a
Author: Ted Choc <tedchoc@chromium.org>
Date: Tue Jun 05 23:51:44 2018

Revert "Make generate_breakpad_symbols.py not silently ignore subprocess errors."

This reverts commit 924c8465fff86ce13f8a4ca4d352807e0616f0b1.

Reason for revert: This appears to break the Android WebView bots.

https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Android%20WebView%20N%20%28dbg%29
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Android%20WebView%20O%20(dbg)

Traceback (most recent call last):
  File "/b/s/w/ir/cache/builder/src/components/crash/content/tools/generate_breakpad_symbols.py", line 327, in <module>
    sys.exit(main())
  File "/b/s/w/ir/cache/builder/src/components/crash/content/tools/generate_breakpad_symbols.py", line 316, in main
    deps = GetSharedLibraryDependencies(options, queue.pop(0), loader_path)
  File "/b/s/w/ir/cache/builder/src/components/crash/content/tools/generate_breakpad_symbols.py", line 151, in GetSharedLibraryDependencies
    deps = GetSharedLibraryDependenciesLinux(binary)
  File "/b/s/w/ir/cache/builder/src/components/crash/content/tools/generate_breakpad_symbols.py", line 67, in GetSharedLibraryDependenciesLinux
    ldd = subprocess.check_output(['ldd', binary])
  File "/b/s/w/ir/cipd_bin_packages/lib/python2.7/subprocess.py", line 219, in check_output
    raise CalledProcessError(retcode, cmd, output=output)

Original change's description:
> Make generate_breakpad_symbols.py not silently ignore subprocess errors.
> 
> GetCommandOuput() used to pipe stderr to /dev/null, and it ignored
> the command's return code. Use check_output() to check the return
> code, and keep stderr attached to parent's stderr.
> 
> Also make breakpad_integration_test.py a bit simpler (this part is
> supposed to be behavior-preserving.)
> 
> Bug: 813163
> Change-Id: I8c55d3da9fff3b944111c3e868121ac34bf65c17
> Reviewed-on: https://chromium-review.googlesource.com/1086981
> Reviewed-by: Jochen Eisinger <jochen@chromium.org>
> Commit-Queue: Nico Weber <thakis@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#564531}

TBR=thakis@chromium.org,jochen@chromium.org

Change-Id: Iee5a81e9b7e9db21f1dda9bdc272d3392438f481
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 813163
Reviewed-on: https://chromium-review.googlesource.com/1087871
Reviewed-by: Ted Choc <tedchoc@chromium.org>
Commit-Queue: Ted Choc <tedchoc@chromium.org>
Cr-Commit-Position: refs/heads/master@{#564707}
[modify] https://crrev.com/5a7973799e44d0313bf5a6df2f11f6fb2ef1474a/components/crash/content/tools/generate_breakpad_symbols.py
[modify] https://crrev.com/5a7973799e44d0313bf5a6df2f11f6fb2ef1474a/content/shell/tools/breakpad_integration_test.py

Issue 850764 has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0cd00af09966bebff27af5a84ab67daf699c3eb7

commit 0cd00af09966bebff27af5a84ab67daf699c3eb7
Author: Nico Weber <thakis@chromium.org>
Date: Fri Jun 08 15:27:29 2018

Reland "Make generate_breakpad_symbols.py not silently ignore subprocess errors."

This reverts commit 5a7973799e44d0313bf5a6df2f11f6fb2ef1474a.

Reason for revert: Relanding without making lld failures critical for now (see https://crbug.com/849904).

Original change's description:
> Revert "Make generate_breakpad_symbols.py not silently ignore subprocess errors."
>
> This reverts commit 924c8465fff86ce13f8a4ca4d352807e0616f0b1.
>
> Reason for revert: This appears to break the Android WebView bots.
>
> https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Android%20WebView%20N%20%28dbg%29
> https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Android%20WebView%20O%20(dbg)
>
> Traceback (most recent call last):
>   File "/b/s/w/ir/cache/builder/src/components/crash/content/tools/generate_breakpad_symbols.py", line 327, in <module>
>     sys.exit(main())
>   File "/b/s/w/ir/cache/builder/src/components/crash/content/tools/generate_breakpad_symbols.py", line 316, in main
>     deps = GetSharedLibraryDependencies(options, queue.pop(0), loader_path)
>   File "/b/s/w/ir/cache/builder/src/components/crash/content/tools/generate_breakpad_symbols.py", line 151, in GetSharedLibraryDependencies
>     deps = GetSharedLibraryDependenciesLinux(binary)
>   File "/b/s/w/ir/cache/builder/src/components/crash/content/tools/generate_breakpad_symbols.py", line 67, in GetSharedLibraryDependenciesLinux
>     ldd = subprocess.check_output(['ldd', binary])
>   File "/b/s/w/ir/cipd_bin_packages/lib/python2.7/subprocess.py", line 219, in check_output
>     raise CalledProcessError(retcode, cmd, output=output)
>
> Original change's description:
> > Make generate_breakpad_symbols.py not silently ignore subprocess errors.
> >
> > GetCommandOuput() used to pipe stderr to /dev/null, and it ignored
> > the command's return code. Use check_output() to check the return
> > code, and keep stderr attached to parent's stderr.
> >
> > Also make breakpad_integration_test.py a bit simpler (this part is
> > supposed to be behavior-preserving.)
> >
> > Bug: 813163
> > Change-Id: I8c55d3da9fff3b944111c3e868121ac34bf65c17
> > Reviewed-on: https://chromium-review.googlesource.com/1086981
> > Reviewed-by: Jochen Eisinger <jochen@chromium.org>
> > Commit-Queue: Nico Weber <thakis@chromium.org>
> > Cr-Commit-Position: refs/heads/master@{#564531}
>
> TBR=thakis@chromium.org,jochen@chromium.org
>
> Change-Id: Iee5a81e9b7e9db21f1dda9bdc272d3392438f481
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Bug: 813163
> Reviewed-on: https://chromium-review.googlesource.com/1087871
> Reviewed-by: Ted Choc <tedchoc@chromium.org>
> Commit-Queue: Ted Choc <tedchoc@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#564707}

TBR=thakis@chromium.org,tedchoc@chromium.org,jochen@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug: 813163, 850055 
Change-Id: I3b241dffac522af75c46dc1a7554bd954b2a8f57
Reviewed-on: https://chromium-review.googlesource.com/1092671
Commit-Queue: Nico Weber <thakis@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#565638}
[modify] https://crrev.com/0cd00af09966bebff27af5a84ab67daf699c3eb7/components/crash/content/tools/generate_breakpad_symbols.py
[modify] https://crrev.com/0cd00af09966bebff27af5a84ab67daf699c3eb7/content/shell/tools/breakpad_integration_test.py

sergeyberenzin: I tried clicking "Completed (failure): 22" hoping that would show me a list of failed tasks (https://chromium-swarm.appspot.com/tasklist?c=name&c=state&c=created_ts&c=duration&c=pending_time&c=pool&c=bot&c=os&et=1528742220000&f=state%3ACOMPLETED_FAILURE&f=name%3Acontent_shell_crash_test&f=os%3AMac-10.13&l=50&n=true&s=created_ts%3Adesc&st=1527812940000) -- but nothing every loads in the table. How do I get a list of recent tasks where this test failed?

Hmm... I also can't load any tasks. The query must be timing out for some reason, possibly because it's not implemented very efficiently. 

I'd check with maruel@ - maybe there is a trick to fetch these tasks?

https://chromium-swarm.appspot.com/tasklist?c=name&c=state&c=created_ts&c=duration&c=pending_time&c=pool&c=bot&c=os&et=1528742220000&f=state%3ACOMPLETED_FAILURE&f=os%3AMac-10.13&f=name-tag%3Acontent_shell_crash_test&l=50&n=true&s=created_ts%3Adesc&st=1527812940000 worked but the -tag suffix is really not obvious.

+Kevin for ideas, especially with the 'name' tag.

+Kevin for real

Idea: refuse to let name:foo be a valid filter?  
second idea: autocorrect name to name-tag if it's on the list of tags we know about?

I prefer the second idea. Otherwise it's unintuitive. In hindsight, that's why I think I would have preferred tags to be the default and having the rest of the special things to have a suffix or something unique.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d5296cb77eccd89a3591115c694b58205203e43b

commit d5296cb77eccd89a3591115c694b58205203e43b
Author: Nico Weber <thakis@chromium.org>
Date: Fri Jun 29 18:56:44 2018

Enable content_shell_crash_test on many bots.

On most bots, we don't know why the test is disabled. Worst case, we'll find
out and redisable it on bots where it doesn't yet work for some reason.

On Mac 10.13 and Win 7, we do know why it's disabled, but the Win 7 bug
has been fixed, and the 10.13 bug was spurious and I have put in better
debug logging if it were to fail again, and I haven't seen any failures in
the last 200 builds on the main waterfall.

Bug:  846729 ,813163,843511
Change-Id: Ibd14dea1c5fb72bc4d298b53f55e87be05aa211e
Reviewed-on: https://chromium-review.googlesource.com/1120225
Commit-Queue: Nico Weber <thakis@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#571572}
[modify] https://crrev.com/d5296cb77eccd89a3591115c694b58205203e43b/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/d5296cb77eccd89a3591115c694b58205203e43b/testing/buildbot/chromium.linux.json
[modify] https://crrev.com/d5296cb77eccd89a3591115c694b58205203e43b/testing/buildbot/chromium.mac.json
[modify] https://crrev.com/d5296cb77eccd89a3591115c694b58205203e43b/testing/buildbot/chromium.win.json
[modify] https://crrev.com/d5296cb77eccd89a3591115c694b58205203e43b/testing/buildbot/client.v8.chromium.json
[modify] https://crrev.com/d5296cb77eccd89a3591115c694b58205203e43b/testing/buildbot/test_suite_exceptions.pyl