Also note that I nothing in the WebRTC and Chromium blamelists for the first failure looks suspicious to me.

This is indeed really weird, but those two waterfalls starting to fail at the same time sends a very strong signal that it's something about a commit in Chromium. It could sometimes also be an infra commit but I don't see anything there.

Here are those two builds again, for visibility
https://ci.chromium.org/buildbot/chromium.webrtc/Mac%20Tester/82482
https://ci.chromium.org/buildbot/chromium.webrtc.fyi/Mac%20Tester/53647

sweilun@, could you confirm or deny the possibility of your CL affecting browser_tests on Mac?
https://chromium-review.googlesource.com/c/chromium/src/+/1181684

In any case... could we get a speculative revert? Just so we don't go looking for a problem where there isn't one...

Then you can see if the next build succeeds on these two:
https://ci.chromium.org/buildbot/chromium.webrtc/Mac%20Tester/
https://ci.chromium.org/buildbot/chromium.webrtc.fyi/Mac%20Tester/

guidou@: https://chromium-review.googlesource.com/1181684 was reverted & relanded, so it looks like that was determined to not be the cause?

(I don't know much about WebRTC tests, but the changes in 1181684 seem unlikely to cause issues there).

ramyan@: That is correct. It was a speculative revert since it was the only CL in the blamelist, but the revert did not fix the bots and the CL was relanded.

Thanks for the confirmation guidou@!

I was able to reproduce the test failures locally with a ToT of today. Then I started to bisect to find where the problem started, but was never able to reproduce again, even with the version that had initially failed.

Maybe this was a temporary tool issue (perhaps some compiler bug) that got fixed at some point and the bots are still using the faulty one?

WRT to WebRTC rolls into Chrome, perhaps we should start a manual one, or disable the bots so the autoroller can resume automatic ones.

Also note that I didn't find any CLs around the time when the failure started related with the stack traces I was getting when I was able to reproduce.

I looked if there's any way to disable crashpad, but I couldn't find a convenient flag for it.

We have plenty of tests that run the whole chrome browser, so I wonder why the apprtc tests in particular are affected. One guess is that we invoke browser_tests twice on these bots in two different build steps, and the first invocation gets the crash reporter binary into a weird state.

Yes, I can see that the large tests bot fails in the same way: if we invoke browser_tests in different build steps on the same bot, all steps but the first fail.

I can't see any changes in recipes, breakpad or crashpad around Aug 22... https://chromium-review.googlesource.com/1181684 is the only one in the chromium blamelist, and it's not that one.

Also unexplained is the content browsertest crashes.

I can see the failed build and successful one differs by one //build CL for chromium.webrtc:
https://chromium.googlesource.com/chromium/tools/build/+/dc454099775dcec7ba4c31dd0fb0eb4c9447b9e2. For FYI, however, //build stays the same so it's most likely not that one.

The error is execvp /b/c/b/Mac_Tester/src/out/Release/Chromium.app/Contents/Versions/70.0.3531.0/Chromium Framework.framework/Helpers/crashpad_handler: No such file or directory (2)

I wonder if the path gets misconstructed or if the binary is accidentally deleted or something.

I think the code to put the handler there in the first place is here: https://cs.chromium.org/chromium/src/chrome/BUILD.gn?type=cs&q=%22crashpad_handler%22&sq=package:chromium&g=0&l=866

When I logged into the bot, I could see everything in the path execvp /b/c/b/Mac_Tester/src/out/Release/Chromium.app/Contents/Versions/70.0.X.X/Chromium Framework.framework/ is there, but not Helpers/crashpad_handler. Looks like the code that asserts is WAI.

For chromium.webrtc it seems the binary is gone when the tests start executing and for fyi it goes away after the first browser_tests invocation.

Aha, I did find the binary here in a slightly different path though:

/b/c/b/Mac_Tester/src/out/Release/Chromium.app/Contents/Versions/70.0.3535.0/Chromium\ Framework.framework/Versions/A/Helpers/crashpad_handler

The path is constructed here in the .cc code: https://cs.chromium.org/chromium/src/components/crash/content/app/crashpad_mac.mm?type=cs&sq=package:chromium&g=0&l=123

All right, another theory: AppRTC tests and perf tests happen to be the only ones that aren't swarmed on that bot, so the swarming tasks obviously get the right build, but the tests that run on the bot itself doesn't. I'll look closer at the build that comes with the bot and compare with what the swarmed shards are getting.

The full mac build has the same dir problem as described in #16 and #17, but the swarmed equivalent DOES have out/Release/Chromium.app/Contents/Versions/70.0.3531.0/Chromium\ Framework.framework/Helpers/crashpad_handler.

So this is a build packaging problem.

I can indeed see that the build https://ci.chromium.org/buildbot/chromium.webrtc/Mac%20Builder/101166 does have a functioning Helpers/breakpad. It turns out Helpers is actually a symlink into Versions/Current/Helpers/. 

I can see breakpad is the same between https://logs.chromium.org/v/?s=chromium%2Fbb%2Fchromium.webrtc%2FMac_Builder%2F101166%2F%2B%2Frecipes%2Fsteps%2Fpackage_build%2F0%2Fstdout (OK) and https://logs.chromium.org/v/?s=chromium%2Fbb%2Fchromium.webrtc%2FMac_Builder%2F101167%2F%2B%2Frecipes%2Fsteps%2Fpackage_build%2F0%2Fstdout (BAD), however "Chromium Framework.framework/Versions/Current" is added in the good version and not the bad one. New theory: the Versions/Current symlink has been broken somehow in packaged builds.

Yeah, the bad one has Versions/A and the good one has

Versions/
  A/
  Current -> A/

Aha, this one is most likely the problem: https://chromium.googlesource.com/chromium/tools/build/+/dc454099775dcec7ba4c31dd0fb0eb4c9447b9e2

I'm going to speculatively revert it for now so we know if it's the culprit or not.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/9a41d66725951f98d7c3a425e418ca8155abf00e

commit 9a41d66725951f98d7c3a425e418ca8155abf00e
Author: Patrik Höglund <phoglund@chromium.org>
Date: Mon Aug 27 10:16:16 2018

Revert "zip_build: Apply the path filter recursively when creating a package archive."

This reverts commit dc454099775dcec7ba4c31dd0fb0eb4c9447b9e2.

Reason for revert: Speculative revert: looks like this breaks WebRTC build packaging for browser_tests.

Original change's description:
> zip_build: Apply the path filter recursively when creating a package archive.
>
> Previously we would only apply it to the direct descendants of the
> build directory and not the contents of any of its subdirectories. This
> meant that, for example, we ended up packaging the 'obj' and
> 'thinlto-cache' directories from the build directories for toolchains
> other than the default toolchain, despite these directories being
> filtered out.
>
> We previously had code that handled the 'initial' directory as a
> special case where the filter was also being applied to the files
> in that directory. It seems like this code is now dead because the
> directory is now named 'initialexe' (?), but in any event, since we
> now have a generalization of that code, I've removed it.
>
> Bug:  876316 
> Change-Id: Ieab27788bf3ca7c7bf970434ac491a053eaa5baf
> Reviewed-on: https://chromium-review.googlesource.com/1184302
> Reviewed-by: Nico Weber <thakis@chromium.org>
> Commit-Queue: Peter Collingbourne <pcc@chromium.org>

TBR=thakis@chromium.org,dpranke@chromium.org,pcc@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug:  876316 ,  876743 
Change-Id: Ie229755a4836e6e80d896801535b3d5ad8a98650
Reviewed-on: https://chromium-review.googlesource.com/1189962
Commit-Queue: Patrik Höglund <phoglund@chromium.org>
Reviewed-by: Patrik Höglund <phoglund@chromium.org>

[modify] https://crrev.com/9a41d66725951f98d7c3a425e418ca8155abf00e/scripts/slave/zip_build.py
[modify] https://crrev.com/9a41d66725951f98d7c3a425e418ca8155abf00e/scripts/common/chromium_utils.py

And it greens up! Yay!

pcc: The problem here was that a symlink doesn't get zipped into the mac release build.

Before: out/Release/Chromium.app/Contents/Versions/70.0.X.X/Chromium Framework.framework/Helpers/ -> Chromium.app/Contents/Versions/70.0.X.X/Chromium Framework.framework/Versions/Current/Helpers

After your patch: 
out/Release/Chromium.app/Contents/Versions/70.0.X.X/Chromium Framework.framework/Helpers/ MISSING
Chromium.app/Contents/Versions/70.0.X.X/Chromium Framework.framework/Versions/Current MISSING

I think both the Helpers symlink itself and Versions/Current go missing, so the path out/Release/Chromium.app/Contents/Versions/70.0.X.X/Chromium Framework.framework/Helpers/crashpad_handler fails to resolve.

This obviously works on swarming. Unfortunately we have some tests that aren't converted to swarming. I plan to do that over the coming months but I can try to accelerate those plans if it's hard for you to fix+reland your CL.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/d550bb44e12759d593d991a7a6f916af5d90b414

commit d550bb44e12759d593d991a7a6f916af5d90b414
Author: Peter Collingbourne <pcc@google.com>
Date: Tue Aug 28 16:47:23 2018

Reland "zip_build: Apply the path filter recursively when creating a package archive."

This relands commit dc454099775dcec7ba4c31dd0fb0eb4c9447b9e2.

The original commit introduced a bug where symlinks that point to
directories were omitted from the archive. The cause was that for
some reason os.walk returns symbolic links that point to directories
in dirs and not files, so we need to handle any symlinks in dirs as
if they appeared in files instead.

Original change's description:
> Revert "zip_build: Apply the path filter recursively when creating a package archive."
>
> This reverts commit dc454099775dcec7ba4c31dd0fb0eb4c9447b9e2.
>
> Reason for revert: Speculative revert: looks like this breaks WebRTC build packaging for browser_tests.
>
> Original change's description:
> > zip_build: Apply the path filter recursively when creating a package archive.
> >
> > Previously we would only apply it to the direct descendants of the
> > build directory and not the contents of any of its subdirectories. This
> > meant that, for example, we ended up packaging the 'obj' and
> > 'thinlto-cache' directories from the build directories for toolchains
> > other than the default toolchain, despite these directories being
> > filtered out.
> >
> > We previously had code that handled the 'initial' directory as a
> > special case where the filter was also being applied to the files
> > in that directory. It seems like this code is now dead because the
> > directory is now named 'initialexe' (?), but in any event, since we
> > now have a generalization of that code, I've removed it.
> >
> > Bug:  876316 
> > Change-Id: Ieab27788bf3ca7c7bf970434ac491a053eaa5baf
> > Reviewed-on: https://chromium-review.googlesource.com/1184302
> > Reviewed-by: Nico Weber <thakis@chromium.org>
> > Commit-Queue: Peter Collingbourne <pcc@chromium.org>
>
> TBR=thakis@chromium.org,dpranke@chromium.org,pcc@chromium.org
>
> # Not skipping CQ checks because original CL landed > 1 day ago.
>
> Bug:  876316 ,  876743 
> Change-Id: Ie229755a4836e6e80d896801535b3d5ad8a98650
> Reviewed-on: https://chromium-review.googlesource.com/1189962
> Commit-Queue: Patrik Höglund <phoglund@chromium.org>
> Reviewed-by: Patrik Höglund <phoglund@chromium.org>

Bug:  876316 ,  876743 
Change-Id: Ia6c329c797374e24d6bcf43d0cadd6517deb27b8
Reviewed-on: https://chromium-review.googlesource.com/1192234
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Peter Collingbourne <pcc@chromium.org>

[modify] https://crrev.com/d550bb44e12759d593d991a7a6f916af5d90b414/scripts/slave/zip_build.py
[modify] https://crrev.com/d550bb44e12759d593d991a7a6f916af5d90b414/scripts/common/chromium_utils.py