I think that this bot just got sick, either in the test environment or for the login session/boot, and that it cleared up when it moved on to other work or was rebooted.

124 is indeed an unknown error code. It seems like that’s the handiest thing to go on here, but it’s not telling me anything useful. It’s probably coming from launchd (the bootstrap server), but we don’t have any insight into that anymore since it’s closed-source. It’s not coming from the mig stub side of bootstrap_check_in() or anything in the kernel, as far as I can tell.

Anyway, I don’t think that this has happened again. Are you seeing this as a persistent problem?

No - we've only seen it once. I'll WontFix this bug, and if it reappears I'll reopen it.

https://logs.chromium.org/v/?s=chromium%2Fbb%2Fchromium.mac%2FMac10.11_Tests%2F24820%2F%2B%2Frecipes%2Fsteps%2Fbrowser_tests_on_Mac-10.11%2F0%2Fstdout has as bunch of similar-looking errors:


[ RUN      ] JavscriptApiTest.JavasScriptEncodedURL
[0326/155742.122684:ERROR:mach_extensions.cc(68)] bootstrap_check_in org.chromium.crashpad.child_port_handshake.13989.509792.ASETEXTGXUTJAOFT: unknown error code (141)
[0326/155742.122802:FATAL:child_port_handshake.cc(111)] Check failed: server_port.is_valid().
0   crashpad_handler                    0x000000010a01031c base::debug::StackTrace::StackTrace(unsigned long) + 28
1   crashpad_handler                    0x000000010a01545f logging::LogMessage::~LogMessage() + 223
2   crashpad_handler                    0x000000010a0302dd crashpad::ChildPortHandshake::RunServerForFD(base::ScopedGeneric<int, base::internal::ScopedFDCloseTraits>, crashpad::ChildPortHandshake::PortRightType) + 413
3   crashpad_handler                    0x000000010a002086 crashpad::HandlerMain(int, char**, std::__1::vector<std::__1::unique_ptr<crashpad::UserStreamDataSource, std::__1::default_delete<crashpad::UserStreamDataSource> >, std::__1::allocator<std::__1::unique_ptr<crashpad::UserStreamDataSource, std::__1::default_delete<crashpad::UserStreamDataSource> > > > const*) + 4790
4   libdyld.dylib                       0x00007fff97a0f5ad start + 1
[0326/155742.123312:ERROR:file_io.cc(89)] ReadExactly: expected 4, observed 0
[0326/155742.123783:ERROR:mach_extensions.cc(68)] bootstrap_check_in org.chromium.crashpad.child_port_handshake.13987.509790.ZRDPWDQQCDUVWVHU: unknown error code (141)
[0326/155742.123853:FATAL:child_port_handshake.cc(111)] Check failed: server_port.is_valid().
0   crashpad_handler                    0x000000010562b31c base::debug::StackTrace::StackTrace(unsigned long) + 28
1   crashpad_handler                    0x000000010563045f logging::LogMessage::~LogMessage() + 223
2   crashpad_handler                    0x000000010564b2dd crashpad::ChildPortHandshake::RunServerForFD(base::ScopedGeneric<int, base::internal::ScopedFDCloseTraits>, crashpad::ChildPortHandshake::PortRightType) + 413
3   crashpad_handler                    0x000000010561d086 crashpad::HandlerMain(int, char**, std::__1::vector<std::__1::unique_ptr<crashpad::UserStreamDataSource, std::__1::default_delete<crashpad::UserStreamDataSource> >, std::__1::allocator<std::__1::unique_ptr<crashpad::UserStreamDataSource, std::__1::default_delete<crashpad::UserStreamDataSource> > > > const*) + 4790
4   libdyld.dylib                       0x00007fff97a0f5ad start + 1
[13985:775:0326/155742.125375:1927348692830:ERROR:file_io.cc(89)] ReadExactly: expected 4, observed 0
Received signal 11 SEGV_MAPERR 000000000008

The problem starts when bootstrap_check_in() fails. The original report has

[0921/040719.186722:ERROR:mach_extensions.cc(68)] bootstrap_check_in org.chromium.crashpad.child_port_handshake.58700.2141078.LXSVGGPFZLMWBIRD: unknown error code (124)

and comment 3 has

[0326/155742.122684:ERROR:mach_extensions.cc(68)] bootstrap_check_in org.chromium.crashpad.child_port_handshake.13989.509792.ASETEXTGXUTJAOFT: unknown error code (141)

But I don’t know what 124 or 141 are either. They’re not bootstrap return codes (which would be in the 1100 range), they’re low-ish like Mach kern_return_t values (the range 0-255 is reserved for these) but don’t map to any defined code (defined codes only go up to 52). And it’s weird that we’re seeing this bug with (at least) two different unknown error codes, but that may be a 10.11–10.12 difference.

Robert, do those numbers, 124 and 141, make any sense in XPC context?

Yeah, they might. It's hard because error codes have become so overloaded (as you note, kern_return_t, errno, etc), but my guess is that they correspond to these:

rsesek@hotwire:/Users/rsesek % launchctl error 124
124: Domain is tearing down
rsesek@hotwire:/Users/rsesek % launchctl error 141
141: Reentrancy avoided

This may be a red herring, but I think there may be something odd with the bootstrap context on another bot, per https://bugs.chromium.org/p/chromium/issues/detail?id=817663#c11. Though we're not seeing those specific errors.

It does seem like there’s some real badness going on with the bootstrap server. Comment 3 is talking about https://ci.chromium.org/buildbot/chromium.mac/Mac10.11%20Tests/, which is always scheduled on https://build.chromium.org/deprecated/chromium.mac/buildslaves/build179-m1. Can we get that bot rebooted?

+Infra-Troopers for a trooper to try rebooting build179-m1.

build179-m1 dispatches the tests to swarming and it's some of the swarming slaves that are dying.

I wonder if what you are seeing is related to crbug.com/825215?

Yes, that adds up. If the test process is descended from something that started in a UI session, and the UI session disappears (such as a WindowServer crash), then the bootstrap context should be lost and I’d expect all bootstrap_*() calls to fail.

I wouldn’t normally expect suspicious error codes, but now that there’s just one launchd, the bootstrap port won’t really die. bootstrap_check_in() will be able to communicate with it, and launchd may be responding with some weirdo number because the bootstrap context associated with the port is gone.

This is compelling enough that I’m marking bug 825215 as blocking.

Issue 826159 has been merged into this issue.

Bringing up the priority as Mac10.11 Tests bot has been down for > 10 days.

Are there any changes we can revert in the meantime?

Also see  issue 828031 .

No, This bug is a symptom, not a cause. It’s also not the only symptom.

Please follow up on bug 825215, which we believe to underlie all of these problems.

I just started today's sheriff and still not sure how it's going.
Let me share the status: Mac 10.11 Tests is continuously failing from #25114 to #25132.

https://uberchromegw.corp.google.com/i/chromium.mac/builders/Mac10.11%20Tests?numbuilds=200

Still seeing these failures on Mac 10.11. 

This bug was originally filed against 10.12 and was WontFixed, but was reopened as 10.11 failures in comment #3. Can we either update the summary or (preferably) just WontFix this again and deal with the 10.11 problem in bug 825215?

reassigning from mark@, since his description says "on leave".

 Issue 840809  has been merged into this issue.

 Issue 852465  has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7220ee517c622e836d57f36fec3a6af4a32bbd6c

commit 7220ee517c622e836d57f36fec3a6af4a32bbd6c
Author: John Budorick <jbudorick@chromium.org>
Date: Thu Jun 14 03:35:25 2018

Mac10.13 Tests: drop browser_tests to experimental and disable window server suspects.

TBR=ellyjones@chromium.org

Bug: 767397
Change-Id: I88764b5dad273ec9dbea975cc6e97ce799ce6fe1
Reviewed-on: https://chromium-review.googlesource.com/1100296
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#567119}
[modify] https://crrev.com/7220ee517c622e836d57f36fec3a6af4a32bbd6c/testing/buildbot/chromium.mac.json
[modify] https://crrev.com/7220ee517c622e836d57f36fec3a6af4a32bbd6c/testing/buildbot/test_suite_exceptions.pyl

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/deda551f496ebb23f44d75fb241fa22ef41088f3

commit deda551f496ebb23f44d75fb241fa22ef41088f3
Author: John Budorick <jbudorick@chromium.org>
Date: Fri Jun 15 19:28:09 2018

Implement --gtest_shuffle in //base/test/launcher.

We were previously passing --gtest_shuffle down to gtest. However,
in some cases (e.g. browser_tests, or other suites using
content::TestLauncherDelegate), gtest wasn't seeing the flag until
it was already in a test-specific worker process -- i.e., until
it only had one test to shuffle.

This moves the shuffling into the test launcher, before we hand
off to the delegate, ensuring that we shuffle across the entire shard.
The implementation is largely based on gtest's: reproducible given a
seed, w/ a seed range of [0, 100000).

This also uses --gtest_shuffle on browser_tests on Mac in an attempt
to collect more data on what's causing VM WindowServer deaths there.

Bug: 767397
Change-Id: I8aced17cdc5a574e3dc127143c0d0a553f62bf62
Reviewed-on: https://chromium-review.googlesource.com/1100966
Reviewed-by: Nico Weber <thakis@chromium.org>
Reviewed-by: Elly Fong-Jones <ellyjones@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#567763}
[modify] https://crrev.com/deda551f496ebb23f44d75fb241fa22ef41088f3/base/test/launcher/test_launcher.cc
[modify] https://crrev.com/deda551f496ebb23f44d75fb241fa22ef41088f3/base/test/launcher/test_launcher.h
[modify] https://crrev.com/deda551f496ebb23f44d75fb241fa22ef41088f3/testing/buildbot/chromium.mac.json
[modify] https://crrev.com/deda551f496ebb23f44d75fb241fa22ef41088f3/testing/buildbot/test_suite_exceptions.pyl

Still seeing these errors in browser_tests on Mac-10.12, e.g.:
https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket.appspot.com/8935144597685416224/+/steps/browser_tests_on__none__GPU_on_Mac_on_Mac-10.12.6/0/stdout

 Issue 862466  also mentions crashes on Mac-10.13 at the end of August.