I don't see anything suspicious in the CLs in the build between the last success until the first size failure.

I noticed however that in the success builds the line

otool: error: unable to find utility "otool", not a developer tool or in PATH

which doesn't appear in the later failure builds.

Adding Infra-Troopers to have a look if this is an infra issue.

Not an infra issue, it's a legitimate test error.

 Issue 832559  has been merged into this issue.

sullivan@, could you please take a look at it?
I am not sure if it's Perf related, but this is my surest guess!

--Build Sheriff.

The static initializers part of the sizes test isn't own by speed ops, but I know the goal of the test is to keep static initializers from the codebase. Here is the log from the failure:

FAILED mac-release/sizes/chrome-si/initializers: actual 4, expected 0, better lower

+thakis who I think may know a bit more about this test.

It's a little confusing to find the right CL range because the build broke then was fixed then sizes failed, but I think the CL range is this: https://chromium.googlesource.com/chromium/src/+log/ed05d62660b3fd83ed7724fbe5609b529f9e31d1%5E..282cdeeb8baa5b52823ee0ea0cc045cba31e34e1?pretty=fuller&n=

rsesek@: do you know anything about this?

The bot output includes:

# Static initializers in /b/c/b/mac/src/out/Release/Chromium Framework.framework/Chromium Framework:
# HINT: To get this list, run tools/mac/show_mod_init_func.py
# HINT: diff against the log from the last run to see what changed
/b/c/b/mac/src/out/Release/Chromium Framework.unstripped
0x00000000020cada0 @ _GLOBAL__sub_I_message_pump_mac.mm (in Chromium Framework.unstripped) + 0
0x00000000027265a0 @ _GLOBAL__sub_I_atomicops_internals_x86_gcc.cc (in Chromium Framework.unstripped) + 0

AFAICT this is real.

It looks like the bots didn't have otool installed, so sizes has been accidentally passing:

otool: error: unable to find utility "otool", not a developer tool or in PATH
RESULT chrome-si: initializers= 0 files

There have been otool issues, and it seems that infra finally got it installed on all the machines, and now the sizes step is finding the initializers:

# Static initializers in /b/c/b/mac/src/out/Release/Chromium Framework.framework/Chromium Framework:
# HINT: To get this list, run tools/mac/show_mod_init_func.py
# HINT: diff against the log from the last run to see what changed
/b/c/b/mac/src/out/Release/Chromium Framework.unstripped
0x00000000020cada0 @ _GLOBAL__sub_I_message_pump_mac.mm (in Chromium Framework.unstripped) + 0
0x00000000027265a0 @ _GLOBAL__sub_I_atomicops_internals_x86_gcc.cc (in Chromium Framework.unstripped) + 0
RESULT chrome-si: initializers= 4 files

Two steps here:
1. Remove the initializers that snuck in
2. Fix the sizes step to fail if it can't determine the number of initializers.

The message pump one is possibly kAllModes in message_pump_mac.mm. It's wrapped by #pragma clang diagnostic ignored "-Wglobal-constructors" but that wouldn't do anything for the sizes step.

The atomicops one is, well, atomicops_internals_x86_gcc.cc in the protobuf library, which explicitly has an initializer and knows it:

// A global to get use initialized on startup via static initialization :/

:/ indeed.

Yup, avi@ assessed correctly, so passing to him.

Looking at the module initializer script, we're using subprocess.check_output(), so I'm wondering if Apple's tools stubs don't exit non-zero if it cannot find the installed binary:

https://cs.chromium.org/chromium/src/tools/mac/show_mod_init_func.py?q=show_mod_init_func.py&sq=package:chromium&dr=C&l=43

Someone needs to do cleanup here. I'm sheriff today and am trying to green the tree.

I filed a  bug 832854 . We should not have a step that can red the tree with no ability to fix it.

Um, thanks for the bug. I will return to it when my plate isn't full of burning bots.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/00d3a24514b59d9628b2074ed2f2d82d4d1953de

commit 00d3a24514b59d9628b2074ed2f2d82d4d1953de
Author: Avi Drissman <avi@chromium.org>
Date: Fri Apr 13 19:45:37 2018

Set the expected number of initializers to 2 on the Mac.

This was broken for a very long time and the initializers regressed.
There was a tool to update the perf_expectations.json file, but it
is either broken or has zero documentation. The perf team disavows
it. This is unmaintained, so I'm manually hacking this file to get
it to a state in which it works.

BUG= 831951 ,  832854 
NOTRY=true
NOPRESUBMIT=true
TBR=thakis@chromium.org

Change-Id: Idf1bd61d40540d0b85497e6669297fb54c2c2a28
Reviewed-on: https://chromium-review.googlesource.com/1012743
Commit-Queue: Avi Drissman <avi@chromium.org>
Reviewed-by: Avi Drissman <avi@chromium.org>
Cr-Commit-Position: refs/heads/master@{#550723}
[modify] https://crrev.com/00d3a24514b59d9628b2074ed2f2d82d4d1953de/tools/perf_expectations/perf_expectations.json

https://cs.chromium.org/chromium/src/third_party/protobuf/src/google/protobuf/stubs/atomicops_internals_x86_gcc.cc?q=atomicops_internals_x86_gcc.cc&sq=package:chromium&dr&l=127

Looks like this is one of the culprits.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/bc0d927cd391a0345979b6b12c58019f9a29bcb2

commit bc0d927cd391a0345979b6b12c58019f9a29bcb2
Author: Dirk Pranke <dpranke@chromium.org>
Date: Fri Apr 13 23:00:22 2018

Bump mac static initializer count to 4 to work around bug.

It looks like when we're computing the number of static
initializers on mac we're still assuming a 32-bit word and not the
64-bit words that we actually use, and so we mis-report the
number of initializers as twice what they actually are.

All of this perf expectation code needs to be ripped out and
replaced with some dumb static checks, but for now, this CL
just updates the expected number from 2 (the correct number) to
4, and updates the checksums in the json file to get the presubmit
to pass.

BUG= 831951 ,  832854 
TBR=avi@chromium.org, wolenetz@chromium.org, thakis@chromium.org
NOTRY=true
NOTREECHECKS=true

Change-Id: Ie6de49af3cc1443b8acc36ce80946ea50e5a76ab
Reviewed-on: https://chromium-review.googlesource.com/1013205
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#550768}
[modify] https://crrev.com/bc0d927cd391a0345979b6b12c58019f9a29bcb2/tools/perf_expectations/make_expectations.py
[modify] https://crrev.com/bc0d927cd391a0345979b6b12c58019f9a29bcb2/tools/perf_expectations/perf_expectations.json

Now that we have a way to turn the bot green this is P2.

I think this should stay as P1; it sounds like this is an obvious regression that should be relatively easily dealt with?

I'm happy to de-assign it and let it go through the normal mac triage though?

Not insistent on the priority either way.

The run loop one will be easy; the protobuf one much less so. Fixing the script to watch for failure in otool is gonna have to be someone who knows python.

I'm sheriff on Monday and then out for two weeks. I can pick it up then, though assistance would be appreciated.

I have a fix in the works for the message loop one. I have no idea how to remove the protobuf one.

(This was broken for half a year at least?)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/909d320df8b095257b4a47d1ec6b0df4559d978a

commit 909d320df8b095257b4a47d1ec6b0df4559d978a
Author: Avi Drissman <avi@chromium.org>
Date: Mon Apr 16 23:49:11 2018

Remove the static initializer from message_pump_mac.mm.

Static initializers are not allowed.

BUG= 831951 

Change-Id: I1c1d41ea66a34d38119c639c2af48183b9d1ab64
Reviewed-on: https://chromium-review.googlesource.com/1012870
Commit-Queue: Avi Drissman <avi@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#551175}
[modify] https://crrev.com/909d320df8b095257b4a47d1ec6b0df4559d978a/base/message_loop/message_pump_mac.mm
[modify] https://crrev.com/909d320df8b095257b4a47d1ec6b0df4559d978a/tools/perf_expectations/README.txt
[modify] https://crrev.com/909d320df8b095257b4a47d1ec6b0df4559d978a/tools/perf_expectations/perf_expectations.json

To Tom Anderson for third_party/protobuf.

Tom: static initializers aren't allowed. The only one in Chrome Mac is the one in third_party/protobuf, https://cs.chromium.org/chromium/src/third_party/protobuf/src/google/protobuf/stubs/atomicops_internals_x86_gcc.cc?l=127

Can you help us get rid of it?

It's likely the patch file needs to be updated
https://cs.chromium.org/chromium/src/third_party/protobuf/patches/0003-remove-static-initializers.patch

Awesome. Refer to https://chromium-review.googlesource.com/1012870 on how to adjust the expected value of the static initializers to 0, or ask dpranke for help.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/00d3a24514b59d9628b2074ed2f2d82d4d1953de

commit 00d3a24514b59d9628b2074ed2f2d82d4d1953de
Author: Avi Drissman <avi@chromium.org>
Date: Fri Apr 13 19:45:37 2018

Set the expected number of initializers to 2 on the Mac.

This was broken for a very long time and the initializers regressed.
There was a tool to update the perf_expectations.json file, but it
is either broken or has zero documentation. The perf team disavows
it. This is unmaintained, so I'm manually hacking this file to get
it to a state in which it works.

BUG= 831951 ,  832854 
NOTRY=true
NOPRESUBMIT=true
TBR=thakis@chromium.org

Change-Id: Idf1bd61d40540d0b85497e6669297fb54c2c2a28
Reviewed-on: https://chromium-review.googlesource.com/1012743
Commit-Queue: Avi Drissman <avi@chromium.org>
Reviewed-by: Avi Drissman <avi@chromium.org>
Cr-Commit-Position: refs/heads/master@{#550723}
[modify] https://crrev.com/00d3a24514b59d9628b2074ed2f2d82d4d1953de/tools/perf_expectations/perf_expectations.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/bc0d927cd391a0345979b6b12c58019f9a29bcb2

commit bc0d927cd391a0345979b6b12c58019f9a29bcb2
Author: Dirk Pranke <dpranke@chromium.org>
Date: Fri Apr 13 23:00:22 2018

Bump mac static initializer count to 4 to work around bug.

It looks like when we're computing the number of static
initializers on mac we're still assuming a 32-bit word and not the
64-bit words that we actually use, and so we mis-report the
number of initializers as twice what they actually are.

All of this perf expectation code needs to be ripped out and
replaced with some dumb static checks, but for now, this CL
just updates the expected number from 2 (the correct number) to
4, and updates the checksums in the json file to get the presubmit
to pass.

BUG= 831951 ,  832854 
TBR=avi@chromium.org, wolenetz@chromium.org, thakis@chromium.org
NOTRY=true
NOTREECHECKS=true

Change-Id: Ie6de49af3cc1443b8acc36ce80946ea50e5a76ab
Reviewed-on: https://chromium-review.googlesource.com/1013205
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#550768}
[modify] https://crrev.com/bc0d927cd391a0345979b6b12c58019f9a29bcb2/tools/perf_expectations/make_expectations.py
[modify] https://crrev.com/bc0d927cd391a0345979b6b12c58019f9a29bcb2/tools/perf_expectations/perf_expectations.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/909d320df8b095257b4a47d1ec6b0df4559d978a

commit 909d320df8b095257b4a47d1ec6b0df4559d978a
Author: Avi Drissman <avi@chromium.org>
Date: Mon Apr 16 23:49:11 2018

Remove the static initializer from message_pump_mac.mm.

Static initializers are not allowed.

BUG= 831951 

Change-Id: I1c1d41ea66a34d38119c639c2af48183b9d1ab64
Reviewed-on: https://chromium-review.googlesource.com/1012870
Commit-Queue: Avi Drissman <avi@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#551175}
[modify] https://crrev.com/909d320df8b095257b4a47d1ec6b0df4559d978a/base/message_loop/message_pump_mac.mm
[modify] https://crrev.com/909d320df8b095257b4a47d1ec6b0df4559d978a/tools/perf_expectations/README.txt
[modify] https://crrev.com/909d320df8b095257b4a47d1ec6b0df4559d978a/tools/perf_expectations/perf_expectations.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/64bf2e447afef8b206c75a4dd5b1776f20997b7e

commit 64bf2e447afef8b206c75a4dd5b1776f20997b7e
Author: Tom Anderson <thomasanderson@chromium.org>
Date: Wed Apr 18 05:08:34 2018

Remove static initializer in protobuf

This CL removes the last static initializer on Mac.

The AtomicOps_Internalx86CPUFeatures structure used to be initialized at start
time.  I've changed it to be initialized lazily.  Code that used to look like:
  if (AtomicOps_Internalx86CPUFeatures.has_amd_lock_mb_bug)
will now look like:
  if (Get_AtomicOps_Internalx86CPUFeatures().has_amd_lock_mb_bug)

This structure is used in low-level atomic operations like compare-and-swap, so
performance is a priority.  In the fast case where the structure is already
initialized, the new implementation shouldn't require any additional cycles on
modern CPUs.

BUG= 831951 
R=pkasting

CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.mac:mac_chromium_archive_rel_ng;master.tryserver.chromium.android:android_archive_rel_ng;master.tryserver.chromium.linux:linux_chromium_archive_rel_ng;master.tryserver.chromium.win:win_archive;master.tryserver.chromium.win:win_x64_archive

Change-Id: I37166109190d7a764c4234e0de343360d18cfbab
Reviewed-on: https://chromium-review.googlesource.com/1015628
Reviewed-by: Peter Kasting <pkasting@chromium.org>
Commit-Queue: Thomas Anderson <thomasanderson@chromium.org>
Cr-Commit-Position: refs/heads/master@{#551587}
[modify] https://crrev.com/64bf2e447afef8b206c75a4dd5b1776f20997b7e/third_party/protobuf/patches/0003-remove-static-initializers.patch
[modify] https://crrev.com/64bf2e447afef8b206c75a4dd5b1776f20997b7e/third_party/protobuf/src/google/protobuf/stubs/atomicops_internals_x86_gcc.cc
[modify] https://crrev.com/64bf2e447afef8b206c75a4dd5b1776f20997b7e/third_party/protobuf/src/google/protobuf/stubs/atomicops_internals_x86_gcc.h

Back over to avi@.  Could you update perf_expectations.json?  I ran mac_chromium_archive_rel_ng and saw the static initializers went down to 0.

Btw, I'm working on fixing the sizes.py script to fail properly if otool is not installed, but it's complicated by the fact that there are two, diverging copies of it (see  issue 783855  for details).

@resek - we should coordinate on changes to that script, because I was going to revamp it completely.

dpranke: Added you to the first review. My plan is to just de-dupe the scripts, fix the bug here, and then you can revamp as necessary ( issue 783855  also discusses separating out static initializers from binary sizes, which I wasn't planning on tackling).

https://chromium-review.googlesource.com/c/chromium/src/+/1018440 will fix the script and lower the expectation back down to 0.

Assigning to rsesek as he has the last piece for this bug.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/29c584bb596312da4300125833885170b9b58612

commit 29c584bb596312da4300125833885170b9b58612
Author: Robert Sesek <rsesek@chromium.org>
Date: Thu Apr 19 13:16:02 2018

Fix two bugs in the Mac implementation of sizes.py.

1) If Xcode's command line tools were not installed, otool would fail to
run. However the script was not checking the return code of the command
and the script would silently fail.

2) The static initializer count assumed a 32-bit word size. Mac has been
exclusively 64-bit for years, so assume an 8-byte word size.

This also re-lowers the expected count of static initializers now that
the ones that crept in have been removed. See the bug for details.

Bug:  831951 
Change-Id: If93b5603531d6a0933a04d67b55b7559869c1266
Reviewed-on: https://chromium-review.googlesource.com/1018440
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Robert Sesek <rsesek@chromium.org>
Cr-Commit-Position: refs/heads/master@{#551997}
[modify] https://crrev.com/29c584bb596312da4300125833885170b9b58612/infra/scripts/legacy/scripts/slave/chromium/sizes.py
[modify] https://crrev.com/29c584bb596312da4300125833885170b9b58612/tools/perf_expectations/README.txt
[modify] https://crrev.com/29c584bb596312da4300125833885170b9b58612/tools/perf_expectations/perf_expectations.json

The CL in #39 has caused the sizes step to turn red on a few Mac bots that don't have otool installed (see  issue 834782 ), so this is now working.