Could there be some new antivirus thing going on with corp mac machines?

Building LLVM also feels slow on my mac.

I was going to post this on crbug.com/224814 yesterday but crbug was giving 500s. Maybe more relevant to this one:

"Was just wondering about this recently, since I've observed that a local Mac non-goma build takes a very long time now (e.g. 2h) when it used to take 30m a couple years ago on a beefy Mac Pro.

Now, obviously Chrome codebase evolves and gets larger, but it seems hard to believe that it regressed so much organically and not due to other things (e.g. regressions in compiler or some other part of build system).

In particular, it seems quite strange to me that near end of compile, with around 2k files left, files are taking on average (eyeballing) around 1s to compile (which seems really slow given that there's like a dozen clang processes running in parallel ... so one new file being finished per second seems very slow).

Also wondering if we could check if speed regressed due to Chrome changes or compiler changes. For example, we could try using current clang to compile Chrome from a couple years ago and compare speed."

I am indeed on 10.13, 

Woops, forgot to finish my thought:

I am indeed on 10.13, but I do recall noticing the slowdown even before 10.13 update and spectre - but I didn't look too deeply before. So it could have gotten worse recently - e.g. my observations of files taking 1s each to make progress in ninja while lots clang processes are running in parallel is something I specifically observed recently.

Nico suggested I sync and build again in case it was something with gtest roll that was reverted. I did that, cleaning the out directory and doing a debug component build (which is what I usually do).

Took 2.8 hours.

[34344/34344] STAMP obj/chrome/chrome.stamp

real	168m24.877s
user	2851m29.605s
sys	237m43.088s

The first 20k files went quite fast (e.g. 20 mins maybe). But from then on, it was much slower.

The bots that build clang itself also got slower on mac, but much less so on linux and win:
https://ci.chromium.org/buildbot/tryserver.chromium.mac/mac_upload_clang/?limit=200
https://ci.chromium.org/buildbot/tryserver.chromium.linux/linux_upload_clang/?limit=200
https://ci.chromium.org/buildbot/tryserver.chromium.win/win_upload_clang/?limit=200

Based on that table, things got a lot slower late Oct 2017. The slaves this bot runs are now on 10.12 which rules out apfs. I think it also rules out corp AV.

dba, do you know if we rolled out any macOS spectre mitigations or similar to the bots in late Oct 2017?

Hans also had the idea that an SDK update with much larger headers could cause this -- did we update Xcodes in late Oct 2017?

I suppose https://chromium.googlesource.com/chromium/src/+/251622585cfd0f0a277aa8041fa583ca50b2a854 landed around that window, but lld feels too small to increase build time that much.

#7: No, all of the mac bots in #6 bots are still running 10.12.2 and still using a system Xcode of 8.0 from what I can see.

I still think the best way to check is to compare speed of TOT build vs. an older version of the checkout. Then, we could also switch the compiler version to see if that makes a difference if we can repro the build time regression.

... I tried to do this via instructions to check out a given branch, via:

gclient config --deps-file=branches/2785/DEPS https://chrome-internal.googlesource.com/a/chrome/tools/buildspec.git
gclient sync --with_branch_heads --with_tags --jobs 16

However, the scripts failed at the end with an error:

ValueError: invalid assignment: overrides var 'recursedeps' (file '/Users/asvitkine/old_chromium/src/DEPS', line 955)

:(

The sync also took forever. (Wonder if we have tarball/zip archives that don't require syncing which would be much faster to download and try to build...)

I finally got around to setting up goma and did a full build:

[36147/36147] STAMP obj/chrome/chrome.stamp

real	92m27.286s
user	25m36.331s
sys	92m26.196s

Well, certainly much better than without goma, but for a goma build - it still seems quite slow? 1.5h is still much slower than I remember builds being a year or two ago (e.g. 30m-1h on the mac pro)

Did you pass -j200 to ninja? Does localhost:8080 say that remote compiles work fine (the "finished" tab)

I passed -j100 per instructions on go/goma (about too much contention with higher values).

localhost:8080 didn't load, but turns out it's port 8088. It looks fine (lots of success on finished tab).

Marking as available, in case anyone has time to do more digging.