Nothing looks interesting in .ci/36135.  Any ideas here?

Every build after 36135 is compiling ~59769 steps.

Looks like we're no longer doing incremental builds?

+sdefresne

sdefresne@ could you PTAL if possible?

This ios-simulator CQ bot now has 100 pending builds, presumably due to the compile time spike:
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/ios-simulator

The build linked in #0 also exhibited an increase in "gclient runhooks" runtime. It went from consistently ~20s to 8min. It's like we're running on empty caches each time... which would lead to no incremental builds like #3 pointed out.

There's a chance this regression is due to infra-related changes. I'll grab this bug and start digging around. Will pass it back if I don't find anything.

Yep, it looks like the CI bot lost its persistent builder cache at 36135.

See the list of caches at 36134: https://chromium-swarm.appspot.com/task?id=3f10138568f60f10
And the caches at 36135: https://chromium-swarm.appspot.com/task?id=3f104380a75f4210

It's possible that this is a regression caused by https://chromium-review.googlesource.com/c/infra/luci/luci-go/+/1158605

Though we're still trying to nail down the exact nature of what happened to the cache.

What is a "persistent builder cache"?  What does it store?

It looks like this is causing us to redownload clang on every run -- that would probably explain why we're doing a full recompile.

It's where the builder places its chromium/src.git checkout along with its out dir. It reuses the same one in between builds, so we can do incremental builds.

But for some reason on this bot, that cache is getting deleted in between builds. So it has to re-checkout src and build from scratch every time. Still trying to figure out *why* it's getting removed.

I think I see what's going on now. Something happened in build 36134 (https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/ios-simulator/36134) that grew the builder's out dir quite a bit (~50 GB). You can see the initial jump in disk usage on the bot:
http://shortn/_6s7kvhrXYg

At that point, the bot no longer had the minimum required free disk usage of 90GB.
https://chrome-internal.googlesource.com/infradata/config/+/477c78a99beff1a2cb211f905b69db7eeb4c6c44/configs/chromium-swarm/scripts/bot_config.py#411

It actually only had ~55 GB. So swarming started aggressively clearing caches to reach the 90 GB free space requirement. And it ends up removing the checkout & out dir caches. That brings us back to the free space req, but also makes every subsequent build a clobber.

The 90 GB requirement is pretty dumb when we're using persistent caches, so I'll work on making that smarter. (55 GB is plenty of free space to have.) But someone might also want to figure out what in https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/ios-simulator/36134 grew the out dir so much. (I glanced through the blamelist but didn't see anything pertinent other than the clang roll.)

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/679932b8f8765e3a9a4591111c46cb9685838c89

commit 679932b8f8765e3a9a4591111c46cb9685838c89
Author: Ben Pastene <bpastene@chromium.org>
Date: Thu Aug 09 02:30:15 2018

Issue 872574 has been merged into this issue.

Commented in
https://bugs.chromium.org/p/chromium/issues/detail?id=866225#c17

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8457b638ed3550460518321e4a4a21b08c2e30f9

commit 8457b638ed3550460518321e4a4a21b08c2e30f9
Author: Reid Kleckner <rnk@chromium.org>
Date: Fri Aug 10 02:44:48 2018

Revert "Roll clang 337439:338452."

This reverts commit cc14e55b5cd9359e09aa9b8aa8795b100031abfb.

Reason for revert:
The new version of clang appears to have regressed -O0 code size significantly ( https://crbug.com/870907 )

Original change's description:
> Roll clang 337439:338452.
>
> Bug:  866225 
> Change-Id: I11d97d9a97432a1abb97e69a1395948b9d3e19fe
> Reviewed-on: https://chromium-review.googlesource.com/1157290
> Commit-Queue: Reid Kleckner <rnk@chromium.org>
> Reviewed-by: Nico Weber <thakis@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#579922}

TBR=thakis@chromium.org,hans@chromium.org,rnk@chromium.org
NOTRY=true

Bug:  866225 ,  870907 
Change-Id: I39ac706d7fc4191d4519f1692f927788a072a081
Reviewed-on: https://chromium-review.googlesource.com/1170006
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Reid Kleckner <rnk@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#582026}
[modify] https://crrev.com/8457b638ed3550460518321e4a4a21b08c2e30f9/tools/clang/scripts/update.py

With the bot change and clang revert, the bot's compiling a lot faster now. The outdir shrank and should be persisted across builds.

However, overall build cycle time (~50min) isn't quite back down to what it was before (~30min). That seems to be due to the fact that it's running ~50 more tests after https://chromium-review.googlesource.com/c/chromium/src/+/1165318. I'm assuming that's WAI, so I'm closing this out.

Yes that is WAI, thanks bpastene!

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d74b4a7338023093d2cc0ada1c8d9964e1f1faf8

commit d74b4a7338023093d2cc0ada1c8d9964e1f1faf8
Author: Hans Wennborg <hans@chromium.org>
Date: Tue Aug 14 17:08:19 2018

Re-land "Roll clang 337439:338452."

And configure libc++ to force inline ABI-hidden functions, as was its behaviour
before r338122. Without this, the symbol tables of Mach-O binaries grow
significantly due to including multiple definitions of these functions.

Bug:  866225 ,  870907 ,  872926 
Change-Id: I4382e9de57fc79b3baff3d329e338887cb4b133b
Reviewed-on: https://chromium-review.googlesource.com/1174546
Commit-Queue: Hans Wennborg <hans@chromium.org>
Commit-Queue: Reid Kleckner <rnk@chromium.org>
Reviewed-by: Reid Kleckner <rnk@chromium.org>
Cr-Commit-Position: refs/heads/master@{#582951}
[modify] https://crrev.com/d74b4a7338023093d2cc0ada1c8d9964e1f1faf8/build/config/compiler/BUILD.gn
[modify] https://crrev.com/d74b4a7338023093d2cc0ada1c8d9964e1f1faf8/tools/clang/scripts/update.py