Example failure. https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/ios-device-xcode-clang/72834

I believe the culprit is either https://chromium-review.googlesource.com/c/1278332 or some infra problem. I will try to revert first.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f8cad916e6d37d507a736db578b70937b35b19d0

commit f8cad916e6d37d507a736db578b70937b35b19d0
Author: Christos Froussios <cfroussios@chromium.org>
Date: Tue Oct 16 08:59:02 2018

Revert "[ios] Move xcode-clang builders to Xcode 10."

This reverts commit eb25f32f656f7c3c4e58b0b384647428cb13e919.

Reason for revert: Suspect for build failures on https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/ios-device-xcode-clang


Original change's description:
> [ios] Move xcode-clang builders to Xcode 10.
> 
> Cq-Include-Trybots: luci.chromium.try:ios-simulator-cronet;luci.chromium.try:ios-simulator-full-configs
> Change-Id: Ie5bdc140b82f49151ca891b39f65112bf6210eac
> Reviewed-on: https://chromium-review.googlesource.com/c/1278332
> Reviewed-by: Justin Cohen <justincohen@chromium.org>
> Reviewed-by: John Budorick <jbudorick@chromium.org>
> Commit-Queue: Justin Cohen <justincohen@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#599804}

TBR=justincohen@chromium.org,jbudorick@chromium.org

Bug:  895761 
Change-Id: I9b1b83455bd9b27fdf9b6e7120467e3e87305007
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Cq-Include-Trybots: luci.chromium.try:ios-simulator-cronet;luci.chromium.try:ios-simulator-full-configs
Reviewed-on: https://chromium-review.googlesource.com/c/1282949
Reviewed-by: Christos Froussios <cfroussios@chromium.org>
Commit-Queue: Christos Froussios <cfroussios@chromium.org>
Cr-Commit-Position: refs/heads/master@{#599923}
[modify] https://crrev.com/f8cad916e6d37d507a736db578b70937b35b19d0/ios/build/bots/chromium.mac/ios-device-xcode-clang.json
[modify] https://crrev.com/f8cad916e6d37d507a736db578b70937b35b19d0/ios/build/bots/chromium.mac/ios-simulator-xcode-clang.json

I'm relanding the reverted CL in https://chromium-review.googlesource.com/c/chromium/src/+/1283016

This looks like an infra problem. The bot is out of space

FAILED: obj/ios/chrome/test/earl_grey/ios_chrome_ui_egtests 
  python ../../build/toolchain/mac/linker_driver.py xcrun lipo -create -output obj/ios/chrome/test/earl_grey/ios_chrome_ui_egtests obj/ios/chrome/test/earl_grey/arm/ios_chrome_ui_egtests ios_clang_arm64/obj/ios/chrome/test/earl_grey/arm64/ios_chrome_ui_egtests
  fatal error: /b/s/w/ir/cache/xcode_ios_10a254a.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo: can't write to output file: obj/ios/chrome/test/earl_grey/ios_chrome_ui_egtests.lipo (No space left on device)
  Traceback (most recent call last):
    File "../../build/toolchain/mac/linker_driver.py", line 229, in <module>
      Main(sys.argv)
    File "../../build/toolchain/mac/linker_driver.py", line 79, in Main
      subprocess.check_call(compiler_driver_args)
    File "/b/s/w/ir/cipd_bin_packages/lib/python2.7/subprocess.py", line 186, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['xcrun', 'lipo', '-create', '-output', 'obj/ios/chrome/test/earl_grey/ios_chrome_ui_egtests', 'obj/ios/chrome/test/earl_grey/arm/ios_chrome_ui_egtests', 'ios_clang_arm64/obj/ios/chrome/test/earl_grey/arm64/ios_chrome_ui_egtests']' returned non-zero exit status 1

Issue 895943 has been merged into this issue.

The bot in question:
https://chromium-swarm.appspot.com/bot?id=vm122-m9&sort_stats=total%3Adesc

Taking as a trooper; will look soon.

Looks like the disk was gradually creeping up, and now it finally hit the wall: http://shortn/_TLCThaaayP . I'll check what's filling it up and hopefully clean it.

There is some junk left over from simulators - about 4G; will clean that up, it should get the build through in the immediate term:

chrome-bot@vm122-m9:(Mac 10.13.3):~/Library/Developer/CoreSimulator/Devices$ ls | wc -l
     145
chrome-bot@vm122-m9:(Mac 10.13.3):~/Library/Developer/CoreSimulator/Devices$ du -sch .
4.0G	.
4.0G	total
chrome-bot@vm122-m9:(Mac 10.13.3):~/Library/Developer/CoreSimulator/Devices$ ls -ltr | head -n5
total 40
drwxr-xr-x@ 4 chrome-bot  staff    136 May 15 15:53 1B739C45-EDC9-4F5E-83A1-27AD01EE9859
drwxr-xr-x@ 4 chrome-bot  staff    136 May 15 15:53 CB0A0271-F398-4430-AA6A-A339BE1CD0B8
drwxr-xr-x@ 4 chrome-bot  staff    136 May 15 15:53 C7452BFC-D606-4234-89C9-E1EB146D3C15
drwxr-xr-x@ 4 chrome-bot  staff    136 May 15 15:53 90F0F6DE-644E-4C84-B457-DABF617D4096
chrome-bot@vm122-m9:(Mac 10.13.3):~/Library/Developer/CoreSimulator/Devices$ ls -ltr | tail -n5
drwxr-xr-x@ 4 chrome-bot  staff    136 Aug  7 22:17 3D6D8E2A-C833-4A19-B638-48CF13FED12E
drwxr-xr-x@ 4 chrome-bot  staff    136 Oct 15 21:35 6FFCFFF4-5959-48BD-BCA7-10327FB7E9D5
drwxr-xr-x@ 4 chrome-bot  staff    136 Oct 15 21:36 59723181-7222-4F15-A1BD-802B45BD64C6
drwxr-xr-x@ 4 chrome-bot  staff    136 Oct 15 21:36 9AA4687A-F2C7-44BB-A6E5-55796E6BDC99
-rw-r--r--@ 1 chrome-bot  staff  18647 Oct 16 16:46 device_set.plist

Clearing disk space fixed the bot for now, but it'll probably blow up again in the future.

A potential problem is running the compile with full symbols. Most other builders now use symbol_level=1 - https://crrev.com/c/1285770 to fix that.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8bef2e268bccaf7ecbe2677c36aa86558c4c3869

commit 8bef2e268bccaf7ecbe2677c36aa86558c4c3869
Author: Sergey Berezin <sergeyberezin@google.com>
Date: Wed Oct 17 16:59:59 2018

[ios] Set symbol_level=0 for ios-device* builds

Both ios-device and ios-device-xcode-clang seem to have disk space issues. This
is an attempt to reduce the disk space required for compilation on bots.

Bug:  895761 
Cq-Include-Trybots: luci.chromium.try:ios-simulator-cronet;luci.chromium.try:ios-simulator-full-configs
Change-Id: Iea34c089b780ecb9231ab2fc9031ae9f89ab998f
Reviewed-on: https://chromium-review.googlesource.com/c/1285770
Reviewed-by: Justin Cohen <justincohen@chromium.org>
Commit-Queue: Sergey Berezin <sergeyberezin@chromium.org>
Cr-Commit-Position: refs/heads/master@{#600433}
[modify] https://crrev.com/8bef2e268bccaf7ecbe2677c36aa86558c4c3869/ios/build/bots/chromium.mac/ios-device-xcode-clang.json
[modify] https://crrev.com/8bef2e268bccaf7ecbe2677c36aa86558c4c3869/ios/build/bots/chromium.mac/ios-device.json

Issue 895758 has been merged into this issue.

Closing - the disk is now stable at 56%. Something happened at 2018-10-16 23:54:44 PDT (Tue) - perhaps an auto-cleaning event?
The change in symbol_level also helped to reduce the usage by another 10%, so we should be good for a while.

Great, thanks for looking into this!