No space left on vm1074-m4 |
|||
Issue descriptionhttps://luci-milo.appspot.com/buildbot/tryserver.chromium.mac/ios-simulator/193391 Compile step failed. [20326/20485] ACTION //ios/chrome/test/earl_grey:ios_chrome_settings_egtests_executable(//build/toolchain/mac:ios_clang_x64) FAILED: obj/ios/chrome/test/earl_grey/ios_chrome_settings_egtests python ../../build/toolchain/mac/linker_driver.py --developer_dir /b/build/slave/ios-simulator/build/src/build/ios_files/Xcode.app xcrun lipo -create -output obj/ios/chrome/test/earl_grey/ios_chrome_settings_egtests obj/ios/chrome/test/earl_grey/x64/ios_chrome_settings_egtests ios_clang_x86/obj/ios/chrome/test/earl_grey/x86/ios_chrome_settings_egtests fatal error: /b/build/slave/ios-simulator/build/src/build/ios_files/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo: can't write to output file: obj/ios/chrome/test/earl_grey/ios_chrome_settings_egtests.lipo (No space left on device) Traceback (most recent call last): File "../../build/toolchain/mac/linker_driver.py", line 229, in <module> Main(sys.argv) File "../../build/toolchain/mac/linker_driver.py", line 79, in Main subprocess.check_call(compiler_driver_args) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 540, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['xcrun', 'lipo', '-create', '-output', 'obj/ios/chrome/test/earl_grey/ios_chrome_settings_egtests', 'obj/ios/chrome/test/earl_grey/x64/ios_chrome_settings_egtests', 'ios_clang_x86/obj/ios/chrome/test/earl_grey/x86/ios_chrome_settings_egtests']' returned non-zero exit status 1
,
Apr 13 2017
,
Apr 13 2017
This builder had "build/build.dead" and "build/ios_simulator" directories, both of which had old timestamps (March 21). I'm deleting both of them. +smut@, +sergeyberezin@, this is the second time I hit this. Is this representative if a pandemic? Do we need to add these directories to a cleanup script?
,
Apr 13 2017
I thought build.dead was supposed to be removed by something. Can we add it to cleanup_disk if it's not there already?
,
Apr 13 2017
/b/build/slave/build.dead is removed. This one was /b/build/slave/build/build.dead. I think someone manually created that one and put in the wrong place :P
,
Apr 13 2017
According to https://chrome-internal.googlesource.com/infra/infra_internal/+/master/go/src/infra_internal/tools/cleanup_disk/cmd/cleanup_disk/main.go#103 we remove /b/build/slave/*/build.dead, so the file you deleted should've been deleted automatically. However, cleanup_disk runs only once a day around midnight PST, so if build.dead appeared recently and caused the disk to fill up, there isn't much we can do automatically at the moment. But in general, if you see problematic files appear at scale, consider adding a glob to the above config and redeploy cleanup_disk (through puppet).
,
Apr 13 2017
/b/build/slave/*/build.dead
^---- one star = one directory substitution
This isn't the one I'm worried about. I'm fairly certain this was a human putting a thing in the wrong place.
I'm more concerned about build/ios_simulator. Maybe it gets installed once and never touched again? But since I've deleted it, several builds have run and it hasn't been recreated.
,
Apr 13 2017
See https://docs.google.com/document/d/1gcI42uOMKLjwjKGOw4rOzCsOu8zl_6g1lXCPIy7coFg/edit#heading=h.t5mv0g3oct11 for build/ios_simulator.
,
Apr 13 2017
So you think this is residual from that? OK so I wasn't too far off in proposing a pandemic situation :P
,
Apr 13 2017
I think so. Dates coincide, and duplicated checkouts in <buildername> directories seems like the same problem. I guess nobody went through and deleted all the duped checkouts. https://chrome-internal.googlesource.com/infra/puppet#Emergency-disk-cleanup-on-many-bots |
|||
►
Sign in to add a comment |
|||
Comment 1 by tikuta@chromium.org
, Apr 13 2017