cyan cros builder cros-beefy191-c2 is out of disk space building chrome |
||||||||||||||
Issue descriptionIt has been failing since at least August 12 17:42 https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/cyan-tot-chrome-pfq-informational?numbuilds=200 Fails during buildpackages chromeos-chrome-54.0.2830.0_alpha-r1: For example: https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/cyan-tot-chrome-pfq-informational/builds/50/steps/BuildPackages/logs/stdio chromeos-chrome-54.0.2830.0_alpha-r1: FAILED: obj/chrome/browser/chromeos/chromeos/async_file_util.o chromeos-chrome-54.0.2830.0_alpha-r1: x86_64-cros-linux-gnu-g++ -B/usr/x86_64-pc-linux-gnu/x86_64-cros-linux-gnu/binutils-bin/2.25.51-gold -MMD -MF obj/chrome/browser/chromeos/chromeos/async_file_util.o.d -DUSE_CRAS -DV8_DEPRECATION_WARNINGS -DENABLE_MDNS=1 -DENABLE_NOTIFICATIONS -DENABLE_PEPPER_CDMS -DENABLE_PLUGINS=1 -DENABLE_PDF=1 -DENABLE_PRINTING=1 -DENABLE_BASIC_PRINTING=1 -DENABLE_PRINT_PREVIEW=1 -DENABLE_SPELLCHECK=1 -DUSE_UDEV -DUI_COMPOSITOR_IMAGE_TRANSPORT -DUSE_ASH=1 -DUSE_AURA=1 -DUSE_DEFAULT_RENDER_THEME=1 -DUSE_NSS_CERTS=1 -DUSE_OZONE=1 -DENABLE_WEBRTC=1 -DENABLE_EXTENSIONS=1 -DENABLE_TASK_MANAGER=1 -DENABLE_THEMES=1 -DENABLE_CAPTIVE_PORTAL_DETECTION=1 -DENABLE_SESSION_SERVICE=1 -DENABLE_RLZ -DENABLE_APP_LIST=1 -DENABLE_SUPERVISED_USERS=1 -DENABLE_SERVICE_DISCOVERY=1 -DIMAGE_LOADER_EXTENSION=1 -DENABLE_WAYLAND_SERVER=1 -DUSE_PROPRIETARY_CODECS -DFULL_SAFE_BROWSING -DSAFE_BROWSING_CSD -DSAFE_BROWSING_DB_LOCAL -DOFFICIAL_BUILD -DGOOGLE_CHROME_BUILD -DENABLE_MEDIA_ROUTER=1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -DNO_UNWIND_TABLES -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D_FORTIFY_SOURCE=2 -DOS_CHROMEOS -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DUSE_EGL -DTOOLKIT_VIEWS=1 -DWEBRTC_BUILD_LIBEVENT -DSK_IGNORE_DW_GRAY_FIX -DSK_IGNORE_LINEONLY_AA_CONVEX_PATH_OPTS -DSK_SUPPORT_GPU=1 -DGOOGLE_PROTOBUF_NO_RTTI -DGOOGLE_PROTOBUF_NO_STATIC_INITIALIZER -DHAVE_PTHREAD -DU_USING_ICU_NAMESPACE=0 -DU_ENABLE_DYLOAD=0 -DU_NOEXCEPT= -DU_STATIC_IMPLEMENTATION -DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_FILE -DV8_USE_EXTERNAL_STARTUP_DATA -DENABLE_WEBSOCKETS -DMESA_EGL_NO_X11_HEADERS -DLEVELDB_PLATFORM_CHROMIUM=1 -DFEATURE_ENABLE_SSL -DFEATURE_ENABLE_VOICEMAIL -DEXPAT_RELATIVE_PATH -DGTEST_RELATIVE_PATH -DNO_MAIN_THREAD_WRAPPING -DNO_SOUND_SYSTEM -DWEBRTC_CHROMIUM_BUILD -DWEBRTC_POSIX -DWEBRTC_LINUX -DCHROMEOS -DSSL_USE_OPENSSL -DHAVE_OPENSSL_SSL_H -DFEATURE_ENABLE_SSL -DLOGGING=1 -DNO_MAIN_THREAD_WRAPPING -I../../../../../../../home/chrome-bot/chrome_root/src -Igen -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/khronos -I../../../../../../../home/chrome-bot/chrome_root/src/gpu -Igen/ash/common/strings -Igen/ash/resources -Igen/chrome -Igen/chrome -Igen/chrome -Igen/chrome -Igen/chrome -Igen/chrome -Igen/chrome -Igen/chrome -Igen/chrome -Igen/chrome -Igen/chrome -Igen/chrome -Igen/chrome -Igen/chrome -Igen/chrome -Igen/chrome -I../../../../../../../home/chrome-bot/chrome_root/src/skia/config -I../../../../../../../home/chrome-bot/chrome_root/src/skia/ext -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/skia/include/c -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/skia/include/config -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/skia/include/core -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/skia/include/effects -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/skia/include/images -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/skia/include/lazy -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/skia/include/pathops -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/skia/include/pdf -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/skia/include/pipe -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/skia/include/ports -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/skia/include/utils -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/skia/include/gpu -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/skia/src/gpu -Igen/ui/chromeos/resources -Igen/ui/chromeos/strings -Igen/ui/resources -Igen/ui/resources -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/protobuf/src -Igen/protoc_out -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/protobuf/src -Igen -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/ced/src -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/icu/source/common -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/icu/source/i18n -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/dom_distiller_js/dist/proto_gen -Igen/components/strings -Igen/components/strings -Igen/components/strings -Igen/components/strings -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/WebKit -Igen/third_party/WebKit -I../../../../../../../home/chrome-bot/chrome_root/src/v8/include -Igen -I../../../../../../../home/chrome-bot/chrome_root/src/v8/include -I/build/cyan/usr/include/nss -I/build/cyan/usr/include/nspr -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/boringssl/src/include -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/mesa/src/include -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/libwebm/source -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/opus/src/include -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/re2/src -Igen/extensions -Igen/extensions -Igen/extensions -Igen -Igen/extensions/strings -I/build/cyan/usr/include/dbus-1.0 -I/build/cyan/usr/lib64/dbus-1.0/include -Igen -Igen/components -Igen/components -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/cacheinvalidation/overrides -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/cacheinvalidation/src -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/leveldatabase -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/leveldatabase/src -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/leveldatabase/src/include -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/webrtc_overrides -I../../../../../../../home/chrome-bot/chrome_root/src/testing/gtest/include -I../../../../../../../home/chrome-bot/chrome_root/src/third_party -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/webrtc_overrides -I../../../../../../../home/chrome-bot/chrome_root/src/third_party -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/jsoncpp/overrides/include -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/jsoncpp/source/include -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/libusb/src/libusb -I../../../../../../../home/chrome-bot/chrome_root/src/third_party/zlib -Igen/ui/views/resources -fno-strict-aliasing -fno-unwind-tables -fno-asynchronous-unwind-tables -fPIC -pipe -pthread -m64 -march=x86-64 -Wall -Werror -Wno-unused-local-typedefs -Wno-maybe-uninitialized -Wno-missing-field-initializers -Wno-unused-parameter -O2 -fno-ident -fdata-sections -ffunction-sections -g2 --sysroot=../../../../../../../build/cyan -fvisibility=hidden -fno-threadsafe-statics -fvisibility-inlines-hidden -std=gnu++11 -Wno-narrowing -Wno-literal-suffix -fno-rtti -fno-exceptions -pipe -pipe -pipe -march=corei7 -D__google_stl_debug_vector=1 -c ../../../../../../../home/chrome-bot/chrome_root/src/chrome/browser/chromeos/drive/fileapi/async_file_util.cc -o obj/chrome/browser/chromeos/chromeos/async_file_util.o chromeos-chrome-54.0.2830.0_alpha-r1: {standard input}: Assembler messages: chromeos-chrome-54.0.2830.0_alpha-r1: {standard input}: Fatal error: can't write obj/chrome/browser/chromeos/chromeos/async_file_util.o: No space left on device chromeos-chrome-54.0.2830.0_alpha-r1: {standard input}: Fatal error: can't close obj/chrome/browser/chromeos/chromeos/async_file_util.o: No space left on device To phobbs for triage
,
Aug 16 2016
Any news here? Could the build get shifted to another builder somehow and see if it works there?
,
Aug 16 2016
After ssh'ing in, I think I understand what's going on, and why we've seen several cases of this lately. We are using relatively small drives for the GCE builders (500G). That's fine in the normal case. But if a build machine gets moved around between multiple builds, it ends up with multiple different buildroot directories (/b/cbuild.<config>). The smaller drives can't handle the multiple checkouts.
,
Aug 16 2016
Options: 1) Re-use the same buildroot for different configs (can affect performance) 2) Monitor and cleanup/reimage as needed. 3) Use bigger drives (and re-image builders as the problem crops up).
,
Aug 16 2016
du -sh /b/* 304M build 114M build_internal 99G cbuild.cyan 290G cbuild.cyan-cheets 51M depot_tools 37G git-cache 392K internal.DEPS I've started a deletion of cbuild.cyan-cheets, which should fix this. It might not be in time for the current run.
,
Aug 16 2016
Please follow http://go/bugatrooper to report infra breakages.
,
Aug 16 2016
Apologies for not following go/bugatrooper. I've added a link to it to our internal gardening docs at go/chrome-on-chromeos-gardeners Reducing to P1 per that doc. (I think this can probably be closed, the cyan builder on cros-beefy191-c2 is green now.)
,
Aug 17 2016
,
Aug 26 2016
,
Oct 7 2016
,
Oct 10 2016
,
Nov 19 2016
,
Jan 21 2017
,
Mar 4 2017
,
Apr 17 2017
,
May 30 2017
,
Aug 1 2017
,
Oct 14 2017
|
||||||||||||||
►
Sign in to add a comment |
||||||||||||||
Comment 1 by jamescook@chromium.org
, Aug 15 2016