build errors on android try bots but the main waterfall it mirrors is fine. |
||||||||||||||||||||||
Issue descriptionI'm seeing failed compile without patches errors on https://build.chromium.org/p/tryserver.chromium.android/builders/android_compile_dbg?numbuilds=200 This is strange since the bot it mirrors is https://build.chromium.org/p/chromium.linux/builders/Android%20Builder%20%28dbg%29?numbuilds=200 is green https://codereview.chromium.org/1750823002 is one fix when I tried to build content_browsertests target locally, to repro a different error (which I can't yet). some examples https://build.chromium.org/p/tryserver.chromium.android/builders/android_compile_dbg/builds/29192/steps/compile%20%28without%20patch%29/logs/stdio FAILED: if [ ! -e lib/libchrome_public.so -o ! -e lib/libchrome_public.so.TOC ]; then /b/build/goma/gomacc /b/build/slave/android/build/src/third_party/android_tools/ndk//toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-g++ -shared -Wl,-z,now -Wl,-z,relro -Wl,--fatal-warnings -Wl,-z,defs -Wl,-z,noexecstack -fPIC -B/b/build/slave/android/build/src/third_party/binutils/Linux_x64/Release/bin -Wl,--gc-sections -Wl,-z,relro -Wl,-z,now -fuse-ld=gold -Wl,--build-id=sha1 -Wl,--no-undefined --sysroot=../../third_party/android_tools/ndk//platforms/android-16/arch-arm -nostdlib -L../../third_party/android_tools/ndk//sources/cxx-stl/llvm-libc++/libs/armeabi-v7a -Wl,--exclude-libs=libgcc.a -Wl,--exclude-libs=libc++_static.a -Wl,--exclude-libs=libcommon_audio.a -Wl,--exclude-libs=libcommon_audio_neon.a -Wl,--exclude-libs=libcommon_audio_sse2.a -Wl,--exclude-libs=libiSACFix.a -Wl,--exclude-libs=libisac_neon.a -Wl,--exclude-libs=libopus.a -Wl,--exclude-libs=libvpx.a -Wl,--icf=all -Wl,-shared,-Bsymbolic ../../third_party/android_tools/ndk//platforms/android-16/arch-arm/usr/lib/crtbegin_so.o -Wl,--version-script=/b/build/slave/android/build/src/build/android/android_no_jni_exports.lst -Wl,--warn-shared-textrel -Wl,-O1 -Wl,--as-needed -o lib/libchrome_public.so -Wl,-soname=libchrome_public.so @lib/libchrome_public.so.rsp && { readelf -d lib/libchrome_public.so | grep SONAME ; nm -gD -f p lib/libchrome_public.so | cut -f1-2 -d' '; } > lib/libchrome_public.so.TOC; else /b/build/goma/gomacc /b/build/slave/android/build/src/third_party/android_tools/ndk//toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-g++ -shared -Wl,-z,now -Wl,-z,relro -Wl,--fatal-warnings -Wl,-z,defs -Wl,-z,noexecstack -fPIC -B/b/build/slave/android/build/src/third_party/binutils/Linux_x64/Release/bin -Wl,--gc-sections -Wl,-z,relro -Wl,-z,now -fuse-ld=gold -Wl,--build-id=sha1 -Wl,--no-undefined --sysroot=../../third_party/android_tools/ndk//platforms/android-16/arch-arm -nostdlib -L../../third_party/android_tools/ndk//sources/cxx-stl/llvm-libc++/libs/armeabi-v7a -Wl,--exclude-libs=libgcc.a -Wl,--exclude-libs=libc++_static.a -Wl,--exclude-libs=libcommon_audio.a -Wl,--exclude-libs=libcommon_audio_neon.a -Wl,--exclude-libs=libcommon_audio_sse2.a -Wl,--exclude-libs=libiSACFix.a -Wl,--exclude-libs=libisac_neon.a -Wl,--exclude-libs=libopus.a -Wl,--exclude-libs=libvpx.a -Wl,--icf=all -Wl,-shared,-Bsymbolic ../../third_party/android_tools/ndk//platforms/android-16/arch-arm/usr/lib/crtbegin_so.o -Wl,--version-script=/b/build/slave/android/build/src/build/android/android_no_jni_exports.lst -Wl,--warn-shared-textrel -Wl,-O1 -Wl,--as-needed -o lib/libchrome_public.so -Wl,-soname=libchrome_public.so @lib/libchrome_public.so.rsp && { readelf -d lib/libchrome_public.so | grep SONAME ; nm -gD -f p lib/libchrome_public.so | cut -f1-2 -d' '; } > lib/libchrome_public.so.tmp && if ! cmp -s lib/libchrome_public.so.tmp lib/libchrome_public.so.TOC; then mv lib/libchrome_public.so.tmp lib/libchrome_public.so.TOC ; fi; fi obj/third_party/WebKit/Source/core/libwebcore_svg.a(obj/third_party/WebKit/Source/core/layout/svg/webcore_svg.LayoutSVGTransformableContainer.o):LayoutSVGTransformableContainer.cpp:vtable for blink::LayoutSVGTransformableContainer: error: undefined reference to 'blink::LayoutObject::mapAbsoluteToLocalPoint(unsigned int, blink::TransformState&) const' obj/third_party/WebKit/Source/core/libwebcore_svg.a(obj/third_party/WebKit/Source/core/layout/svg/webcore_svg.LayoutSVGViewportContainer.o):LayoutSVGViewportContainer.cpp:vtable for blink::LayoutSVGViewportContainer: error: undefined reference to 'blink::LayoutObject::mapAbsoluteToLocalPoint(unsigned int, blink::TransformState&) const' collect2: error: ld returned 1 exit status ninja: build stopped: subcommand failed. (more later)
,
Mar 1 2016
https://build.chromium.org/p/tryserver.chromium.android/builders/android_compile_dbg/builds/29169/steps/compile%20%28without%20patch%29/logs/stdio (mine) FAILED: if [ ! -e lib/libchrome_public.so -o ! -e lib/libchrome_public.so.TOC ]; then /b/build/goma/gomacc /b/build/slave/android/build/src/third_party/android_tools/ndk//toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-g++ -shared -Wl,-z,now -Wl,-z,relro -Wl,--fatal-warnings -Wl,-z,defs -Wl,-z,noexecstack -fPIC -B/b/build/slave/android/build/src/third_party/binutils/Linux_x64/Release/bin -Wl,--gc-sections -Wl,-z,relro -Wl,-z,now -fuse-ld=gold -Wl,--build-id=sha1 -Wl,--no-undefined --sysroot=../../third_party/android_tools/ndk//platforms/android-16/arch-arm -nostdlib -L../../third_party/android_tools/ndk//sources/cxx-stl/llvm-libc++/libs/armeabi-v7a -Wl,--exclude-libs=libgcc.a -Wl,--exclude-libs=libc++_static.a -Wl,--exclude-libs=libcommon_audio.a -Wl,--exclude-libs=libcommon_audio_neon.a -Wl,--exclude-libs=libcommon_audio_sse2.a -Wl,--exclude-libs=libiSACFix.a -Wl,--exclude-libs=libisac_neon.a -Wl,--exclude-libs=libopus.a -Wl,--exclude-libs=libvpx.a -Wl,--icf=all -Wl,-shared,-Bsymbolic ../../third_party/android_tools/ndk//platforms/android-16/arch-arm/usr/lib/crtbegin_so.o -Wl,--version-script=/b/build/slave/android/build/src/build/android/android_no_jni_exports.lst -Wl,--warn-shared-textrel -Wl,-O1 -Wl,--as-needed -o lib/libchrome_public.so -Wl,-soname=libchrome_public.so @lib/libchrome_public.so.rsp && { readelf -d lib/libchrome_public.so | grep SONAME ; nm -gD -f p lib/libchrome_public.so | cut -f1-2 -d' '; } > lib/libchrome_public.so.TOC; else /b/build/goma/gomacc /b/build/slave/android/build/src/third_party/android_tools/ndk//toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-g++ -shared -Wl,-z,now -Wl,-z,relro -Wl,--fatal-warnings -Wl,-z,defs -Wl,-z,noexecstack -fPIC -B/b/build/slave/android/build/src/third_party/binutils/Linux_x64/Release/bin -Wl,--gc-sections -Wl,-z,relro -Wl,-z,now -fuse-ld=gold -Wl,--build-id=sha1 -Wl,--no-undefined --sysroot=../../third_party/android_tools/ndk//platforms/android-16/arch-arm -nostdlib -L../../third_party/android_tools/ndk//sources/cxx-stl/llvm-libc++/libs/armeabi-v7a -Wl,--exclude-libs=libgcc.a -Wl,--exclude-libs=libc++_static.a -Wl,--exclude-libs=libcommon_audio.a -Wl,--exclude-libs=libcommon_audio_neon.a -Wl,--exclude-libs=libcommon_audio_sse2.a -Wl,--exclude-libs=libiSACFix.a -Wl,--exclude-libs=libisac_neon.a -Wl,--exclude-libs=libopus.a -Wl,--exclude-libs=libvpx.a -Wl,--icf=all -Wl,-shared,-Bsymbolic ../../third_party/android_tools/ndk//platforms/android-16/arch-arm/usr/lib/crtbegin_so.o -Wl,--version-script=/b/build/slave/android/build/src/build/android/android_no_jni_exports.lst -Wl,--warn-shared-textrel -Wl,-O1 -Wl,--as-needed -o lib/libchrome_public.so -Wl,-soname=libchrome_public.so @lib/libchrome_public.so.rsp && { readelf -d lib/libchrome_public.so | grep SONAME ; nm -gD -f p lib/libchrome_public.so | cut -f1-2 -d' '; } > lib/libchrome_public.so.tmp && if ! cmp -s lib/libchrome_public.so.tmp lib/libchrome_public.so.TOC; then mv lib/libchrome_public.so.tmp lib/libchrome_public.so.TOC ; fi; fi obj/third_party/WebKit/Source/platform/libblink_platform.a(obj/third_party/WebKit/Source/platform/heap/blink_platform.ThreadState.o):ThreadState.cpp:function blink::ThreadState::ThreadState(): error: undefined reference to 'blink::CallbackStack::CallbackStack()' collect2: error: ld returned 1 exit status
,
Mar 1 2016
,
Mar 1 2016
Weird, looks like the bot is compiling other builds though.
,
Mar 1 2016
hm I'm seeing this on https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/31084 as well. This might be widespread?
,
Mar 1 2016
FAILED: if [ ! -e lib/libnet_java_test_support.so -o ! -e lib/libnet_java_test_support.so.TOC ]; then /b/build/goma/gomacc /b/build/slave/android/build/src/third_party/android_tools/ndk//toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-g++ -shared -Wl,-z,now -Wl,-z,relro -Wl,--fatal-warnings -Wl,-z,defs -Wl,-z,noexecstack -fPIC -B/b/build/slave/android/build/src/third_party/binutils/Linux_x64/Release/bin -Wl,-z,relro -Wl,-z,now -fuse-ld=gold -Wl,--build-id=sha1 -Wl,--no-undefined --sysroot=../../third_party/android_tools/ndk//platforms/android-16/arch-arm -nostdlib -L../../third_party/android_tools/ndk//sources/cxx-stl/llvm-libc++/libs/armeabi-v7a -Wl,--exclude-libs=libgcc.a -Wl,--exclude-libs=libc++_static.a -Wl,--exclude-libs=libcommon_audio.a -Wl,--exclude-libs=libcommon_audio_neon.a -Wl,--exclude-libs=libcommon_audio_sse2.a -Wl,--exclude-libs=libiSACFix.a -Wl,--exclude-libs=libisac_neon.a -Wl,--exclude-libs=libopus.a -Wl,--exclude-libs=libvpx.a -Wl,--icf=all -Wl,-shared,-Bsymbolic ../../third_party/android_tools/ndk//platforms/android-16/arch-arm/usr/lib/crtbegin_so.o -Wl,--version-script=/b/build/slave/android/build/src/build/android/android_no_jni_exports.lst -Wl,-O1 -Wl,--as-needed -Wl,--gc-sections -Wl,--warn-shared-textrel -o lib/libnet_java_test_support.so -Wl,-soname=libnet_java_test_support.so @lib/libnet_java_test_support.so.rsp && { readelf -d lib/libnet_java_test_support.so | grep SONAME ; nm -gD -f p lib/libnet_java_test_support.so | cut -f1-2 -d' '; } > lib/libnet_java_test_support.so.TOC; else /b/build/goma/gomacc /b/build/slave/android/build/src/third_party/android_tools/ndk//toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-g++ -shared -Wl,-z,now -Wl,-z,relro -Wl,--fatal-warnings -Wl,-z,defs -Wl,-z,noexecstack -fPIC -B/b/build/slave/android/build/src/third_party/binutils/Linux_x64/Release/bin -Wl,-z,relro -Wl,-z,now -fuse-ld=gold -Wl,--build-id=sha1 -Wl,--no-undefined --sysroot=../../third_party/android_tools/ndk//platforms/android-16/arch-arm -nostdlib -L../../third_party/android_tools/ndk//sources/cxx-stl/llvm-libc++/libs/armeabi-v7a -Wl,--exclude-libs=libgcc.a -Wl,--exclude-libs=libc++_static.a -Wl,--exclude-libs=libcommon_audio.a -Wl,--exclude-libs=libcommon_audio_neon.a -Wl,--exclude-libs=libcommon_audio_sse2.a -Wl,--exclude-libs=libiSACFix.a -Wl,--exclude-libs=libisac_neon.a -Wl,--exclude-libs=libopus.a -Wl,--exclude-libs=libvpx.a -Wl,--icf=all -Wl,-shared,-Bsymbolic ../../third_party/android_tools/ndk//platforms/android-16/arch-arm/usr/lib/crtbegin_so.o -Wl,--version-script=/b/build/slave/android/build/src/build/android/android_no_jni_exports.lst -Wl,-O1 -Wl,--as-needed -Wl,--gc-sections -Wl,--warn-shared-textrel -o lib/libnet_java_test_support.so -Wl,-soname=libnet_java_test_support.so @lib/libnet_java_test_support.so.rsp && { readelf -d lib/libnet_java_test_support.so | grep SONAME ; nm -gD -f p lib/libnet_java_test_support.so | cut -f1-2 -d' '; } > lib/libnet_java_test_support.so.tmp && if ! cmp -s lib/libnet_java_test_support.so.tmp lib/libnet_java_test_support.so.TOC; then mv lib/libnet_java_test_support.so.tmp lib/libnet_java_test_support.so.TOC ; fi; fi
obj/net/libnet_test_support.a(obj/net/test/embedded_test_server/android/net_test_support.embedded_test_server_android.o):embedded_test_server_android.cc:function Java_org_chromium_net_test_EmbeddedTestServerImpl_nativeInit: error: undefined reference to 'base::trace_event::TraceLog::AddTraceEventWithThreadIdAndTimestamp(char, unsigned char const*, char const*, char const*, unsigned long long, unsigned long long, int, base::TimeTicks const&, int, char const**, unsigned char const*, unsigned long long const*, scoped_refptr<base::trace_event::ConvertableToTraceFormat> const*, unsigned int)'
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
happened on
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/31122/steps/compile%20%28without%20patch%29/logs/stdio
and
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/31105/steps/compile%20%28without%20patch%29/logs/stdio
(didnt look on others yet)
,
Mar 1 2016
,
Mar 1 2016
linker failures would lead me to believe this is something bad that landed in src/ rather than an infra-land failure
,
Mar 1 2016
this doesn't appear to be android-specific: https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/189107
,
Mar 1 2016
One interesting data point: this seems to always be linker errors (undefined reference to symbol) for random code that should be inline. Examples: In [1] HTMLAnchorElement.cpp:function blink::HTMLAnchorElement::handleClick(blink::Event*): error: undefined reference to 'blink::LayoutObject::absoluteToLocal(blink::FloatPoint const&, unsigned int) const. But absoluteToLocal defined in the header itself, so how can you have a missing symbol ref to something that is supposed to be inline? In [2] ditto: libnet_test_support.a(obj/net/test/embedded_test_server/android/net_test_support.embedded_test_server_android.o):embedded_test_server_android.cc:function Java_org_chromium_net_test_EmbeddedTestServerImpl_nativeInit: error: undefined reference to 'base::trace_event::TraceLog::AddTraceEventWithThreadIdAndTimestamp But embedded_test_server_android.cc uses a trace macro that expands in AddTraceEventWithThreadIdAndTimestamp, which is defined inline in a tracing .h file. How can that cause a missing symbol ref? Hmmmmmm. [1] https://build.chromium.org/p/tryserver.chromium.android/builders/android_compile_dbg/builds/29193/steps/compile%20%28with%20patch%29/logs/stdio [2] https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/31122/steps/compile%20%28with%20patch%29/logs/stdio
,
Mar 1 2016
primiano & I spent a while looking at this (specifically https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/189248/steps/compile%20%28with%20patch%29/logs/stdio on slave172-c4) canned_syncable_file_system.o wasn't rebuilt after his change to trace_event.h, despite including trace_event.h, and had a lingering reference to the previous scoped_refptr version of AddTraceEventWithThreadIdAndTimestamp
,
Mar 1 2016
from ninja -t deps: obj/chrome/test/unit_tests/canned_syncable_file_system.o: #deps 0, deps mtime 1456761206 (VALID)
,
Mar 1 2016
,
Mar 1 2016
Nico was suggesting that if the machine in the past ended up with the disk full, that might have caused them to be in a weird state. It's definitely weird that some .o files (like the one in #12) have 0 builds dependencies. The weird thing is that, IIUC, this seems to affect more than just one slave. But IIRC there was an issue where lot of bots were running out of disk due to something forgetting to clean the build.stale directory, so that might have caused the broken situation. Does it sound a plausible theory?
,
Mar 1 2016
adding some more folks. picking https://build.chromium.org/p/tryserver.chromium.android/buildslaves/slave481-c4 as an example, it doesn't look like it ran out of space (impossible to tell of course, but it seems unlikely given that it has so much free space). I don't see it on Mac or Windows. was there a clang roll recently?
,
Mar 1 2016
Re #15 clang roll: this happens also on linux_android_rel_ng while building target .so-s, which uses still gcc. So it's doesn't seem to be a clang-only thing.
,
Mar 1 2016
ah you're right. I checked and ninja hasn't been updated in a long time so it's not that.
,
Mar 2 2016
I have no useful theories to add here :( ...
,
Mar 2 2016
I guess the other interesting data point here would be: if we take a bot in a broken state and re-gyp with the same exact arguments, does it still generate a broken ninja file? In other words, is it a flake (smells like that) or repros deterministically?
,
Mar 2 2016
19: note that linux_chromium_rel_ng is on gn, not gyp
,
Mar 2 2016
New trooper checking in. The trybot waterfall looks green now https://build.chromium.org/p/tryserver.chromium.android/builders/android_compile_dbg?numbuilds=200 Is this still an issue? Also, re: #15. A quick way to tell what the disk usage look like is to look at the step text for bot_update. ie. bot_update [177GB/492GB used (35%)]
,
Mar 2 2016
Closing as fixed as there are no more build errors on the builder.
,
Mar 3 2016
Looks like this might still be happening, in a slightly different form: https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_rel_ng/builds/176202 etc. See bug 591834 .
,
Mar 3 2016
Issue 591834 has been merged into this issue.
,
Mar 4 2016
,
Mar 4 2016
(new trooper here) What's the next course of action to debug this? Is this still a p0?
,
Mar 4 2016
@estaab - I think it depends on how often this is happening. I think someone should take a look across most of the linux/cros/android trybots and see if they can get a sense of what percentage of builds this is happening on.
,
Mar 8 2016
Ok, here are a couple of queries to try to move this forward a little. Here's a summary of the number of builds in the last 24 hours with "compile (without patch)" steps that resulted in failure. This is obviously a superset of failures resulting from the bug described in this issue but it gives us an upper bound. The denominator only includes builds that included a "compile (without patch)" step, so the ratio isn't incredibly useful. (dremel query provided in [1]) +-------+-------+-----------------------------------+-------------------------------------+ | ratio | count | master | builder | +-------+-------+-----------------------------------+-------------------------------------+ | 0.05 | 2 | master.tryserver.chromium.android | android_chromium_gn_compile_dbg | | 0.45 | 10 | master.tryserver.chromium.linux | linux_chromium_chromeos_asan_rel_ng | | 0.19 | 13 | master.tryserver.chromium.linux | linux_chromium_chromeos_rel_ng | | 0.82 | 37 | master.tryserver.chromium.mac | mac_chromium_10.10_rel_ng | | 0.24 | 14 | master.tryserver.chromium.mac | mac_chromium_compile_dbg_ng | | 1.0 | 1 | master.tryserver.chromium.perf | android_nexus6_perf_bisect | | 1.0 | 1 | master.tryserver.chromium.perf | android_webview_aosp_perf_bisect | | 0.08 | 5 | master.tryserver.chromium.win | win_chromium_compile_dbg_ng | | 0.05 | 10 | master.tryserver.chromium.win | win_chromium_rel_ng | | 0.04 | 5 | master.tryserver.chromium.win | win_chromium_x64_rel_ng | | 0.33 | 1 | master.tryserver.chromium.win | win_clang_dbg | +-------+-------+-----------------------------------+-------------------------------------+ Here's the full list of links to builds with "compile (without patch)" steps that failed. (query in [2]) +-----------------------------------------------------------------------------------------------------------------+ | link | +-----------------------------------------------------------------------------------------------------------------+ | https://build.chromium.org/p/tryserver.chromium.android/builders/android_chromium_gn_compile_dbg/builds/32316 | | https://build.chromium.org/p/tryserver.chromium.android/builders/android_chromium_gn_compile_dbg/builds/32333 | | https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_asan_rel_ng/builds/10772 | | https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_asan_rel_ng/builds/10776 | | https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_asan_rel_ng/builds/10780 | | https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_asan_rel_ng/builds/10782 | | https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_asan_rel_ng/builds/10811 | | https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_asan_rel_ng/builds/10813 | | https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_asan_rel_ng/builds/10825 | | https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_asan_rel_ng/builds/10827 | | https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_asan_rel_ng/builds/10835 | | https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_asan_rel_ng/builds/10842 | | https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_asan_rel_ng/builds/10843 | | https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_rel_ng/builds/177252 | | https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_rel_ng/builds/177280 | | https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_rel_ng/builds/177324 | | https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_rel_ng/builds/177365 | | https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_rel_ng/builds/177385 | | https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_rel_ng/builds/177401 | | https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_rel_ng/builds/177405 | | https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_rel_ng/builds/177417 | | https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_rel_ng/builds/177436 | | https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_rel_ng/builds/177448 | | https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_rel_ng/builds/177489 | | https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_rel_ng/builds/177514 | | https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_rel_ng/builds/177536 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60486 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60504 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60520 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60561 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60575 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60577 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60589 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60632 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60651 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60670 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60673 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60675 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60686 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60690 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60700 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60705 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60710 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60714 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60718 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60722 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60724 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60730 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60737 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60742 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60754 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60761 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60773 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60781 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60785 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60796 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60798 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60807 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60820 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60821 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60830 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng/builds/60831 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_compile_dbg_ng/builds/169441 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_compile_dbg_ng/builds/169458 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_compile_dbg_ng/builds/169515 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_compile_dbg_ng/builds/169588 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_compile_dbg_ng/builds/169647 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_compile_dbg_ng/builds/169671 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_compile_dbg_ng/builds/169703 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_compile_dbg_ng/builds/169768 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_compile_dbg_ng/builds/169802 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_compile_dbg_ng/builds/169810 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_compile_dbg_ng/builds/169830 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_compile_dbg_ng/builds/169849 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_compile_dbg_ng/builds/169901 | | https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_compile_dbg_ng/builds/169915 | | https://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus6_perf_bisect/builds/2006 | | https://build.chromium.org/p/tryserver.chromium.perf/builders/android_webview_aosp_perf_bisect/builds/65 | | https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_compile_dbg_ng/builds/155769 | | https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_compile_dbg_ng/builds/155921 | | https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_compile_dbg_ng/builds/155969 | | https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_compile_dbg_ng/builds/155998 | | https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_compile_dbg_ng/builds/156035 | | https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/184991 | | https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/185012 | | https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/185067 | | https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/185114 | | https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/185200 | | https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/185252 | | https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/185294 | | https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/185311 | | https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/185319 | | https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/185374 | | https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_rel_ng/builds/178668 | | https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_rel_ng/builds/178881 | | https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_rel_ng/builds/178931 | | https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_rel_ng/builds/179006 | | https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_rel_ng/builds/179063 | | https://build.chromium.org/p/tryserver.chromium.win/builders/win_clang_dbg/builds/269 | +-----------------------------------------------------------------------------------------------------------------+ [1] SELECT ROUND(SUM(result = "FAILURE") / COUNT(*), 2) as ratio, SUM(result = "FAILURE") as count, master, builder FROM chrome_infra.completed_steps WHERE step_name = "compile (without patch)" AND step_started_msec * 1000 >= NOW() - 24*3600*1000*1000 GROUP BY step_name, master, builder HAVING count > 0 ORDER BY master, builder; [2] SELECT CONCAT("https://build.chromium.org/p/", REPLACE(master, "master.", ""), "/builders/", builder, "/builds/", STRING(build_number)) AS link FROM chrome_infra.completed_steps WHERE step_name = "compile (without patch)" AND result = "FAILURE" AND step_started_msec * 1000 >= NOW() - 24*3600*1000*1000 ORDER BY link;
,
Mar 8 2016
And now we're hitting errors with Goma (can't start because of a bug in compile.py apparently). I'll investigate. https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_rel_ng/builds/177964
,
Mar 8 2016
compile.py is innocent. Looking toward bot_update now.
,
Mar 8 2016
And now assigning to phajdan.jr@ who knows what to do.
,
Mar 8 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/b27144c84f1075684f0ec76b4c687e565a536d8c commit b27144c84f1075684f0ec76b4c687e565a536d8c Author: Paweł Hajdan, Jr <phajdan.jr@chromium.org> Date: Tue Mar 08 18:46:06 2016 Fix .gitignore for build/goma This should result in it no longer being removed by bot_update. BUG= 590923 , 585401 R=hinoka@google.com Review URL: https://codereview.chromium.org/1773223002 . Cr-Commit-Position: refs/heads/master@{#379871} [modify] https://crrev.com/b27144c84f1075684f0ec76b4c687e565a536d8c/.gitignore
,
Mar 8 2016
The above fix worked, we got an example of compile failing with patch and being green without patch: https://build.chromium.org/p/tryserver.chromium.android/builders/android_compile_dbg/builds/32895 Thanks Pawel! I'm closing this ticket now since everything seems to be fixed. Please reopen if I'm wrong.
,
Apr 26 2016
|
||||||||||||||||||||||
►
Sign in to add a comment |
||||||||||||||||||||||
Comment 1 by jam@chromium.org
, Mar 1 2016