New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 595226 link

Starred by 1 user

Issue metadata

Status: Duplicate
Merged: issue 595798
Owner: ----
Closed: Mar 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

Android GN compile step timeout

Project Member Reported by ukai@chromium.org, Mar 16 2016

Issue description

https://build.chromium.org/p/chromium.linux/builders/Android%20GN
e.g.
https://build.chromium.org/p/chromium.linux/builders/Android%20GN/builds/33348

I log in build36-b1.labs while running compile step of
https://build.chromium.org/p/chromium.linux/builders/Android%20GN/builds/33349

memory thrashing by two ld.gold?

chrome-bot@build36-b1:~$ free -h
             total       used       free     shared    buffers     cached
Mem:           23G        23G       191M       2.3M        14M        10G
-/+ buffers/cache:        12G        11G
Swap:          15G       106M        15G
top - 21:59:38 up  2:14,  1 user,  load average: 2.44, 2.47, 2.51
Tasks: 174 total,   1 running, 173 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.0 us,  0.1 sy,  0.0 ni, 87.3 id, 11.6 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:  24520720 total, 24317244 used,   203476 free,    14932 buffers
KiB Swap: 15998972 total,   109516 used, 15889456 free. 11406200 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND     
16611 chrome-+  20   0 15.985g 0.012t 7.712g D   3.3 54.2   1:07.66 ld.gold     
16718 chrome-+  20   0 16.914g 0.010t 5.612g D   3.0 44.3   1:03.32 ld.gold  

          \_ /b/build/scripts/slave/.recipe_deps/depot_tools/ninja-linux64 -C /b/build/slave/Android_GN/build/src/out/Release all android_webview_test_apk android_webview_unittests_apk base_junit_tests base_unittests_apk blimp_unittests_apk
 breakpad_unittests_deps cc_unittests_apk chrome_junit_tests chrome_public_test_apk chrome_sync_shell_test_apk components_browsertests_apk components_invalidation_impl_junit_tests components_policy_junit_tests components_unittests_apk conte
nt_browsertests_apk content_junit_tests content_shell_test_apk content_unittests_apk device_unittests_apk events_unittests_apk gl_tests_apk gl_unittests_apk gpu_unittests_apk ipc_mojo_unittests_apk ipc_tests_apk junit_unittests media_unittests_apk mojo_common_unittests_apk mojo_public_application_unittests_apk mojo_public_bindings_unittests_apk mojo_public_system_unittests_apk mojo_runner_host_unittests_apk mojo_surfaces_lib_unittests_apk mojo_system_unittests_apk mojo_test_apk net_junit_tests net_unittests_apk sandbox_linux_unittests_deps sql_unittests_apk sync_unit_tests_apk ui_android_unittests_apk ui_base_unittests_apk ui_junit_tests ui_touch_selection_unittests_apk unit_tests_apk -j50
chrome-+ 16510  0.0  0.0   4444   660 ?        Ss   21:52   0:00                
              \_ /bin/sh -c python "/b/build/slave/Android_GN/build/src/build/toolchain/gcc_solink_wrapper.py" --readelf="../../third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androidea
bi-readelf" --nm="../../third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-nm" --strip=../../third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-strip --sofile="./lib.unstripped/lib_components_browsertests__library.so" --tocfile="./lib_components_browsertests__library.so.TOC" --output="./lib_components_browsertests__library.so" -- ../../third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-g++ -shared -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -Wl,-z,defs -fuse-ld=gold -Wl,--build-id=sha1 -Wl,--no-undefined -Wl,--exclude-libs=libgcc.a -Wl,--exclude-libs=libc++_static.a -Wl,--exclude-libs=libvpx_assembly_arm.a -Wl,--icf=all -Wl,--warn-shared-textrel -Wl,-O1 -Wl,--gc-sections -Wl,--as-needed -nostdlib -Wl,--warn-shared-textrel --sysroot=../../third_party/android_tools/ndk/platforms/android-16/arch-arm  -Wl,--version-script=/b/build/slave/Android_GN/build/src/build/android/android_no_jni_exports.lst -L../../third_party/android_tools/ndk/sources/cxx-stl/llvm-libc++/libs/armeabi-v7a -o "./lib.unstripped/lib_components_browsertests__library.so" -Wl,-soname="lib_components_browsertests__library.so" @"./lib_components_browsertests__library.so.rsp"
chrome-+ 16511  0.0  0.0  27384  6208 ?        S    21:52   0:00                
              |   \_ python /b/build/slave/Android_GN/build/src/build/toolchain/
gcc_solink_wrapper.py --readelf=../../third_party/android_tools/ndk/toolchains/a
rm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-readelf
 --nm=../../third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-nm --strip=../../third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-strip --sofile=./lib.unstripped/lib_components_browsertests__library.so --tocfile=./lib_components_browsertests__library.so.TOC --output=./lib_components_browsertests__library.so -- ../../third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-g++ -shared -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -Wl,-z,defs -fuse-ld=gold -Wl,--build-id=sha1 -Wl,--no-undefined -Wl,--exclude-libs=libgcc.a -Wl,--exclude-libs=libc++_static.a -Wl,--exclude-libs=libvpx_assembly_arm.a -Wl,--icf=all -Wl,--warn-shared-textrel -Wl,-O1 -Wl,--gc-sections -Wl,--as-needed -nostdlib -Wl,--warn-shared-textrel --sysroot=../../third_party/android_tools/ndk/platforms/android-16/arch-arm -Wl,--version-script=/b/build/slave/Android_GN/build/src/build/android/android_no_jni_exports.lst -L../../third_party/android_tools/ndk/sources/cxx-stl/llvm-libc++/libs/armeabi-v7a -o ./lib.unstripped/lib_components_browsertests__library.so -Wl,-soname=lib_components_browsertests__library.so @./lib_components_browsertests__library.so.rsp
chrome-+ 16514  0.0  0.0  11312  4084 ?        S    21:52   0:00                              |       \_ ../../third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-g++ -shared -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -Wl,-z,defs -fuse-ld=gold -Wl,--build-id=sha1 -Wl,--no-undefined -Wl,--exclude-libs=libgcc.a -Wl,--exclude-libs=libc++_static.a -Wl,--exclude-libs=libvpx_assembly_arm.a -Wl,--icf=all -Wl,--warn-shared-textrel -Wl,-O1 -Wl,--gc-sections -Wl,--as-needed -nostdlib -Wl,--warn-shared-textrel --sysroot=../../third_party/android_tools/ndk/platforms/android-16/arch-arm -Wl,--version-script=/b/build/slave/Android_GN/build/src/build/android/android_no_jni_exports.lst -L../../third_party/android_tools/ndk/sources/cxx-stl/llvm-libc++/libs/armeabi-v7a -o ./lib.unstripped/lib_components_browsertests__library.so -Wl,-soname=lib_components_browsertests__library.so @./lib_components_browsertests__library.so.rsp
chrome-+ 16610  0.0  0.0   9060  1856 ?        S    21:52   0:00                              |           \_ /b/build/slave/Android_GN/build/src/third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/../libexec/gcc/arm-linux-androideabi/4.9/collect2 -plugin /b/build/slave/Android_GN/build/src/third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/../libexec/gcc/arm-linux-androideabi/4.9/liblto_plugin.so -plugin-opt=/b/build/slave/Android_GN/build/src/third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/../libexec/gcc/arm-linux-androideabi/4.9/lto-wrapper -plugin-opt=-fresolution=/tmp/ccopueOc.res --sysroot=../../third_party/android_tools/ndk/platforms/android-16/arch-arm --eh-frame-hdr -shared -dynamic-linker /system/bin/linker -X -m armelf_linux_eabi -Bsymbolic -z noexecstack -z relro -z now -fuse-ld=gold -o ./lib.unstripped/lib_components_browsertests__library.so -L../../third_party/android_tools/ndk/sources/cxx-stl/llvm-libc++/libs/armeabi-v7a -L/b/build/slave/Android_GN/build/src/third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.9 -L/b/build/slave/Android_GN/build/src/third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/../lib/gcc -L/b/build/slave/Android_GN/build/src/third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.9/../../../../arm-linux-androideabi/lib -L../../third_party/android_tools/ndk/platforms/android-16/arch-arm/usr/lib @/tmp/cccvI6rf
chrome-+ 16611 13.7 52.9 16823708 12985048 ?   D    21:52   1:12                              |               \_ /b/build/slave/Android_GN/build/src/third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.9/../../../../arm-linux-androideabi/bin/ld.gold @/tmp/ccWmKZJq
chrome-+ 16615  0.0  0.0   4444   660 ?        Ss   21:52   0:00                              \_ /bin/sh -c python "/b/build/slave/Android_GN/build/src/build/toolchain/gcc_solink_wrapper.py" --readelf="../../third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-readelf" --nm="../../third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-nm" --strip=../../third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-strip --sofile="./lib.unstripped/lib_components_unittests__library.so" --tocfile="./lib_components_unittests__library.so.TOC" --output="./lib_components_unittests__library.so" -- ../../third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-g++ -shared -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -Wl,-z,defs -fuse-ld=gold -Wl,--build-id=sha1 -Wl,--no-undefined -Wl,--exclude-libs=libgcc.a -Wl,--exclude-libs=libc++_static.a -Wl,--exclude-libs=libvpx_assembly_arm.a -Wl,--icf=all -Wl,--warn-shared-textrel -Wl,-O1 -Wl,--gc-sections -Wl,--as-needed -nostdlib -Wl,--warn-shared-textrel --sysroot=../../third_party/android_tools/ndk/platforms/android-16/arch-arm  -Wl,--version-script=/b/build/slave/Android_GN/build/src/build/android/android_no_jni_exports.lst -L../../third_party/android_tools/ndk/sources/cxx-stl/llvm-libc++/libs/armeabi-v7a -o "./lib.unstripped/lib_components_unittests__library.so" -Wl,-soname="lib_components_unittests__library.so" @"./lib_components_unittests__library.so.rsp"
chrome-+ 16617  0.0  0.0  27384  6212 ?        S    21:52   0:00                                  \_ python /b/build/slave/Android_GN/build/src/build/toolchain/gcc_solink_wrapper.py --readelf=../../third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-readelf --nm=../../third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-nm --strip=../../third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-strip --sofile=./lib.unstripped/lib_components_unittests__library.so --tocfile=./lib_components_unittests__library.so.TOC --output=./lib_components_unittests__library.so -- ../../third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-g++ -shared -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -Wl,-z,defs -fuse-ld=gold -Wl,--build-id=sha1 -Wl,--no-undefined -Wl,--exclude-libs=libgcc.a -Wl,--exclude-libs=libc++_static.a -Wl,--exclude-libs=libvpx_assembly_arm.a -Wl,--icf=all -Wl,--warn-shared-textrel -Wl,-O1 -Wl,--gc-sections -Wl,--as-needed -nostdlib -Wl,--warn-shared-textrel --sysroot=../../third_party/android_tools/ndk/platforms/android-16/arch-arm -Wl,--version-script=/b/build/slave/Android_GN/build/src/build/android/android_no_jni_exports.lst -L../../third_party/android_tools/ndk/sources/cxx-stl/llvm-libc++/libs/armeabi-v7a -o ./lib.unstripped/lib_components_unittests__library.so -Wl,-soname=lib_components_unittests__library.so @./lib_components_unittests__library.so.rsp
chrome-+ 16621  0.0  0.0  11872  4344 ?        S    21:52   0:00                                      \_ ../../third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-g++ -shared -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -Wl,-z,defs -fuse-ld=gold -Wl,--build-id=sha1 -Wl,--no-undefined -Wl,--exclude-libs=libgcc.a -Wl,--exclude-libs=libc++_static.a -Wl,--exclude-libs=libvpx_assembly_arm.a -Wl,--icf=all -Wl,--warn-shared-textrel -Wl,-O1 -Wl,--gc-sections -Wl,--as-needed -nostdlib -Wl,--warn-shared-textrel --sysroot=../../third_party/android_tools/ndk/platforms/android-16/arch-arm -Wl,--version-script=/b/build/slave/Android_GN/build/src/build/android/android_no_jni_exports.lst -L../../third_party/android_tools/ndk/sources/cxx-stl/llvm-libc++/libs/armeabi-v7a -o ./lib.unstripped/lib_components_unittests__library.so -Wl,-soname=lib_components_unittests__library.so @./lib_components_unittests__library.so.rsp
chrome-+ 16717  0.0  0.0   9084  2012 ?        S    21:52   0:00                                          \_ /b/build/slave/Android_GN/build/src/third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/../libexec/gcc/arm-linux-androideabi/4.9/collect2 -plugin /b/build/slave/Android_GN/build/src/third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/../libexec/gcc/arm-linux-androideabi/4.9/liblto_plugin.so -plugin-opt=/b/build/slave/Android_GN/build/src/third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/../libexec/gcc/arm-linux-androideabi/4.9/lto-wrapper -plugin-opt=-fresolution=/tmp/ccttwYJB.res --sysroot=../../third_party/android_tools/ndk/platforms/android-16/arch-arm --eh-frame-hdr -shared -dynamic-linker /system/bin/linker -X -m armelf_linux_eabi -Bsymbolic -z noexecstack -z relro -z now -fuse-ld=gold -o ./lib.unstripped/lib_components_unittests__library.so -L../../third_party/android_tools/ndk/sources/cxx-stl/llvm-libc++/libs/armeabi-v7a -L/b/build/slave/Android_GN/build/src/third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.9 -L/b/build/slave/Android_GN/build/src/third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/../lib/gcc -L/b/build/slave/Android_GN/build/src/third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.9/../../../../arm-linux-androideabi/lib -L../../third_party/android_tools/ndk/platforms/android-16/arch-arm/usr/lib @/tmp/ccEwGdj3
chrome-+ 16718 14.7 57.4 18525808 14085636 ?   D    21:52   1:17                                              \_ /b/build/slave/Android_GN/build/src/third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.9/../../../../arm-linux-androideabi/bin/ld.gold @/tmp/ccHIJmPj

 

Comment 1 by ukai@chromium.org, Mar 16 2016

one ld.gold seems to be finished and java steps started..
too slow in ld.gold?

Comment 2 by ukai@chromium.org, Mar 16 2016

compile step time

#33340 1:46:01
#33341 0:59:06
#33342 1:12:28
#33343 1:14:09
#33344 1:33:44
#33345 0:54:45
#33346 1:18:24
#33347 3:02:28
#33348 timed out

some CL in #33346? 
https://build.chromium.org/p/chromium.linux/builders/Android%20GN/builds/33346

Comment 3 by ukai@chromium.org, Mar 16 2016

need to fix mem_per_link_gb in build/toolchain/get_concurrent_links.py ?

Comment 4 by ukai@chromium.org, Mar 16 2016

Cc: krasin@chromium.org brettw@chromium.org

Comment 5 by ukai@chromium.org, Mar 16 2016

oops. i mean #33347 in Comment2
https://build.chromium.org/p/chromium.linux/builders/Android%20GN/builds/33347
(compile time from ~1 hour to >3 hours)

Comment 7 by ukai@chromium.org, Mar 16 2016

note Android GN (dbg) is not so slow...
https://build.chromium.org/p/chromium.linux/builders/Android%20GN%20%28dbg%29/

Link time optimization or something needs cpu and memory?
Labels: -Pri-2 Infra Pri-1
This is closing the tree regularly.

Comment 9 by ukai@chromium.org, Mar 18 2016

Mergedinto: 595798
Status: Duplicate (was: Untriaged)

Sign in to add a comment