All pre-cq / cq builders failing InitSDK/SetupBoard stage due to incorrect prebuilts |
||||||||||||
Issue descriptione.g.: https://luci-milo.appspot.com/buildbot/chromiumos.tryserver/pre_cq/41024 metrics-0.0.1-r2183: FAILED: metrics_daemon metrics-0.0.1-r2183: flock linker.lock x86_64-pc-linux-gnu-clang++ -Wl,-O2 -Wl,--as-needed -Wl,-O2 -Wl,--as-needed -Wl,-z,relro -Wl,-z,noexecstack -Wl,-z,now -Wl,--as-needed -pie -pthread -Wl,-rpath=\$ORIGIN/lib/ -Wl,-rpath-link=lib/ -o metrics_daemon -Wl,--start-group obj/metrics/libmetrics_daemon.a obj/metrics/libmetrics_proto.a obj/metrics/libupload_service.a lib/libmetrics-395517.so -Wl,--end-group -lrootdev -lvboot_host -ldbus-1 -lbrillo-395517 -lbase-395517 -lprotobuf-lite -lpthread metrics-0.0.1-r2183: /usr/x86_64-pc-linux-gnu/binutils-bin/2.25.51/ld.bfd.real: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/4.9.x/../../../../lib64/libvboot_host.a(crossystem.o): relocation R_X86_64_PC32 against undefined symbol `stderr@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC metrics-0.0.1-r2183: /usr/x86_64-pc-linux-gnu/binutils-bin/2.25.51/ld.bfd.real: final link failed: Bad value metrics-0.0.1-r2183: clang-5.0: error: linker command failed with exit code 1 (use -v to see invocation) metrics-0.0.1-r2183: ninja: build stopped: subcommand failed. metrics-0.0.1-r2183: Traceback (most recent call last): metrics-0.0.1-r2183: File "/var/tmp/portage/chromeos-base/metrics-0.0.1-r2183/work/metrics-0.0.1/platform2/common-mk/platform2.py", line 333, in <module> metrics-0.0.1-r2183: main(sys.argv[1:]) metrics-0.0.1-r2183: File "/var/tmp/portage/chromeos-base/metrics-0.0.1-r2183/work/metrics-0.0.1/platform2/common-mk/platform2.py", line 329, in main metrics-0.0.1-r2183: getattr(p2, options.action)(options.args) metrics-0.0.1-r2183: File "/var/tmp/portage/chromeos-base/metrics-0.0.1-r2183/work/metrics-0.0.1/platform2/common-mk/platform2.py", line 270, in compile metrics-0.0.1-r2183: % ' '.join(map(repr, ninja_args))) metrics-0.0.1-r2183: AssertionError: Error running: 'ninja' '-C' '/var/cache/portage/chromeos-base/metrics/out/Default' '-j' '32' 'all' metrics-0.0.1-r2183: * ERROR: chromeos-base/metrics-0.0.1-r2183::chromiumos failed (compile phase): metrics-0.0.1-r2183: * (no error message) metrics-0.0.1-r2183: * metrics-0.0.1-r2183: * Call stack: metrics-0.0.1-r2183: * ebuild.sh, line 93: Called src_compile metrics-0.0.1-r2183: * environment, line 3582: Called platform_src_compile metrics-0.0.1-r2183: * environment, line 3182: Called platform 'compile' 'all' metrics-0.0.1-r2183: * environment, line 3137: Called die metrics-0.0.1-r2183: * The specific snippet of code: metrics-0.0.1-r2183: * "${cmd[@]}" || die metrics-0.0.1-r2183: * metrics-0.0.1-r2183: * If you need support, post the output of `emerge --info '=chromeos-base/metrics-0.0.1-r2183::chromiumos'`, metrics-0.0.1-r2183: * the complete build log and the output of `emerge -pqv '=chromeos-base/metrics-0.0.1-r2183::chromiumos'`. metrics-0.0.1-r2183: * The complete build log is located at '/var/log/portage/chromeos-base:metrics-0.0.1-r2183:20170627-004910.log'. metrics-0.0.1-r2183: * For convenience, a symlink to the build log is located at '/var/tmp/portage/chromeos-base/metrics-0.0.1-r2183/temp/build.log'. metrics-0.0.1-r2183: * The ebuild environment file is located at '/var/tmp/portage/chromeos-base/metrics-0.0.1-r2183/temp/environment'. metrics-0.0.1-r2183: * Working directory: '/var/tmp/portage/chromeos-base/metrics-0.0.1-r2183/work/metrics-0.0.1/platform2/metrics' metrics-0.0.1-r2183: * S: '/var/tmp/portage/chromeos-base/metrics-0.0.1-r2183/work/metrics-0.0.1/platform2/metrics' metrics-0.0.1-r2183: >>> Failed to emerge chromeos-base/metrics-0.0.1-r2183, Log file: metrics-0.0.1-r2183: >>> '/var/log/portage/chromeos-base:metrics-0.0.1-r2183:20170627-004910.log' metrics-0.0.1-r2183: metrics-0.0.1-r2183: * Messages for package chromeos-base/metrics-0.0.1-r2183: metrics-0.0.1-r2183: metrics-0.0.1-r2183: * ERROR: chromeos-base/metrics-0.0.1-r2183::chromiumos failed (compile phase): metrics-0.0.1-r2183: * (no error message) metrics-0.0.1-r2183: * metrics-0.0.1-r2183: * Call stack: metrics-0.0.1-r2183: * ebuild.sh, line 93: Called src_compile metrics-0.0.1-r2183: * environment, line 3582: Called platform_src_compile metrics-0.0.1-r2183: * environment, line 3182: Called platform 'compile' 'all' metrics-0.0.1-r2183: * environment, line 3137: Called die metrics-0.0.1-r2183: * The specific snippet of code: metrics-0.0.1-r2183: * "${cmd[@]}" || die metrics-0.0.1-r2183: * metrics-0.0.1-r2183: * If you need support, post the output of `emerge --info '=chromeos-base/metrics-0.0.1-r2183::chromiumos'`, metrics-0.0.1-r2183: * the complete build log and the output of `emerge -pqv '=chromeos-base/metrics-0.0.1-r2183::chromiumos'`. metrics-0.0.1-r2183: * The complete build log is located at '/var/log/portage/chromeos-base:metrics-0.0.1-r2183:20170627-004910.log'. metrics-0.0.1-r2183: * For convenience, a symlink to the build log is located at '/var/tmp/portage/chromeos-base/metrics-0.0.1-r2183/temp/build.log'. metrics-0.0.1-r2183: * The ebuild environment file is located at '/var/tmp/portage/chromeos-base/metrics-0.0.1-r2183/temp/environment'. metrics-0.0.1-r2183: * Working directory: '/var/tmp/portage/chromeos-base/metrics-0.0.1-r2183/work/metrics-0.0.1/platform2/metrics' metrics-0.0.1-r2183: * S: '/var/tmp/portage/chromeos-base/metrics-0.0.1-r2183/work/metrics-0.0.1/platform2/metrics' === Complete: job metrics-0.0.1-r2183 (0m14.3s) ===
,
Jun 27 2017
Note that pre-cq is consistently dying right now. Hence P0. This calls for tree throttling at the least.
,
Jun 27 2017
we are taking a look. We will regenerate prebuilds and see if this fixes the issue.
,
Jun 27 2017
,
Jun 27 2017
adding some other people for help. I thought the prebuilds should not have been regenerated with the toolchain upgrade but it looks like they were regenerated. So, We are trying to clean them up following these intructions: https://sites.google.com/a/google.com/chromeos/for-team-members/infrastructure/chromeos-admin/clear_binaries?pli=1 We have the following CL: https://chromium-review.googlesource.com/c/549114/ but how is this going to pass if the pre-CQ is failing with the same error. I may have to chump it.
,
Jun 27 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/00f414d2349cc25f2b8817fc9621a51dd5f81bdb commit 00f414d2349cc25f2b8817fc9621a51dd5f81bdb Author: Manoj Gupta <manojgupta@google.com> Date: Tue Jun 27 03:52:38 2017 generic_stages: temporarily ignore all target prebuilts. Clear prebuilts after bad binutils roll. This procedure is documented at https://sites.google.com/a/google.com/chromeos/for-team-members/infrastructure/chromeos-admin/clear_binaries Revert this CL once it has participated in a green CQ run. TEST=None BUG= chromium:736991 Change-Id: Ib64a00c45c1fc2c60b5e68c135e330ab2d01779b Reviewed-on: https://chromium-review.googlesource.com/549114 Tested-by: Manoj Gupta <manojgupta@chromium.org> Tested-by: Luis Lozano <llozano@chromium.org> Reviewed-by: Luis Lozano <llozano@chromium.org> [modify] https://crrev.com/00f414d2349cc25f2b8817fc9621a51dd5f81bdb/cbuildbot/config_dump.json [modify] https://crrev.com/00f414d2349cc25f2b8817fc9621a51dd5f81bdb/cbuildbot/stages/build_stages_unittest.py [modify] https://crrev.com/00f414d2349cc25f2b8817fc9621a51dd5f81bdb/cbuildbot/stages/generic_stages.py [modify] https://crrev.com/00f414d2349cc25f2b8817fc9621a51dd5f81bdb/lib/config_lib.py
,
Jun 27 2017
pre-CQ continued to fail and was not going to let this CL go through. So, I chumped it. Lets see..
,
Jun 27 2017
Chumping is the right thing to do. I'll interrupt the current CQ run so we get a new one with this chump.
,
Jun 27 2017
thanks for coming to the rescue!
,
Jun 27 2017
Chumped another one to reset target prebuilts https://chromium-review.googlesource.com/c/549719/.
,
Jun 27 2017
After some more prebuilt cleanup from Manoj thinkgs are looking better. However, there are some CQ builders like lumpy and parrot that are still failing with some errors like these: chromeos-installer-0.0.1-r2508: /usr/x86_64-pc-linux-gnu/i686-pc-linux-gnu/binutils-bin/2.25.51/ld.gold.real: error: /build/parrot/usr/lib/vboot32/libvboot_host.a(cgpt_common.o): unsupported reloc 43 against global symbol stderr note the vboot32. I assume we have some 32-bit prebuilts that we have not cleaned up.. Trying to figure out where they are.
,
Jun 27 2017
Manoj fixed this one too: https://chrome-internal-review.googlesource.com/c/401995/ I think these are the last ones... Not doing more work tonight.
,
Jun 27 2017
,
Jun 27 2017
,
Jun 27 2017
hings are looking much better this morning but the master-paladin is still failing for this unrelated reason:
06:40:55: ERROR: unexpected error error(111, 'Connection refused') while sending data
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/statsd/connection.py", line 77, in send
self.udp_sock.send(send_data)
error: [Errno 111] Connection refused
06:40:55: ERROR: Could not submit chiranjeevi.rapolu:537094:b066ad4d
06:40:58: INFO: Running cidb query on pid 10481, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x7f26f05115d0>
@@@STEP_FAILURE@@@
06:40:58: ERROR: FAILED TO SUBMIT ALL CHANGES: Could not verify that changes chiranjeevi.rapolu:537094:b066ad4d were submitted.
Submitted 6 changes successfully.
06:40:58: INFO: Translating result FAILED TO SUBMIT ALL CHANGES: Could not verify that changes chiranjeevi.rapolu:537094:b066ad4d were submitted.
We need a green master-paladin to be done with the binutils rollback.
Can anyone from infra help with this failure?
,
Jun 27 2017
+cernekee, yllin and ayatane (sheriffs and infra deputy)
,
Jun 27 2017
The message in #15 is due to somebody modifying their CL while it was in the CQ.
,
Jun 27 2017
Can we mark this fixed now?
,
Jun 27 2017
This can be downgraded to P1. Still need to revert #6 once prebuilts are rolled.
,
Jun 27 2017
changing owner to manoj since he has dome most of the work. The CL to revert #6 is stuck in CQ. Let us know if we should chump. https://chromium-review.googlesource.com/c/550318/
,
Jun 28 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/a40dd842b04ea49ee3ef68d145aa975856bfd36d commit a40dd842b04ea49ee3ef68d145aa975856bfd36d Author: Allen Li <ayatane@chromium.org> Date: Wed Jun 28 06:50:32 2017 Revert "generic_stages: temporarily ignore all target prebuilts." This reverts commit 00f414d2349cc25f2b8817fc9621a51dd5f81bdb. Reason for revert: Green CQ run happened Original change's description: > generic_stages: temporarily ignore all target prebuilts. > > Clear prebuilts after bad binutils roll. > This procedure is documented at > https://sites.google.com/a/google.com/chromeos/for-team-members/infrastructure/chromeos-admin/clear_binaries > > Revert this CL once it has participated in a green CQ run. > > TEST=None > BUG= chromium:736991 > > Change-Id: Ib64a00c45c1fc2c60b5e68c135e330ab2d01779b > Reviewed-on: https://chromium-review.googlesource.com/549114 > Tested-by: Manoj Gupta <manojgupta@chromium.org> > Tested-by: Luis Lozano <llozano@chromium.org> > Reviewed-by: Luis Lozano <llozano@chromium.org> Bug: chromium:736991 Change-Id: Ife4b66458142d76c5230194bb5c968e25c4cc82d Reviewed-on: https://chromium-review.googlesource.com/550318 Reviewed-by: Manoj Gupta <manojgupta@chromium.org> Tested-by: Manoj Gupta <manojgupta@chromium.org> [modify] https://crrev.com/a40dd842b04ea49ee3ef68d145aa975856bfd36d/cbuildbot/config_dump.json [modify] https://crrev.com/a40dd842b04ea49ee3ef68d145aa975856bfd36d/cbuildbot/stages/build_stages_unittest.py [modify] https://crrev.com/a40dd842b04ea49ee3ef68d145aa975856bfd36d/cbuildbot/stages/generic_stages.py [modify] https://crrev.com/a40dd842b04ea49ee3ef68d145aa975856bfd36d/lib/config_lib.py
,
Jun 28 2017
,
Jan 4 2018
|
||||||||||||
►
Sign in to add a comment |
||||||||||||
Comment 1 by pprabhu@chromium.org
, Jun 27 2017