New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 736991 link

Starred by 2 users

Issue metadata

Status: Verified
Owner:
Closed: Jun 2017
Cc:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug

Blocking:
issue 736000



Sign in to add a comment

All pre-cq / cq builders failing InitSDK/SetupBoard stage due to incorrect prebuilts

Project Member Reported by pprabhu@chromium.org, Jun 27 2017

Issue description

e.g.: https://luci-milo.appspot.com/buildbot/chromiumos.tryserver/pre_cq/41024

metrics-0.0.1-r2183: FAILED: metrics_daemon 
metrics-0.0.1-r2183: flock linker.lock x86_64-pc-linux-gnu-clang++ -Wl,-O2 -Wl,--as-needed -Wl,-O2 -Wl,--as-needed -Wl,-z,relro -Wl,-z,noexecstack -Wl,-z,now -Wl,--as-needed -pie -pthread -Wl,-rpath=\$ORIGIN/lib/ -Wl,-rpath-link=lib/ -o metrics_daemon -Wl,--start-group obj/metrics/libmetrics_daemon.a obj/metrics/libmetrics_proto.a obj/metrics/libupload_service.a lib/libmetrics-395517.so -Wl,--end-group -lrootdev -lvboot_host -ldbus-1 -lbrillo-395517 -lbase-395517 -lprotobuf-lite -lpthread
metrics-0.0.1-r2183: /usr/x86_64-pc-linux-gnu/binutils-bin/2.25.51/ld.bfd.real: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/4.9.x/../../../../lib64/libvboot_host.a(crossystem.o): relocation R_X86_64_PC32 against undefined symbol `stderr@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC
metrics-0.0.1-r2183: /usr/x86_64-pc-linux-gnu/binutils-bin/2.25.51/ld.bfd.real: final link failed: Bad value
metrics-0.0.1-r2183: clang-5.0: error: linker command failed with exit code 1 (use -v to see invocation)
metrics-0.0.1-r2183: ninja: build stopped: subcommand failed.
metrics-0.0.1-r2183: Traceback (most recent call last):
metrics-0.0.1-r2183:   File "/var/tmp/portage/chromeos-base/metrics-0.0.1-r2183/work/metrics-0.0.1/platform2/common-mk/platform2.py", line 333, in <module>
metrics-0.0.1-r2183:     main(sys.argv[1:])
metrics-0.0.1-r2183:   File "/var/tmp/portage/chromeos-base/metrics-0.0.1-r2183/work/metrics-0.0.1/platform2/common-mk/platform2.py", line 329, in main
metrics-0.0.1-r2183:     getattr(p2, options.action)(options.args)
metrics-0.0.1-r2183:   File "/var/tmp/portage/chromeos-base/metrics-0.0.1-r2183/work/metrics-0.0.1/platform2/common-mk/platform2.py", line 270, in compile
metrics-0.0.1-r2183:     % ' '.join(map(repr, ninja_args)))
metrics-0.0.1-r2183: AssertionError: Error running: 'ninja' '-C' '/var/cache/portage/chromeos-base/metrics/out/Default' '-j' '32' 'all'
metrics-0.0.1-r2183:  * ERROR: chromeos-base/metrics-0.0.1-r2183::chromiumos failed (compile phase):
metrics-0.0.1-r2183:  *   (no error message)
metrics-0.0.1-r2183:  * 
metrics-0.0.1-r2183:  * Call stack:
metrics-0.0.1-r2183:  *     ebuild.sh, line   93:  Called src_compile
metrics-0.0.1-r2183:  *   environment, line 3582:  Called platform_src_compile
metrics-0.0.1-r2183:  *   environment, line 3182:  Called platform 'compile' 'all'
metrics-0.0.1-r2183:  *   environment, line 3137:  Called die
metrics-0.0.1-r2183:  * The specific snippet of code:
metrics-0.0.1-r2183:  *       "${cmd[@]}" || die
metrics-0.0.1-r2183:  * 
metrics-0.0.1-r2183:  * If you need support, post the output of `emerge --info '=chromeos-base/metrics-0.0.1-r2183::chromiumos'`,
metrics-0.0.1-r2183:  * the complete build log and the output of `emerge -pqv '=chromeos-base/metrics-0.0.1-r2183::chromiumos'`.
metrics-0.0.1-r2183:  * The complete build log is located at '/var/log/portage/chromeos-base:metrics-0.0.1-r2183:20170627-004910.log'.
metrics-0.0.1-r2183:  * For convenience, a symlink to the build log is located at '/var/tmp/portage/chromeos-base/metrics-0.0.1-r2183/temp/build.log'.
metrics-0.0.1-r2183:  * The ebuild environment file is located at '/var/tmp/portage/chromeos-base/metrics-0.0.1-r2183/temp/environment'.
metrics-0.0.1-r2183:  * Working directory: '/var/tmp/portage/chromeos-base/metrics-0.0.1-r2183/work/metrics-0.0.1/platform2/metrics'
metrics-0.0.1-r2183:  * S: '/var/tmp/portage/chromeos-base/metrics-0.0.1-r2183/work/metrics-0.0.1/platform2/metrics'
metrics-0.0.1-r2183: >>> Failed to emerge chromeos-base/metrics-0.0.1-r2183, Log file:
metrics-0.0.1-r2183: >>>  '/var/log/portage/chromeos-base:metrics-0.0.1-r2183:20170627-004910.log'
metrics-0.0.1-r2183: 
metrics-0.0.1-r2183:  * Messages for package chromeos-base/metrics-0.0.1-r2183:
metrics-0.0.1-r2183: 
metrics-0.0.1-r2183:  * ERROR: chromeos-base/metrics-0.0.1-r2183::chromiumos failed (compile phase):
metrics-0.0.1-r2183:  *   (no error message)
metrics-0.0.1-r2183:  * 
metrics-0.0.1-r2183:  * Call stack:
metrics-0.0.1-r2183:  *     ebuild.sh, line   93:  Called src_compile
metrics-0.0.1-r2183:  *   environment, line 3582:  Called platform_src_compile
metrics-0.0.1-r2183:  *   environment, line 3182:  Called platform 'compile' 'all'
metrics-0.0.1-r2183:  *   environment, line 3137:  Called die
metrics-0.0.1-r2183:  * The specific snippet of code:
metrics-0.0.1-r2183:  *       "${cmd[@]}" || die
metrics-0.0.1-r2183:  * 
metrics-0.0.1-r2183:  * If you need support, post the output of `emerge --info '=chromeos-base/metrics-0.0.1-r2183::chromiumos'`,
metrics-0.0.1-r2183:  * the complete build log and the output of `emerge -pqv '=chromeos-base/metrics-0.0.1-r2183::chromiumos'`.
metrics-0.0.1-r2183:  * The complete build log is located at '/var/log/portage/chromeos-base:metrics-0.0.1-r2183:20170627-004910.log'.
metrics-0.0.1-r2183:  * For convenience, a symlink to the build log is located at '/var/tmp/portage/chromeos-base/metrics-0.0.1-r2183/temp/build.log'.
metrics-0.0.1-r2183:  * The ebuild environment file is located at '/var/tmp/portage/chromeos-base/metrics-0.0.1-r2183/temp/environment'.
metrics-0.0.1-r2183:  * Working directory: '/var/tmp/portage/chromeos-base/metrics-0.0.1-r2183/work/metrics-0.0.1/platform2/metrics'
metrics-0.0.1-r2183:  * S: '/var/tmp/portage/chromeos-base/metrics-0.0.1-r2183/work/metrics-0.0.1/platform2/metrics'
=== Complete: job metrics-0.0.1-r2183 (0m14.3s) ===
 
Cc: rahulchaudhry@chromium.org manojgupta@chromium.org llozano@chromium.org
I think this is a fallout of the toolchain rollback: https://bugs.chromium.org/p/chromium/issues/detail?id=736534#c23
Note that pre-cq is consistently dying right now. Hence P0.
This calls for tree throttling at the least.
Labels: Build-Toolchain
Owner: rahulchaudhry@chromium.org
Status: Assigned (was: Untriaged)
we are taking a look. We will regenerate prebuilds and see if this fixes the issue.


Cc: akes...@chromium.org vapier@chromium.org dgarr...@chromium.org
adding some other people for help. 

I thought the prebuilds should not have been regenerated with the toolchain upgrade but 
it looks like they were regenerated. 

So, We are trying to clean them up following these intructions:

https://sites.google.com/a/google.com/chromeos/for-team-members/infrastructure/chromeos-admin/clear_binaries?pli=1

We have the following CL:
https://chromium-review.googlesource.com/c/549114/

but how is this going to pass if the pre-CQ is failing with the same error.
I may have to chump it.




Project Member

Comment 6 by bugdroid1@chromium.org, Jun 27 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/00f414d2349cc25f2b8817fc9621a51dd5f81bdb

commit 00f414d2349cc25f2b8817fc9621a51dd5f81bdb
Author: Manoj Gupta <manojgupta@google.com>
Date: Tue Jun 27 03:52:38 2017

generic_stages: temporarily ignore all target prebuilts.

Clear prebuilts after bad binutils roll.
This procedure is documented at
https://sites.google.com/a/google.com/chromeos/for-team-members/infrastructure/chromeos-admin/clear_binaries

Revert this CL once it has participated in a green CQ run.

TEST=None
BUG= chromium:736991 

Change-Id: Ib64a00c45c1fc2c60b5e68c135e330ab2d01779b
Reviewed-on: https://chromium-review.googlesource.com/549114
Tested-by: Manoj Gupta <manojgupta@chromium.org>
Tested-by: Luis Lozano <llozano@chromium.org>
Reviewed-by: Luis Lozano <llozano@chromium.org>

[modify] https://crrev.com/00f414d2349cc25f2b8817fc9621a51dd5f81bdb/cbuildbot/config_dump.json
[modify] https://crrev.com/00f414d2349cc25f2b8817fc9621a51dd5f81bdb/cbuildbot/stages/build_stages_unittest.py
[modify] https://crrev.com/00f414d2349cc25f2b8817fc9621a51dd5f81bdb/cbuildbot/stages/generic_stages.py
[modify] https://crrev.com/00f414d2349cc25f2b8817fc9621a51dd5f81bdb/lib/config_lib.py

pre-CQ continued to fail and was not going to let this CL go through. So, I chumped it.
Lets see..

Summary: All pre-cq / cq builders failing InitSDK/SetupBoard stage due to incorrect prebuilts (was: All pre-cq builders failing InitSDK stage)
Chumping is the right thing to do.

I'll interrupt the current CQ run so we get a new one with this chump.
thanks for coming to the rescue! 
Chumped another one to reset target prebuilts https://chromium-review.googlesource.com/c/549719/.
After some more prebuilt cleanup from Manoj thinkgs are looking better.

However, there are some CQ builders like lumpy and parrot that are still failing with some errors like these: 

chromeos-installer-0.0.1-r2508: /usr/x86_64-pc-linux-gnu/i686-pc-linux-gnu/binutils-bin/2.25.51/ld.gold.real: error: /build/parrot/usr/lib/vboot32/libvboot_host.a(cgpt_common.o): unsupported reloc 43 against global symbol stderr

note the vboot32. I assume we have some 32-bit prebuilts that we have not cleaned up..

Trying to figure out where they are.

Labels: OS-Chrome
Manoj fixed this one too: https://chrome-internal-review.googlesource.com/c/401995/

I think these are the last ones...
Not doing more work tonight. 

Comment 13 by sjg@google.com, Jun 27 2017

Cc: sjg@chromium.org

Comment 14 by sjg@google.com, Jun 27 2017

Blocking: 736000
hings are looking much better this morning but the master-paladin is still failing for this unrelated reason:
06:40:55: ERROR: unexpected error error(111, 'Connection refused') while sending data
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/statsd/connection.py", line 77, in send
    self.udp_sock.send(send_data)
error: [Errno 111] Connection refused
06:40:55: ERROR: Could not submit chiranjeevi.rapolu:537094:b066ad4d
06:40:58: INFO: Running cidb query on pid 10481, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x7f26f05115d0>

@@@STEP_FAILURE@@@
06:40:58: ERROR: FAILED TO SUBMIT ALL CHANGES:  Could not verify that changes chiranjeevi.rapolu:537094:b066ad4d were submitted.
Submitted 6 changes successfully.
06:40:58: INFO: Translating result FAILED TO SUBMIT ALL CHANGES:  Could not verify that changes chiranjeevi.rapolu:537094:b066ad4d were submitted.

We need a green master-paladin to be done with the binutils rollback.

Can anyone from infra help with this failure?


Cc: yllin@chromium.org ayatane@chromium.org cernekee@chromium.org
+cernekee, yllin and ayatane (sheriffs and infra deputy)
The message in #15 is due to somebody modifying their CL while it was in the CQ.

Comment 18 by sjg@google.com, Jun 27 2017

Can we mark this fixed now?
Labels: -Pri-0 Pri-1
This can be downgraded to P1. Still need to revert #6 once prebuilts are rolled.
Owner: manojgupta@chromium.org
changing owner to manoj since he has dome most of the work. 

The CL to revert #6 is stuck in CQ. Let us know if we should chump.

https://chromium-review.googlesource.com/c/550318/

Project Member

Comment 21 by bugdroid1@chromium.org, Jun 28 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/a40dd842b04ea49ee3ef68d145aa975856bfd36d

commit a40dd842b04ea49ee3ef68d145aa975856bfd36d
Author: Allen Li <ayatane@chromium.org>
Date: Wed Jun 28 06:50:32 2017

Revert "generic_stages: temporarily ignore all target prebuilts."

This reverts commit 00f414d2349cc25f2b8817fc9621a51dd5f81bdb.

Reason for revert: Green CQ run happened 

Original change's description:
> generic_stages: temporarily ignore all target prebuilts.
> 
> Clear prebuilts after bad binutils roll.
> This procedure is documented at
> https://sites.google.com/a/google.com/chromeos/for-team-members/infrastructure/chromeos-admin/clear_binaries
> 
> Revert this CL once it has participated in a green CQ run.
> 
> TEST=None
> BUG= chromium:736991 
> 
> Change-Id: Ib64a00c45c1fc2c60b5e68c135e330ab2d01779b
> Reviewed-on: https://chromium-review.googlesource.com/549114
> Tested-by: Manoj Gupta <manojgupta@chromium.org>
> Tested-by: Luis Lozano <llozano@chromium.org>
> Reviewed-by: Luis Lozano <llozano@chromium.org>

Bug:  chromium:736991 
Change-Id: Ife4b66458142d76c5230194bb5c968e25c4cc82d
Reviewed-on: https://chromium-review.googlesource.com/550318
Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
Tested-by: Manoj Gupta <manojgupta@chromium.org>

[modify] https://crrev.com/a40dd842b04ea49ee3ef68d145aa975856bfd36d/cbuildbot/config_dump.json
[modify] https://crrev.com/a40dd842b04ea49ee3ef68d145aa975856bfd36d/cbuildbot/stages/build_stages_unittest.py
[modify] https://crrev.com/a40dd842b04ea49ee3ef68d145aa975856bfd36d/cbuildbot/stages/generic_stages.py
[modify] https://crrev.com/a40dd842b04ea49ee3ef68d145aa975856bfd36d/lib/config_lib.py

Status: Verified (was: Assigned)
Components: -Infra>Client>ChromeOS

Sign in to add a comment