New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 836296 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: May 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug
Build-Toolchain

Blocked on:
issue 836344



Sign in to add a comment

terra-chrome-pfq fails BuildPackages

Project Member Reported by tbarzic@chromium.org, Apr 24 2018

Issue description

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8948379408418693600

chromeos-chrome-68.0.3405.0_rc-r2: [43311/43311] LINK ./chrome
chromeos-chrome-68.0.3405.0_rc-r2: Copying Chrome tests into /build/terra/tmp/portage/chromeos-base/chromeos-chrome-68.0.3405.0_rc-r2/work/test_src
chromeos-chrome-68.0.3405.0_rc-r2:  * Installing test targets: jpeg_decode_accelerator_unittest media_unittests sandbox_linux_unittests video_decode_accelerator_unittest video_encode_accelerator_unittest wayland_client_perftests ppapi/examples/video_decode libppapi_tests.so chrome_sandbox
chromeos-chrome-68.0.3405.0_rc-r2: x86_64-cros-linux-gnu-strip: '/var/cache/chromeos-chrome/chrome-src-internal/src/out_terra/Release/ppapi/examples/video_decode': No such file
chromeos-chrome-68.0.3405.0_rc-r2: x86_64-cros-linux-gnu-strip: '/var/cache/chromeos-chrome/chrome-src-internal/src/out_terra/Release/libppapi_tests.so': No such file
chromeos-chrome-68.0.3405.0_rc-r2: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_terra/Release/pseudo_locales/fake-bidi.pak': No such file or directory
chromeos-chrome-68.0.3405.0_rc-r2: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_terra/Release/lib32': No such file or directory
chromeos-chrome-68.0.3405.0_rc-r2: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_terra/Release/lib64': No such file or directory
chromeos-chrome-68.0.3405.0_rc-r2: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_terra/Release/mock_nacl_gdb': No such file or directory
chromeos-chrome-68.0.3405.0_rc-r2: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_terra/Release/ppapi_nacl_tests_newlib.nmf': No such file or directory
chromeos-chrome-68.0.3405.0_rc-r2: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_terra/Release/ppapi_nacl_tests_glibc.nmf': No such file or directory
chromeos-chrome-68.0.3405.0_rc-r2: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_terra/Release/ppapi_nacl_tests_newlib_x32.nexe': No such file or directory
chromeos-chrome-68.0.3405.0_rc-r2: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_terra/Release/ppapi_nacl_tests_newlib_x64.nexe': No such file or directory
chromeos-chrome-68.0.3405.0_rc-r2: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_terra/Release/ppapi_nacl_tests_newlib_arm.nexe': No such file or directory
chromeos-chrome-68.0.3405.0_rc-r2: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_terra/Release/ppapi_nacl_tests_glibc_x32.nexe': No such file or directory
chromeos-chrome-68.0.3405.0_rc-r2: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_terra/Release/ppapi_nacl_tests_glibc_x64.nexe': No such file or directory
chromeos-chrome-68.0.3405.0_rc-r2: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_terra/Release/ppapi_nacl_tests_glibc_arm.nexe': No such file or directory
chromeos-chrome-68.0.3405.0_rc-r2: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_terra/Release/test_case.html': No such file or directory
chromeos-chrome-68.0.3405.0_rc-r2: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_terra/Release/test_case.html.mock-http-headers': No such file or directory
chromeos-chrome-68.0.3405.0_rc-r2: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_terra/Release/test_page.css': No such file or directory
chromeos-chrome-68.0.3405.0_rc-r2: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_terra/Release/test_url_loader_data': No such file or directory
chromeos-chrome-68.0.3405.0_rc-r2:  * install_test_resources to /build/terra/tmp/portage/chromeos-base/chromeos-chrome-68.0.3405.0_rc-r2/work/test_src
chromeos-chrome-68.0.3405.0_rc-r2: rsync: link_stat "/home/chrome-bot/chrome_root/src/chrome/test/functional" failed: No such file or directory (2)
chromeos-chrome-68.0.3405.0_rc-r2: rsync: link_stat "/home/chrome-bot/chrome_root/src/content/common/gpu/testdata" failed: No such file or directory (2)
chromeos-chrome-68.0.3405.0_rc-r2: rsync: link_stat "/home/chrome-bot/chrome_root/src/third_party/accessibility-developer-tools/gen/axs_testing.js" failed: No such file or directory (2)
chromeos-chrome-68.0.3405.0_rc-r2: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
chromeos-chrome-68.0.3405.0_rc-r2: rsync: link_stat "/var/cache/chromeos-chrome/chrome-src-internal/src/chrome/test/functional" failed: No such file or directory (2)
chromeos-chrome-68.0.3405.0_rc-r2: rsync: link_stat "/var/cache/chromeos-chrome/chrome-src-internal/src/content/common/gpu/testdata" failed: No such file or directory (2)
chromeos-chrome-68.0.3405.0_rc-r2: rsync: link_stat "/var/cache/chromeos-chrome/chrome-src-internal/src/third_party/accessibility-developer-tools/gen/axs_testing.js" failed: No such file or directory (2)
chromeos-chrome-68.0.3405.0_rc-r2: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
chromeos-chrome-68.0.3405.0_rc-r2:  * install_test_resources to /build/terra/tmp/portage/chromeos-base/chromeos-chrome-68.0.3405.0_rc-r2/work/test_src
chromeos-chrome-68.0.3405.0_rc-r2: rsync: link_stat "/home/chrome-bot/chrome_root/src/pdf/test" failed: No such file or directory (2)
chromeos-chrome-68.0.3405.0_rc-r2: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
chromeos-chrome-68.0.3405.0_rc-r2: rsync: link_stat "/var/cache/chromeos-chrome/chrome-src-internal/src/pdf/test" failed: No such file or directory (2)
chromeos-chrome-68.0.3405.0_rc-r2: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
chromeos-chrome-68.0.3405.0_rc-r2:  * install_test_resources to /build/terra/tmp/portage/chromeos-base/chromeos-chrome-68.0.3405.0_rc-r2/work/test_src
chromeos-chrome-68.0.3405.0_rc-r2: Copying Page Cycler Data into /build/terra/tmp/portage/chromeos-base/chromeos-chrome-68.0.3405.0_rc-r2/work/page_cycler_src
chromeos-chrome-68.0.3405.0_rc-r2:  * install_test_resources to /build/terra/tmp/portage/chromeos-base/chromeos-chrome-68.0.3405.0_rc-r2/work/page_cycler_src
chromeos-chrome-68.0.3405.0_rc-r2: Copying Perf Data into /build/terra/tmp/portage/chromeos-base/chromeos-chrome-68.0.3405.0_rc-r2/work/perf_data_src
chromeos-chrome-68.0.3405.0_rc-r2:  * install_test_resources to /build/terra/tmp/portage/chromeos-base/chromeos-chrome-68.0.3405.0_rc-r2/work/perf_data_src
chromeos-chrome-68.0.3405.0_rc-r2: Copying Telemetry Framework into /build/terra/tmp/portage/chromeos-base/chromeos-chrome-68.0.3405.0_rc-r2/work/telemetry_src
chromeos-chrome-68.0.3405.0_rc-r2: WARNING:root:Unable to import cv2 due to: No module named cv2
chromeos-chrome-68.0.3405.0_rc-r2:  * install_test_resources to /build/terra/tmp/portage/chromeos-base/chromeos-chrome-68.0.3405.0_rc-r2/work/telemetry_src
chromeos-chrome-68.0.3405.0_rc-r2:  * install_test_resources to /build/terra/tmp/portage/chromeos-base/chromeos-chrome-68.0.3405.0_rc-r2/work/telemetry_src
chromeos-chrome-68.0.3405.0_rc-r2:  * Installing telemetry binaries: bitmaptools clear_system_cache minidump_stackwalk
chromeos-chrome-68.0.3405.0_rc-r2:  * Preparing tests
chromeos-chrome-68.0.3405.0_rc-r2: cp: cannot create hard link '/build/terra/tmp/portage/chromeos-base/chromeos-chrome-68.0.3405.0_rc-r2/work/autotest-work/client/deps/chrome_test/README' to 'chrome_test/README': Invalid cross-device link
chromeos-chrome-68.0.3405.0_rc-r2: cp: cannot create hard link '/build/terra/tmp/portage/chromeos-base/chromeos-chrome-68.0.3405.0_rc-r2/work/autotest-work/client/deps/chrome_test/chrome_test.py' to 'chrome_test/chrome_test.py': Invalid cross-device link
chromeos-chrome-68.0.3405.0_rc-r2: cp: cannot create hard link '/build/terra/tmp/portage/chromeos-base/chromeos-chrome-68.0.3405.0_rc-r2/work/autotest-work/client/deps/chrome_test/common.py' to 'chrome_test/common.py': Invalid cross-device link
chromeos-chrome-68.0.3405.0_rc-r2: cp: cannot create hard link '/build/terra/tmp/portage/chromeos-base/chromeos-chrome-68.0.3405.0_rc-r2/work/autotest-work/client/deps/chrome_test/control' to 'chrome_test/control': Invalid cross-device link
....
 
Cc: tbarzic@chromium.org
Owner: rahulchaudhry@chromium.org
Also part of the failure:


chromeos-chrome-68.0.3405.0_rc-r2:  * Number of unique text symbols in chrome got suspiciously large.
chromeos-chrome-68.0.3405.0_rc-r2:  * ERROR: chromeos-base/chromeos-chrome-68.0.3405.0_rc-r2::chromiumos failed (install phase):
chromeos-chrome-68.0.3405.0_rc-r2:  *   Chrome not built with ICF (Identical Code Folding) enabled.
chromeos-chrome-68.0.3405.0_rc-r2:  * 
chromeos-chrome-68.0.3405.0_rc-r2:  * Call stack:
chromeos-chrome-68.0.3405.0_rc-r2:  *     ebuild.sh, line 133:  Called src_install
chromeos-chrome-68.0.3405.0_rc-r2:  *   environment, line 21617:  Called die
chromeos-chrome-68.0.3405.0_rc-r2:  * The specific snippet of code:
chromeos-chrome-68.0.3405.0_rc-r2:  *           die "Chrome not built with ICF (Identical Code Folding) enabled.";


Seems like this was introduced in https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/1025133

Comment 2 by j...@chromium.org, Apr 24 2018

Seeing this error consistently since build #361:
https://uberchromegw.corp.google.com/i/chromeos/builders/terra-chrome-pfq

Error is "Chrome not built with ICF (Identical Code Folding) enabled."

Marking terra-chrome-pfq experimental.

Comment 3 by j...@chromium.org, Apr 24 2018

Cc: j...@chromium.org
I'm reverting the cl that added this error, until we figure out the underlying issue.
Project Member

Comment 5 by bugdroid1@chromium.org, Apr 25 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/3f43effcfdb77fd62f098b2546a1ff6af50f43c6

commit 3f43effcfdb77fd62f098b2546a1ff6af50f43c6
Author: Toni Barzic <tbarzic@chromium.org>
Date: Wed Apr 25 08:18:55 2018

Revert "chromeos-base/chromeos-chrome: check that chrome was built with ICF enabled."

This reverts commit b694ee00b3c0aff0130dfc768bb096423deb3269.

Reason for revert:
Breaking terra-chrome-pfq. See https://bugs.chromium.org/p/chromium/issues/detail?id=836296

Original change's description:
> chromeos-base/chromeos-chrome: check that chrome was built with ICF enabled.
> 
> For both amd64 and arm, when ICF (Identical Code Folding) is enabled,
> there is ~170,000 drop in the number of unique addresses of text symbols.
> We check here that the drop is at least 100,000.
> 
> BUG= chromium:813272 
> TEST='emerge-chell chromeos-base/chromeos-chrome' works.
> TEST='emerge-elm chromeos-base/chromeos-chrome' works.
> 
> Change-Id: I3213135b48525063fc922f98244113127b6b3b0b
> Reviewed-on: https://chromium-review.googlesource.com/1025133
> Commit-Ready: Rahul Chaudhry <rahulchaudhry@chromium.org>
> Tested-by: Rahul Chaudhry <rahulchaudhry@chromium.org>
> Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
> Reviewed-by: Ting-Yuan Huang <laszio@chromium.org>

Bug:  chromium:813272 ,  chromium:836296 
Change-Id: Ied7022aafa20753ce9d97f310db82bf95ade84f8
Reviewed-on: https://chromium-review.googlesource.com/1026590
Reviewed-by: Toni Barzic <tbarzic@chromium.org>
Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
Reviewed-by: Rahul Chaudhry <rahulchaudhry@chromium.org>
Tested-by: Toni Barzic <tbarzic@chromium.org>

[modify] https://crrev.com/3f43effcfdb77fd62f098b2546a1ff6af50f43c6/chromeos-base/chromeos-chrome/chromeos-chrome-9999.ebuild

Components: Tools>ChromeOS-Toolchain
Rahul, I think the issue with terra is use of thinlto which might have reduced the ICFed symbols.

Comment 7 by laszio@chromium.org, Apr 25 2018

Blockedon: 836344
Cc: vapier@chromium.org
It was reverted in chromeos-chrome-9999.ebuild, but not the versioned ebuild yet. Chrome-PFQs are blocked by other issues.

(Chrome-PFQ failures)
https://bugs.chromium.org/p/chromium/issues/detail?id=836344

This build was passing with THINLTO enabled.

I marked it as important, because we needed to have it's prebuilts published to fix problems with the terra-paladin.  https://crbug.com/824482 

Comment 9 by gu...@chromium.org, Apr 25 2018

We have lots of terra-paladin build failures since last night. Not fully sure if related to this.
They seem to be failing with "ERROR: Cannot find prebuilts for chromeos-base/chromeos-chrome on terra", which is due to terra-chrome-pfq failing because of this issue.

The CL adding the test and error message was reverted last night: https://chromium-review.googlesource.com/1026590
But terra-chrome-pfq is now failing due to another issue: https://bugs.chromium.org/p/chromium/issues/detail?id=836344
Once terra-chrome-pfq gets to a healthy state, terra-paladin would too.

Comment 11 by ihf@chromium.org, Apr 25 2018

Cc: ihf@chromium.org
Chrome will need to be pinned back to the last good revision.
Not sure if I'm doing it the right way, but would this work: https://chromium-review.googlesource.com/1028624

Project Member

Comment 13 by bugdroid1@chromium.org, Apr 25 2018

Project Member

Comment 14 by bugdroid1@chromium.org, Apr 25 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/8d328ef1595b93c57da20b373a5bf64c737df43d

commit 8d328ef1595b93c57da20b373a5bf64c737df43d
Author: Toni Barzic <tbarzic@chromium.org>
Date: Wed Apr 25 22:15:33 2018

Chrome: Pin to version 68.0.3405.0_rc-r1

DO NOT REVERT THIS CL.
In general, reverting chrome (un)pin CLs does not do what you expect.
Instead, use `cros pinchrome` to generate new CLs.

BUG= 836296 
TEST=None
CQ-DEPEND=*I54494127e9af1ee291aba6401da4f64acf5cebc8

Change-Id: Ia9a192d7c75cbd14767f555c93ac65a6a8ccbee3
Reviewed-on: https://chromium-review.googlesource.com/1028910
Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
Reviewed-by: Toni Barzic <tbarzic@chromium.org>
Commit-Queue: Toni Barzic <tbarzic@chromium.org>
Tested-by: Toni Barzic <tbarzic@chromium.org>

[rename] https://crrev.com/8d328ef1595b93c57da20b373a5bf64c737df43d/chromeos-base/chromeos-chrome/chromeos-chrome-68.0.3405.0_rc-r1.ebuild
[add] https://crrev.com/8d328ef1595b93c57da20b373a5bf64c737df43d/profiles/default/linux/package.mask/chromepin
[modify] https://crrev.com/8d328ef1595b93c57da20b373a5bf64c737df43d/chromeos-base/chromeos-chrome/Manifest
[modify] https://crrev.com/8d328ef1595b93c57da20b373a5bf64c737df43d/chromeos/binhost/target/amd64-generic-LATEST_RELEASE_CHROME_BINHOST.conf
[modify] https://crrev.com/8d328ef1595b93c57da20b373a5bf64c737df43d/chromeos/binhost/target/daisy-LATEST_RELEASE_CHROME_BINHOST.conf
[modify] https://crrev.com/8d328ef1595b93c57da20b373a5bf64c737df43d/chromeos/binhost/target/arm-generic-LATEST_RELEASE_CHROME_BINHOST.conf
[modify] https://crrev.com/8d328ef1595b93c57da20b373a5bf64c737df43d/chromeos/binhost/target/veyron_jerry-LATEST_RELEASE_CHROME_BINHOST.conf
[modify] https://crrev.com/8d328ef1595b93c57da20b373a5bf64c737df43d/chromeos/binhost/host/amd64-LATEST_RELEASE_CHROME_BINHOST.conf

Comment 15 by j...@chromium.org, Apr 26 2018

terra-paladin has passed for 6 consecutive builds, unmarking it as experimental.  terra-chrome-pfq is likely only failing because of the pfq issues right now.
Project Member

Comment 16 by bugdroid1@chromium.org, Apr 27 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/ebd3be3f32fcef8ee840b0ddac48cacc24c53c31

commit ebd3be3f32fcef8ee840b0ddac48cacc24c53c31
Author: Toni Barzic <tbarzic@chromium.org>
Date: Fri Apr 27 17:33:33 2018

Chrome: Unpin chrome

DO NOT REVERT THIS CL.
In general, reverting chrome (un)pin CLs does not do what you expect.
Instead, use `cros pinchrome` to generate new CLs.

BUG= 836296 
TEST=None

Change-Id: I303161a1777c5a1e1b34a1e530e09f95d3e6db0e
Reviewed-on: https://chromium-review.googlesource.com/1033185
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
Tested-by: Ilja H. Friedel <ihf@chromium.org>

[delete] https://crrev.com/a108fbb8f1216eb28b6781fc586fe6a37e954ecb/profiles/default/linux/package.mask/chromepin

Status: Fixed (was: Assigned)

Sign in to add a comment