New issue
Advanced search Search tips

Issue 659874 link

Starred by 1 user

Issue metadata

Status: Archived
Owner:
Closed: Oct 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

kevin public overlay build fails in chromeos-factory build

Project Member Reported by briannorris@chromium.org, Oct 27 2016

Issue description

Use the public/external manifest and try:

./build_packages --board=kevin

This fails with:

>>> Install chromeos-factory-0.0.1-r3671 into /build/kevin/tmp/portage/chromeos-base/chromeos-factory-0.0.1-r3671/image/ category chromeos-base
make -j48 bundle 
make -C py/goofy/static OUTPUT_DIR=/build/kevin/tmp/portage/chromeos-base/chromeos-factory-0.0.1-r3671/work/build/closure
make[1]: Entering directory '/mnt/host/source/src/platform/factory/py/goofy/static'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/mnt/host/source/src/platform/factory/py/goofy/static'
Invalid '-' operator in non-incremental variable 'TTY_CONSOLE': '-tty2'
Invalid '-' operator in non-incremental variable 'TTY_CONSOLE': '-tty2'
Invalid '-' operator in non-incremental variable 'TTY_CONSOLE': '-tty2'
Invalid '-' operator in non-incremental variable 'TTY_CONSOLE': '-tty2'
Invalid '-' operator in non-incremental variable 'TTY_CONSOLE': '-tty2'
Invalid '-' operator in non-incremental variable 'TTY_CONSOLE': '-tty2'
Invalid '-' operator in non-incremental variable 'TTY_CONSOLE': '-tty2'
Invalid '-' operator in non-incremental variable 'TTY_CONSOLE': '-tty2'
Invalid '-' operator in non-incremental variable 'TTY_CONSOLE': '-tty2'
Invalid '-' operator in non-incremental variable 'TTY_CONSOLE': '-tty2'
Invalid '-' operator in non-incremental variable 'TTY_CONSOLE': '-tty2'
Invalid '-' operator in non-incremental variable 'TTY_CONSOLE': '-tty2'
rm -rf /build/kevin/tmp/portage/chromeos-base/chromeos-factory-0.0.1-r3671/work/build/tmp/docsrc; mkdir -p /build/kevin/tmp/portage/chromeos-base/chromeos-factory-0.0.1-r3671/work/build/tmp/docsrc
# Do the actual build in the DOC_TEMP_DIR directory, since we need to
# munge the docs a bit.
rsync -a doc/ /build/kevin/tmp/portage/chromeos-base/chromeos-factory-0.0.1-r3671/work/build/tmp/docsrc
ERROR: Need to run 'emerge-kevin chromeos-factory-board' for rule ' [ /mnt/host/source/src/third_party/chromiumos-overlay/chromeos-base/chromeos-factory-board/chromeos-factory-board-0.0.1-r2.ebuild -ot /build/kevin/packages/chromeos-base/chromeos-factory-board-0.0.1-r2.tbz2 ] '.
make: *** [Makefile:183: check-board-resources] Error 1
make: *** Waiting for unfinished jobs....
# Generate rst sources for test cases
bin/generate_rsts -o /build/kevin/tmp/portage/chromeos-base/chromeos-factory-0.0.1-r3671/work/build/tmp/docsrc
Copying gs://chromeos-localmirror/distfiles/webgl-aquarium-20130524.tar.bz2...
WARNING:root:No test cases found in cros.factory.test.pytests.tablet_mode_uiiB    
WARNING:root:No test cases found in cros.factory.test.pytests.wifi_throughput_host
CROS_FACTORY_PY_ROOT=/mnt/host/source/src/platform/factory/py_pkg make -C /build/kevin/tmp/portage/chromeos-base/chromeos-factory-0.0.1-r3671/work/build/tmp/docsrc html
make[1]: Entering directory '/build/kevin/tmp/portage/chromeos-base/chromeos-factory-0.0.1-r3671/work/build/tmp/docsrc'
env DOC_TITLE_SUFFIX="" \
    sphinx-build -b html -d _build/doctrees   . _build/html
Making output directory...rk//./webgl-aquarium-20130524.tar.bz2: 7.1 MiB/12.06 MiB     
Running Sphinx v1.1.3
Downloading ....1-r3671/work//./webgl-aquarium-20130524.tar.bz2: 12.06 MiB/12.06 MiB    
loading pickled environment... not yet created
building [html]: targets for 156 source files that are out of date
updating environment: 156 added, 0 changed, 0 removed
reading sources... [100%] test_ui_api                                                                                                      
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] test_ui_api                                                                                                       
writing additional files... genindex py-modindex search
copying static files... done
dumping search index... done
dumping object inventory... done
build succeeded.
# Add copyright in postprocessing, so we can override this if necessary.
COPYRIGHT_HTML='<a rel="license" href="http://creativecommons.org/licenses/by/3.0/us/"><img alt="Creative Commons License" style="border-width:0" src="http://i.creativecommons.org/l/by/3.0/us/88x31.png" /> </a><br />This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by/3.0/us/">Creative Commons Attribution 3.0 United States License</a>.' perl -pi -e \
    's/<!--COPYRIGHT_HTML-->/$ENV{COPYRIGHT_HTML}/g' \
    _build/html/*.html

Build finished. The HTML pages are in _build/html.
make[1]: Leaving directory '/build/kevin/tmp/portage/chromeos-base/chromeos-factory-0.0.1-r3671/work/build/tmp/docsrc'
mkdir -p /build/kevin/tmp/portage/chromeos-base/chromeos-factory-0.0.1-r3671/work/build/
rm -rf /build/kevin/tmp/portage/chromeos-base/chromeos-factory-0.0.1-r3671/work/build/doc
cp -r /build/kevin/tmp/portage/chromeos-base/chromeos-factory-0.0.1-r3671/work/build/tmp/docsrc/_build/html /build/kevin/tmp/portage/chromeos-base/chromeos-factory-0.0.1-r3671/work/build/doc
(cd /build/kevin/tmp/portage/chromeos-base/chromeos-factory-0.0.1-r3671/work/build/doc/..; zip -qr9 - doc) \
  >/build/kevin/tmp/portage/chromeos-base/chromeos-factory-0.0.1-r3671/work/build/doc.zip
 * ERROR: chromeos-base/chromeos-factory-0.0.1-r3671::chromiumos failed (install phase):
 *   emake failed
 * 
 * If you need support, post the output of `emerge --info '=chromeos-base/chromeos-factory-0.0.1-r3671::chromiumos'`,
 * the complete build log and the output of `emerge -pqv '=chromeos-base/chromeos-factory-0.0.1-r3671::chromiumos'`.
 * The complete build log is located at '/build/kevin/tmp/portage/logs/chromeos-base:chromeos-factory-0.0.1-r3671:20161027-023129.log'.
 * For convenience, a symlink to the build log is located at '/build/kevin/tmp/portage/chromeos-base/chromeos-factory-0.0.1-r3671/temp/build.log'.
 * The ebuild environment file is located at '/build/kevin/tmp/portage/chromeos-base/chromeos-factory-0.0.1-r3671/temp/environment'.
 * Working directory: '/mnt/host/source/src/platform/factory'
 * S: '/mnt/host/source/src/platform/factory'

>>> Failed to emerge chromeos-base/chromeos-factory-0.0.1-r3671 for /build/kevin/, Log file:

>>>  '/build/kevin/tmp/portage/logs/chromeos-base:chromeos-factory-0.0.1-r3671:20161027-023129.log'

 * Messages for package chromeos-base/chromeos-factory-0.0.1-r3671 merged to /build/kevin/:

 * ERROR: chromeos-base/chromeos-factory-0.0.1-r3671::chromiumos failed (install phase):
 *   emake failed
 * 
 * If you need support, post the output of `emerge --info '=chromeos-base/chromeos-factory-0.0.1-r3671::chromiumos'`,
 * the complete build log and the output of `emerge -pqv '=chromeos-base/chromeos-factory-0.0.1-r3671::chromiumos'`.
 * The complete build log is located at '/build/kevin/tmp/portage/logs/chromeos-base:chromeos-factory-0.0.1-r3671:20161027-023129.log'.
 * For convenience, a symlink to the build log is located at '/build/kevin/tmp/portage/chromeos-base/chromeos-factory-0.0.1-r3671/temp/build.log'.
 * The ebuild environment file is located at '/build/kevin/tmp/portage/chromeos-base/chromeos-factory-0.0.1-r3671/temp/environment'.
 * Working directory: '/mnt/host/source/src/platform/factory'
 * S: '/mnt/host/source/src/platform/factory'
 

Comment 2 by hungte@chromium.org, Oct 27 2016

Is this on buildbot or your local environment?

Can you try to merge explicitly "emerge-kevin chromeos-factory-board" and see if that helps?

The chromeos-factory has a direct dependency to chromeos-factory-board so I thought chromeos-factory-board should be always rebuilt before chromeos-factory is built. Seems like for some reason, your chromeos-factory-board was either not built or not updated  properly. Maybe due to old environment?
It's a fresh sync; I just checked out everything new today. Local environment.

Comment 4 by hungte@chromium.org, Oct 27 2016

ok, let me do a fresh sync and see if I can reproduce. thanks!

Comment 5 by hungte@chromium.org, Oct 27 2016

hmmm, that's weird. your log shows that it has only found chromeos-factory-board from public repo. Was that log captured in public repo?

Can you provide the log of failure using private repo?

Comment 6 by hungte@chromium.org, Oct 27 2016

P.S: Can you do this in chroot:

ls -l /mnt/host/source/src/third_party/chromiumos-overlay/chromeos-base/chromeos-factory-board/chromeos-factory-board-0.0.1-r2.ebuild
ls -l /build/kevin/packages/chromeos-base/chromeos-factory-board-0.0.1-r2.tbz2
The bug $subject is about the public overlay.
i.e., this is intentionally about the public overlay. We have people building off of it.
$ ls -l /mnt/host/source/src/third_party/chromiumos-overlay/chromeos-base/chromeos-factory-board/chromeos-factory-board-0.0.1-r2.ebuild
lrwxrwxrwx 1 briannorris eng 35 Oct 26 18:18 /mnt/host/source/src/third_party/chromiumos-overlay/chromeos-base/chromeos-factory-board/chromeos-factory-board-0.0.1-r2.ebuild -> chromeos-factory-board-0.0.1.ebuild

$ ls -l /build/kevin/packages/chromeos-base/chromeos-factory-board-0.0.1-r2.tbz2
-rw-r--r-- 1 root root 7366 Oct 19 03:58 /build/kevin/packages/chromeos-base/chromeos-factory-board-0.0.1-r2.tbz2
Yes, I understand we do want to make sure public manifest is working properly. Having the log of builds from private manifest may help to analyse what goes wrong, but I will first focus on public manifest.

Can you provide the information in #6? I wonder if it's something related to system clock error or prebuilt system timestamp.
Re#9: Thanks for your logs. Seems like caused by symlink dates...

Do you mind to give me one more info?

ls -l /mnt/host/source/src/third_party/chromiumos-overlay/chromeos-base/chromeos-factory-board/chromeos-factory-board-0.0.1.ebuild
Owner: hungte@chromium.org
Status: Started (was: Untriaged)
I think I've figured out why. Fix on the way. Thanks for the report and all the provided logs!
For people who gets blocked on this, please apply https://chromium-review.googlesource.com/#/c/403670/
It doesn't impact incremental build, right?
I didn't see the same failure on my public checkout. (I repo sync 2 hours ago.)
@11:

$ ls -l /mnt/host/source/src/third_party/chromiumos-overlay/chromeos-base/chromeos-factory-board/chromeos-factory-board-0.0.1.ebuild
-rw-r----- 1 briannorris eng 389 Oct 26 18:18 /mnt/host/source/src/third_party/chromiumos-overlay/chromeos-base/chromeos-factory-board/chromeos-factory-board-0.0.1.ebuild

I'll review your CL tomorrow, if no one else gets around to it. Thanks for the work!
Project Member

Comment 16 by bugdroid1@chromium.org, Oct 27 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/factory/+/1c0d6cfb46517658e2f26ef6126373a0964f2383

commit 1c0d6cfb46517658e2f26ef6126373a0964f2383
Author: Hung-Te Lin <hungte@chromium.org>
Date: Thu Oct 27 03:21:32 2016

Makefile: Fix build failure for fresh checkout.

In fresh checkout, ebuild files may have a newer timestamp than
downloaded pre-built packages. This can be a problem when you run
build_packages, which only checks package version instead of timestamps.

The solution is to prevent timestamp-based comparison when running
build_packages.

Developers calling 'make' or 'emerge-$BOARD' will still perform the
right checks.

BUG= chromium:659874 
TEST=touch chromeos-factory-board-*.ebuild
     emerge-$BOARD chromeos-factory
     ./build_packages --board $BOARD

Change-Id: Iaa0d47e92c9005f7f21dd85e13ce2be7b0937907
Reviewed-on: https://chromium-review.googlesource.com/403670
Commit-Ready: Hung-Te Lin <hungte@chromium.org>
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Shun-Hsing Ou <shunhsingou@chromium.org>

[modify] https://crrev.com/1c0d6cfb46517658e2f26ef6126373a0964f2383/Makefile

Status: Fixed (was: Started)
The build failure should be fixed.

Regarding the fix itself, we may improve by preventing the special trick of detecting build_packages - that can be discussed and solved later.
Project Member

Comment 19 by bugdroid1@chromium.org, Nov 5 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/factory/+/5ab142153f2d25f209a8eabc2d5380b6334e4b77

commit 5ab142153f2d25f209a8eabc2d5380b6334e4b77
Author: Hung-Te Lin <hungte@chromium.org>
Date: Thu Nov 03 16:39:57 2016

Makefile: Prevent checking build_packages

It is not recommended to detect if we're building under build_packages since it
was supposed to be a simple wrapper for emerge commands.

This change tries to solve the problem differently: consider "if the build
system instructs to not pause = not an interactive bug-fix cycle, then don't
check board resource timestamps".

Developers calling 'make' manually will still perform the right checks.

BUG= chromium:659874 
TEST=touch chromeos-factory-board-*.ebuild
     emerge-$BOARD chromeos-factory
     ./build_packages --board $BOARD

Change-Id: Id81d5ea90fef4b7f97cf17db47338be15b61a536
Reviewed-on: https://chromium-review.googlesource.com/407129
Commit-Ready: Hung-Te Lin <hungte@chromium.org>
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Shun-Hsing Ou <shunhsingou@chromium.org>

[modify] https://crrev.com/5ab142153f2d25f209a8eabc2d5380b6334e4b77/Makefile

Project Member

Comment 20 by bugdroid1@chromium.org, Nov 8 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/0ec4e76b7b4f540428475ed6d245379799286a59

commit 0ec4e76b7b4f540428475ed6d245379799286a59
Author: Mike Frysinger <vapier@chromium.org>
Date: Thu Oct 27 20:26:13 2016

chromeos_version.sh: do not export version vars

We don't need to export these to the env because the only consumers
are build scripts which source this directly.

BUG= chromium:659874 
TEST=precq still passes and has right version info in image

Change-Id: I849655f1e4d3a546d339c122914d2dfdaece6ee8
Reviewed-on: https://chromium-review.googlesource.com/406708
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/0ec4e76b7b4f540428475ed6d245379799286a59/chromeos/config/chromeos_version.sh

These changes landed before the M56 branch AFAICT, but I'm hearing reports of build failure on chromeos-factory again. Did something here not make the M56 branch? Or are there other issues still?

I'll try to collect more info from the reporter, but I wanted to make sure my branch reading was correct.
there should be no problem for M56.

Comment 23 by dchan@google.com, Jan 21 2017

Labels: VerifyIn-57

Comment 24 by dchan@google.com, Mar 4 2017

Labels: VerifyIn-58

Comment 25 by dchan@google.com, Apr 17 2017

Labels: VerifyIn-59

Comment 26 by dchan@google.com, May 30 2017

Labels: VerifyIn-60
Labels: VerifyIn-61

Comment 28 by dchan@chromium.org, Oct 14 2017

Status: Archived (was: Fixed)

Sign in to add a comment