Issue metadata
Sign in to add a comment
|
portage no longer respects order of PORTAGE_BINHOST |
||||||||||||||||||||||
Issue descriptionAfter building an eve image with the standard ./build_packages && ./build_image flow, the resulting image shows a dialog at the top of the login screen that says that "Google API keys are missing". I cannot log in or add any existing users. The only difference in my environment is that I am setting USE=kvm_host, which should affect some OS packages but should not affect chrome in any way. I am now going to try a test image from a builder to see if that one works.
,
Feb 6 2018
Well, that didn't take long:
$ cros flash $BIP xbuddy://remote/eve/latest/test
17:48:44: NOTICE: Preparing to update the remote device 100.127.1.63
17:48:49: NOTICE: Using image eve-release/R64-10176.68.0/chromiumos_test_image.bin
17:48:56: ERROR: Devserver responded with HTTP error (HTTP Error 500: Internal Server Error)
17:48:56: WARNING: --- Start output from /tmp/devserver_wrapperUDA6oV/dev_server.log ---[05/Feb/2018:17:48:54] XBUDDY Linking to /mnt/host/source/devserver/static/cave/R62-9783.0.2017_07_27_1034-a1 from /mnt/host/source/src/build/images/cave/R62-9783.0.2017_07_27_1034-a1
[05/Feb/2018:17:48:54] XBUDDY Linking to /mnt/host/source/devserver/static/cave/R59-9364.0.2017_03_15_1522-a1 from /mnt/host/source/src/build/images/cave/R59-9364.0.2017_03_15_1522-a1
[05/Feb/2018:17:48:54] XBUDDY Linking to /mnt/host/source/devserver/static/tatl/R65-10273.0.2018_01_05_1928-a1 from /mnt/host/source/src/build/images/tatl/R65-10273.0.2018_01_05_1928-a1
[05/Feb/2018:17:48:54] XBUDDY Linking to /mnt/host/source/devserver/static/tatl/R65-10317.0.2018_01_17_1836-a1 from /mnt/host/source/src/build/images/tatl/R65-10317.0.2018_01_17_1836-a1
[05/Feb/2018:17:48:54] XBUDDY Linking to /mnt/host/source/devserver/static/tatl/R64-10104.0.2017_11_06_1622-a1 from /mnt/host/source/src/build/images/tatl/R64-10104.0.2017_11_06_1622-a1
[05/Feb/2018:17:48:54] XBUDDY Linking to /mnt/host/source/devserver/static/tatl/R64-10148.0.2017_11_20_1711-a1 from /mnt/host/source/src/build/images/tatl/R64-10148.0.2017_11_20_1711-a1
[05/Feb/2018:17:48:54] XBUDDY Linking to /mnt/host/source/devserver/static/tatl/R64-10148.0.2017_11_20_1742-a1 from /mnt/host/source/src/build/images/tatl/R64-10148.0.2017_11_20_1742-a1
[05/Feb/2018:17:48:54] XBUDDY Linking to /mnt/host/source/devserver/static/tatl/R65-10230.0.2017_12_19_1551-a1 from /mnt/host/source/src/build/images/tatl/R65-10230.0.2017_12_19_1551-a1
[05/Feb/2018:17:48:54] XBUDDY Linking to /mnt/host/source/devserver/static/tatl/R64-10153.0.2017_11_22_1521-a1 from /mnt/host/source/src/build/images/tatl/R64-10153.0.2017_11_22_1521-a1
[05/Feb/2018:17:48:54] XBUDDY Linking to /mnt/host/source/devserver/static/eve-kvm/R66-10368.0.2018_02_05_1529-a1 from /mnt/host/source/src/build/images/eve-kvm/R66-10368.0.2018_02_05_1529-a1
[05/Feb/2018:17:48:54] XBUDDY Linking to /mnt/host/source/devserver/static/eve-kvm/R63-10003.0.2017_10_04_2306-a1 from /mnt/host/source/src/build/images/eve-kvm/R63-10003.0.2017_10_04_2306-a1
[05/Feb/2018:17:48:54] XBUDDY Linking to /mnt/host/source/devserver/static/eve-kvm/R63-10003.0.2017_10_04_2319-a1 from /mnt/host/source/src/build/images/eve-kvm/R63-10003.0.2017_10_04_2319-a1
[05/Feb/2018:17:48:54] XBUDDY Linking to /mnt/host/source/devserver/static/eve-kvm/R64-10148.0.2017_11_20_1533-a1 from /mnt/host/source/src/build/images/eve-kvm/R64-10148.0.2017_11_20_1533-a1
[05/Feb/2018:17:48:54] XBUDDY Linking to /mnt/host/source/devserver/static/eve-kvm/R65-10317.0.2018_01_17_1806-a1 from /mnt/host/source/src/build/images/eve-kvm/R65-10317.0.2018_01_17_1806-a1
[05/Feb/2018:17:48:54] XBUDDY Linking to /mnt/host/source/devserver/static/eve-kvm/R65-10230.0.2017_12_19_1526-a1 from /mnt/host/source/src/build/images/eve-kvm/R65-10230.0.2017_12_19_1526-a1
[05/Feb/2018:17:48:54] XBUDDY Linking to /mnt/host/source/devserver/static/eve-kvm/R66-10360.0.2018_02_01_1208-a1 from /mnt/host/source/src/build/images/eve-kvm/R66-10360.0.2018_02_01_1208-a1
[05/Feb/2018:17:48:54] XBUDDY Linking to /mnt/host/source/devserver/static/eve-kvm/R64-10104.0.2017_11_06_1304-a1 from /mnt/host/source/src/build/images/eve-kvm/R64-10104.0.2017_11_06_1304-a1
[05/Feb/2018:17:48:54] XBUDDY Linking to /mnt/host/source/devserver/static/eve-kvm/R66-10335.0.2018_01_24_1559-a1 from /mnt/host/source/src/build/images/eve-kvm/R66-10335.0.2018_01_24_1559-a1
[05/Feb/2018:17:48:54] XBUDDY Linking to /mnt/host/source/devserver/static/eve-kvm/R66-10368.0.2018_02_02_1841-a1 from /mnt/host/source/src/build/images/eve-kvm/R66-10368.0.2018_02_02_1841-a1
[05/Feb/2018:17:48:54] XBUDDY Path is remote/eve-release/R64-10176.68.0/test, location suffix is -release
[05/Feb/2018:17:48:54] XBUDDY Get artifact 'test' with board eve-release and version R64-10176.68.0'. Locally? False
[05/Feb/2018:17:48:56] HTTP Traceback (most recent call last):
File "/usr/lib64/python2.7/site-packages/cherrypy/_cprequest.py", line 656, in respond
response.body = self.handler()
File "/usr/lib64/python2.7/site-packages/cherrypy/lib/encoding.py", line 188, in __call__
self.body = self.oldhandler(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__
return self.callable(*self.args, **self.kwargs)
File "/usr/lib/devserver/devserver.py", line 1503, in xbuddy
build_id = self._xbuddy.StageTestArtifactsForUpdate(args)
File "/usr/lib64/devserver/xbuddy.py", line 909, in StageTestArtifactsForUpdate
build_id, file_name = self.Translate(path_list)
File "/usr/lib64/devserver/xbuddy.py", line 896, in Translate
image_dir=image_dir)
File "/usr/lib64/devserver/xbuddy.py", line 841, in _GetArtifact
image_dir=image_dir)
File "/usr/lib64/devserver/xbuddy.py", line 517, in _ResolveVersionToBuildId
return self._RemoteBuildId(board, suffix, version)
File "/usr/lib64/devserver/xbuddy.py", line 476, in _RemoteBuildId
board, version))
XBuddyException: Could not find remote build_id for eve-release R64-10176.68.0
[05/Feb/2018:17:48:56] HTTP
Request Headers:
HOST: 172.22.113.45:44659
CONNECTION: close
Remote-Addr: ::ffff:172.22.113.45
USER-AGENT: Python-urllib/2.7
ACCEPT-ENCODING: identity
::ffff:172.22.113.45 - - [05/Feb/2018:17:48:56] "GET /xbuddy/remote/eve-release/R64-10176.68.0/test?for_update=true&return_dir=true HTTP/1.1" 500 1990 "" "Python-urllib/2.7"
--- End output from /tmp/devserver_wrapperUDA6oV/dev_server.log ---
17:48:56: ERROR: Device update failed.
17:48:56: ERROR: cros flash failed before completing.
17:48:56: ERROR: HTTP Error 500: Internal Server Error
,
Feb 6 2018
> can you show emerge-eve -pv output for it?
% emerge-eve -pv chromeos-chrome
These are the packages that would be merged, in order:
Calculating dependencies... done!
[ebuild R ] chromeos-base/chromeos-chrome-66.0.3336.3_rc-r1::chromiumos to /build/eve/ USE="accessibility authpolicy autotest build_tests buildcheck chrome_debug chrome_remoting clang cups debug_fission evdev_gestures fonts gold highdpi libcxx nacl opengles runhooks smbprovider v4l2_codec vaapi xkbcommon -afdo_chrome_exp1 -afdo_chrome_exp2 -afdo_use -app_shell -asan -chrome_debug_tests -chrome_internal -chrome_media -component_build -goma -hardfp -internal_gles_conform -lld -mojo (-neon) -opengl -thinlto -v4lplugin -verbose -vtable_verify" OZONE_PLATFORM="gbm -caca -cast -egltest {-test}" OZONE_PLATFORM_DEFAULT="gbm -caca -cast -egltest {-test}" 0 KiB
Total: 1 package (1 reinstall), Size of downloads: 0 KiB
,
Feb 6 2018
I tried out the latest test image from the canary channel and that seemed to work fine so something wonky is happening when I build an image on my machine. However, smbarber@ is also having this issue so at least we have a sample size of 2.
,
Feb 6 2018
Looks like you have a "-" before chrome_internal.
,
Feb 6 2018
looks like for eve builds, the binpkg is coming from amd64-generic chromium pfq
$ portageq-eve envvar PORTAGE_BINHOST
gs://chromeos-prebuilt/board/amd64-generic/paladin-R66-10377.0.0-rc2/packages/ gs://chromeos-prebuilt/board/eve/paladin-R66-10377.0.0-rc2/packages/ gs://chromeos-prebuilt/board/amd64-generic/chrome-R66-10378.0.0-rc2/packages/
$ emerge-eve -pvgkq chromeos-chrome --nodeps
[binary N ] chromeos-base/chromeos-chrome-66.0.3341.0_rc-r1 to /build/eve/ USE="accessibility authpolicy autotest build_tests buildcheck chrome_debug chrome_remoting clang cups debug_fission evdev_gestures fonts gold highdpi libcxx nacl opengles runhooks smbprovider v4l2_codec vaapi xkbcommon -afdo_chrome_exp1 -afdo_chrome_exp2 -afdo_use -app_shell -asan -chrome_debug_tests -chrome_internal -chrome_media -component_build -goma -hardfp -internal_gles_conform -lld -mojo (-neon) -opengl -thinlto -v4lplugin -verbose -vtable_verify" OZONE_PLATFORM="gbm -caca -cast -egltest {-test}" OZONE_PLATFORM_DEFAULT="gbm -caca -cast -egltest {-test}"
as a chromium pfq, we don't set USE=chrome_internal, but API keys are not dependent upon that flag. we are setting USE=internal as we're doing an internal CrOS manifest checkout, and that's how API keys get added in the ebuild:
local WHOAMI=$(whoami)
# Get the credentials to fake home directory so that the version of chromium
# we build can access Google services. First, check for Chrome credentials.
if [[ ! -d google_apis/internal ]]; then
# Then look for Chrome OS supplied credentials.
local PRIVATE_OVERLAYS_DIR=/home/${WHOAMI}/trunk/src/private-overlays
local GAPI_CONFIG_FILE=${PRIVATE_OVERLAYS_DIR}/chromeos-overlay/googleapikeys
if [[ ! -f "${GAPI_CONFIG_FILE}" ]]; then
# Then developer credentials.
GAPI_CONFIG_FILE=/home/${WHOAMI}/.googleapikeys
fi
if [[ -f "${GAPI_CONFIG_FILE}" ]]; then
add_api_keys "${GAPI_CONFIG_FILE}"
fi
fi
looking at src/private-overlays/chromeos-overlay/googleapikeys, that file still exists.
downloading the binpkg for the ver you quoted from the amd64-generic chromium pfq indicates that the API keys are correctly baked into it.
$ gsutil cp gs://chromeos-prebuilt/board/amd64-generic/chrome-R66-10365.0.0-rc1/packages/chromeos-base/chromeos-chrome-66.0.3336.3_rc-r1.tbz2 ./
$ tar xf *.tbz2 opt/google/chrome/chrome
$ strings opt/google/chrome/chrome | grep -i ^AIza
<key that matches googleapikeys file>
can you check to see if your eve's chrome binary has the keys baked in it ? even if you're building from source locally, you should get the keys if you have an internal CrOS checkout.
,
Feb 6 2018
% strings /build/eve/opt/google/chrome/chrome | grep -i -e '^AIza' doesn't have any results for me. I have the googleapikeys file in my checkout so I'm going to try building from source.
,
Feb 7 2018
Ok so I built chromeos-chrome-9999 from source. This is what I see:
(cr) chirantan@belgium ~/trunk/src/scripts (git-svn)-[93c015d...] % strings /build/eve/opt/google/chrome/chrome | grep -i -e '^AIza'
<redacted>
So the key exists in my build from source. Next I tried to install the binary package again:
(cr) chirantan@belgium ~/trunk/src/scripts (git-svn)-[93c015d...] % emerge-eve --getbinpkg --usepkgonly -pv chromeos-chrome
Local copy of remote index is up-to-date and will be used.
Local copy of remote index is up-to-date and will be used.
Local copy of remote index is up-to-date and will be used.
Local copy of remote index is up-to-date and will be used.
Local copy of remote index is up-to-date and will be used.
Local copy of remote index is up-to-date and will be used.
Local copy of remote index is up-to-date and will be used.
These are the packages that would be merged, in order:
Calculating dependencies... done!
[binary UD ] chromeos-base/chromeos-chrome-66.0.3336.3_rc-r1::chromiumos [9999::chromiumos] to /build/eve/ USE="accessibility authpolicy autotest build_tests buildcheck chrome_debug chrome_remoting clang cups debug_fission evdev_gestures fonts gold highdpi libcxx nacl opengles runhooks smbprovider v4l2_codec vaapi
xkbcommon -afdo_chrome_exp1 -afdo_chrome_exp2 -afdo_use -app_shell -asan -chrome_debug_tests -chrome_internal -chrome_media -component_build -goma* -hardfp -internal_gles_conform -lld -mojo (-neon) -opengl -thinlto -v4lplugin -verbose -vtable_verify" OZONE_PLATFORM="gbm -caca -cast -egltest {-test}" OZONE_PLATFORM_DE
FAULT="gbm -caca -cast -egltest {-test}" 0 KiB
Total: 1 package (1 downgrade, 1 binary), Size of downloads: 0 KiB
So it's downgrading the -9999 package to the binary package. Ok so far. After I installed the binary package:
(cr) chirantan@belgium ~/trunk/src/scripts (git-svn)-[93c015d...] % strings /build/eve/opt/google/chrome/chrome | grep -i -e '^AIza'
This returns nothing. So the binary package that I currently have does not have the api keys baked into chrome.
Here's the output of the commands you ran:
(cr) chirantan@belgium ~/trunk/src/scripts (git-svn)-[93c015d...] % portageq-eve envvar PORTAGE_BINHOST
gs://chromeos-prebuilt/board/amd64-generic/paladin-R66-10367.0.0-rc2/packages/ gs://chromeos-prebuilt/board/eve/paladin-R66-10367.0.0-rc2/packages/ gs://chromeos-prebuilt/board/amd64-generic/chrome-R66-10365.0.0-rc1/packages/
Interestingly there is no chromeos-chrome package for the first or second google storage URL:
(cr) chirantan@belgium ~/trunk/src/scripts (git-svn)-[93c015d...] % gsutil ls gs://chromeos-prebuilt/board/amd64-generic/paladin-R66-10367.0.0-rc2/packages/chromeos-base | grep chromeos-chrome
(cr) chirantan@belgium ~/trunk/src/scripts (git-svn)-[93c015d...] % gsutil ls gs://chromeos-prebuilt/board/eve/paladin-R66-10367.0.0-rc2/packages/ | grep chromeos-chrome
Neither command returns anything.
I downloaded the package from google storage using the third URL:
(cr) chirantan@belgium ~/trunk/src/scripts (git-svn)-[93c015d...] % gsutil cp gs://chromeos-prebuilt/board/amd64-generic/chrome-R66-10365.0.0-rc1/packages/chromeos-base/chromeos-chrome-66.0.3336.3_rc-r1.tbz2 ./
(cr) chirantan@belgium ~/trunk/src/scripts (git-svn)-[93c015d...] % tar xvf *.tbz2 ./opt/google/chrome/chrome
(cr) chirantan@belgium ~/trunk/src/scripts (git-svn)-[93c015d...] % strings ./opt/google/chrome/chrome | grep -i -e '^AIza'
<redacted>
Interesting... So it seems the binary package on google storage also has the keys. What's up with the packages?
(cr) chirantan@belgium ~/trunk/src/scripts (git-svn)-[93c015d...] % diff chromeos-chrome-66.0.3336.3_rc-r1.tbz2 /build/eve/packages/chromeos-base/chromeos-chrome-66.0.3336.3_rc-r1.tbz2
Binary files chromeos-chrome-66.0.3336.3_rc-r1.tbz2 and /build/eve/packages/chromeos-base/chromeos-chrome-66.0.3336.3_rc-r1.tbz2 differ
Despite being binary packages with the exact same version, these are different. How did I end up with a different package than what's on google storage?
,
Feb 7 2018
Ok, progress. I removed the binary package from /build/eve/packages and tried again:
(cr) chirantan@belgium ~/trunk/src/scripts (git-svn)-[93c015d...] % sudo mv /build/eve/packages/chromeos-base/chromeos-chrome-66.0.3336.3_rc-r1.tbz2 ~/
(cr) chirantan@belgium ~/trunk/src/scripts (git-svn)-[93c015d...] % emerge-eve --getbinpkg --usepkgonly -v chromeos-chrome
Local copy of remote index is up-to-date and will be used.
Local copy of remote index is up-to-date and will be used.
Local copy of remote index is up-to-date and will be used.
Local copy of remote index is up-to-date and will be used.
Local copy of remote index is up-to-date and will be used.
Local copy of remote index is up-to-date and will be used.
Local copy of remote index is up-to-date and will be used.
These are the packages that would be merged, in order:
Calculating dependencies... done!
[binary R ] chromeos-base/chromeos-chrome-66.0.3336.3_rc-r1::chromiumos to /build/eve/ USE="accessibility authpolicy autotest build_tests buildcheck chrome_debug chrome_remoting clang cups debug_fission evdev_gestures fonts gold highdpi libcxx nacl opengles runhooks smbprovider v4l2_codec vaapi xkbcommon -afdo_ch
rome_exp1 -afdo_chrome_exp2 -afdo_use -app_shell -asan -chrome_debug_tests -chrome_internal -chrome_media -component_build -goma -hardfp -internal_gles_conform -lld -mojo (-neon) -opengl -thinlto -v4lplugin -verbose -vtable_verify" OZONE_PLATFORM="gbm -caca -cast -egltest {-test}" OZONE_PLATFORM_DEFAULT="gbm -caca -ca
st -egltest {-test}" 406,924 KiB
Total: 1 package (1 reinstall, 1 binary), Size of downloads: 406,924 KiB
>>> Emerging binary (1 of 1) chromeos-base/chromeos-chrome-66.0.3336.3_rc-r1::chromiumos for /build/eve/
16:23:21: INFO: RunCommand: /mnt/host/source/.cache/common/gsutil_4.27.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cp -v -- gs://chromeos-prebuilt/board/amd64-generic/full-2018.02.02.070328/packages/chromeos-base/chromeos-chrome-66.0.3336.3_rc-r1.tbz2 /build/eve/packages/chromeos-base/chromeos-chrome-66.0.3336.3_rc-
r1.tbz2.partial.tmp
So the URL that portage is using to download the binpkg is gs://chromeos-prebuilt/board/amd64-generic/full-2018.02.02.070328 instead of gs://chromeos-prebuilt/board/amd64-generic/chrome-R66-10365.0.0-rc1. And the chrome binary from that package does not have the keys:
(cr) chirantan@belgium ~/trunk/src/scripts (git-svn)-[93c015d...] % gsutil cp gs://chromeos-prebuilt/board/amd64-generic/full-2018.02.02.070328/packages/chromeos-base/chromeos-chrome-66.0.3336.3_rc-r1.tbz2 ./
(cr) chirantan@belgium ~/trunk/src/scripts (git-svn)-[93c015d...] % tar xvf *.tbz2 ./opt/google/chrome/chrome
(cr) chirantan@belgium ~/trunk/src/scripts (git-svn)-[93c015d...] % strings ./opt/google/chrome/chrome | grep -i -e '^AIza'
<returns nothing>
So the next questions are:
- why is portage downloading from the amd64-generic-full builder?
- why does that builder not have access to the api keys?
- how do we catch different binary packages with the same version being uploaded to different places?
,
Feb 7 2018
Potential answer to the first question:
(cr) chirantan@belgium ~/trunk/src/scripts (git-svn)-[93c015d...] % grep chromeos-chrome /build/eve/var/cache/edb/binhost/chromeos-prebuilt/board/amd64-generic/paladin-R66-10367.0.0-rc2/packages/Packages | grep PATH
PATH: board/amd64-generic/full-2018.02.02.070328/packages/chromeos-base/chromeos-chrome-66.0.3336.3_rc-r1.tbz2
Ok, so the portage manpage says we can safely delete the /var/cache/edb directory. I tried that next:
(cr) chirantan@belgium ~/trunk/src/scripts (git-svn)-[93c015d...] % sudo mv /build/eve/var/cache/edb /build/eve/var/cache/edb2
(cr) chirantan@belgium ~/trunk/src/scripts (git-svn)-[93c015d...] % emerge-eve --getbinpkg --usepkgonly -v chromeos-chrome
16:49:39: INFO: RunCommand: /mnt/host/source/.cache/common/gsutil_4.27.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cp -v -- gs://chromeos-prebuilt/board/amd64-generic/paladin-R66-10367.0.0-rc2/packages/Packages /tmp/tmpXrgTg8.tmp
16:49:41: INFO: RunCommand: /mnt/host/source/.cache/common/gsutil_4.27.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cp -v -- gs://chromeos-prebuilt/board/eve/paladin-R66-10367.0.0-rc2/packages/Packages /tmp/tmpYGt3pG.tmp
16:49:43: INFO: RunCommand: /mnt/host/source/.cache/common/gsutil_4.27.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cp -v -- gs://chromeos-prebuilt/board/amd64-generic/chrome-R66-10365.0.0-rc1/packages/Packages /tmp/tmp1Q8vyA.tmp
Local copy of remote index is up-to-date and will be used.
Local copy of remote index is up-to-date and will be used.
Local copy of remote index is up-to-date and will be used.
Local copy of remote index is up-to-date and will be used.
These are the packages that would be merged, in order:
Calculating dependencies... done!
[binary R ] chromeos-base/chromeos-chrome-66.0.3336.3_rc-r1::chromiumos to /build/eve/ USE="accessibility authpolicy autotest build_tests buildcheck chrome_debug chrome_remoting clang cups debug_fission evdev_gestures fonts gold highdpi libcxx nacl opengles runhooks smbprovider v4l2_codec vaapi xkbcommon -afdo_ch
rome_exp1 -afdo_chrome_exp2 -afdo_use -app_shell -asan -chrome_debug_tests -chrome_internal -chrome_media -component_build -goma -hardfp -internal_gles_conform -lld -mojo (-neon) -opengl -thinlto -v4lplugin -verbose -vtable_verify" OZONE_PLATFORM="gbm -caca -cast -egltest {-test}" OZONE_PLATFORM_DEFAULT="gbm -caca -ca
st -egltest {-test}" 406,924 KiB
Total: 1 package (1 reinstall, 1 binary), Size of downloads: 406,924 KiB
>>> Emerging binary (1 of 1) chromeos-base/chromeos-chrome-66.0.3336.3_rc-r1::chromiumos for /build/eve/
16:49:53: INFO: RunCommand: /mnt/host/source/.cache/common/gsutil_4.27.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cp -v -- gs://chromeos-prebuilt/board/amd64-generic/full-2018.02.02.070328/packages/chromeos-base/chromeos-chrome-66.0.3336.3_rc-r1.tbz2 /build/eve/packages/chromeos-base/chromeos-chrome-66.0.3336.3_rc-
r1.tbz2.partial.tmp
^C
Ok. So it looks like it re-downloaded the package/Packages file and then fell back to pulling from the amd64-generic-full prebuilts. Let's check out the Packages files:
(cr) chirantan@belgium ~/trunk/src/scripts (git-svn)-[93c015d...] % gsutil cp gs://chromeos-prebuilt/board/amd64-generic/chrome-R66-10365.0.0-rc1/packages/Packages ./
(cr) chirantan@belgium ~/trunk/src/scripts (git-svn)-[93c015d...] % grep PATH Packages
PATH: board/amd64-generic/chrome-R66-10365.0.0-rc1/packages/chromeos-base/chromeos-chrome-66.0.3336.3_rc-r1.tbz2
Looks fine. Let's try the next one:
(cr) chirantan@belgium ~/trunk/src/scripts (git-svn)-[93c015d...] % gsutil cp gs://chromeos-prebuilt/board/amd64-generic/paladin-R66-10367.0.0-rc2/packages/Packages ./
(cr) chirantan@belgium ~/trunk/src/scripts (git-svn)-[93c015d...] % grep PATH Packages | grep chromeos-chrome
PATH: board/amd64-generic/full-2018.02.02.070328/packages/chromeos-base/chromeos-chrome-66.0.3336.3_rc-r1.tbz2
Bingo. Next, where is PORTAGE_BINHOST being set?
(cr) chirantan@belgium ~/trunk/src/scripts (git-svn)-[93c015d...] % sudo grep -r PORTAGE_BINHOST /build/eve
/build/eve/etc/make.conf.board:PORTAGE_BINHOST="$FULL_BINHOST"
/build/eve/etc/make.conf.board:PORTAGE_BINHOST="$PORTAGE_BINHOST $PREFLIGHT_BINHOST"
/build/eve/etc/make.conf.board:PORTAGE_BINHOST="$PORTAGE_BINHOST $PREFLIGHT_BINHOST"
/build/eve/etc/make.conf.board:PORTAGE_BINHOST="$PORTAGE_BINHOST $LATEST_RELEASE_CHROME_BINHOST"
Looks promising. According to make.conf.board, BINHOSTs that appear earlier should have lower priority. However it looks like the opposite is happening and BINHOSTs that appear earlier are being prioritized over BINHOSTs that appear later.
Looks like this might be related to the portage upgrade. vapier@: Any thoughts? I tried going through the portage code but I couldn't find what changed.
The other question is why are the full builders building chrome without API keys? It's pretty useless since you can't add users. The only thing you can really do is use guest mode.
,
Feb 7 2018
I can confirm this works around the issue for me: $ ./build_packages --board eve $ sudo rm /build/eve/packages/chromeos-base/chromeos-chrome-66.0.3336.3_rc-r1.tbz2 $ gsutil cp gs://chromeos-prebuilt/board/amd64-generic/chrome-R66-10365.0.0-rc1/packages/chromeos-base/chromeos-chrome-66.0.3336.3_rc-r1.tbz2 ./ $ sudo chown root:root chromeos-chrome-66.0.3336.3_rc-r1.tbz2 $ sudo chmod 0644 chromeos-chrome-66.0.3336.3_rc-r1.tbz2 $ sudo mv chromeos-chrome-66.0.3336.3_rc-r1.tbz2 /build/eve/packages/chromeos-base/ $ ./build_image --board eve However, it would still be good to figure out why portage is pulling the prebuilt from the full builder instead of the chrome builder.
,
Feb 7 2018
So after a repo sync today portage is now downloading the binpkg from the chrome builder instead of the full builder.
,
Feb 8 2018
gs://chromeos-prebuilt/board/amd64-generic/chrome-xxx is not the "full" builder, it's the chromium-pfq builder. that should have API keys. you might have had a typo in comment #9 in that regard. a particular sub-path not having a binpkg isn't a bug by itself. if you look at the Packages file, we blend previous binpkgs from previous builds together to avoid duplicating a lot of content on the servers (both upload & storage) thereby speeding up the builders. that is why your amd64-generic/paladin file ended up referring to amd64-generic-full. it doesn't have api keys (by design) because it's a public builder that we publish to the world. Googlers get access to the internal builds (like amd64-generic/chrome-xxx) so we should get that binpkg. i agree that it looks like an error in priority handling. lets recreate your scenario: $ export PORTAGE_BINHOST='gs://chromeos-prebuilt/board/amd64-generic/paladin-R66-10367.0.0-rc2/packages/ gs://chromeos-prebuilt/board/eve/paladin-R66-10367.0.0-rc2/packages/ gs://chromeos-prebuilt/board/amd64-generic/chrome-R66-10365.0.0-rc1/packages/' $ sudo rm -r /build/eve/var/cache/edb/binhost/ $ emerge-eve chromeos-chrome -Gfpq --nodeps gs://chromeos-prebuilt/board/amd64-generic/full-2018.02.02.070328/packages/chromeos-base/chromeos-chrome-66.0.3336.3_rc-r1.tbz2 whoops. lets poke further. $ cd /build/eve/var/cache/edb/binhost/chromeos-prebuilt/board/ $ grep -E '^(CPV|PATH): .*chromeos-base/chromeos-chrome' */*/packages/Packages amd64-generic/chrome-R66-10365.0.0-rc1/packages/Packages:CPV: chromeos-base/chromeos-chrome-66.0.3336.3_rc-r1 amd64-generic/chrome-R66-10365.0.0-rc1/packages/Packages:PATH: board/amd64-generic/chrome-R66-10365.0.0-rc1/packages/chromeos-base/chromeos-chrome-66.0.3336.3_rc-r1.tbz2 amd64-generic/paladin-R66-10367.0.0-rc2/packages/Packages:CPV: chromeos-base/chromeos-chrome-66.0.3336.3_rc-r1 amd64-generic/paladin-R66-10367.0.0-rc2/packages/Packages:PATH: board/amd64-generic/full-2018.02.02.070328/packages/chromeos-base/chromeos-chrome-66.0.3336.3_rc-r1.tbz2 eve/paladin-R66-10367.0.0-rc2/packages/Packages:CPV: chromeos-base/chromeos-chrome-66.0.3336.3_rc-r1 eve/paladin-R66-10367.0.0-rc2/packages/Packages:PATH: board/caroline/chrome-R66-10365.0.0-rc1/packages/chromeos-base/chromeos-chrome-66.0.3336.3_rc-r1.tbz2 so each binhost is offering a binpkg, and they're pointing at the respectively expected places. inverting the order of PORTAGE_BINHOST doesn't help ... portage still prefers the full version. going back to our previous portage-2.2.12 code shows that it respects the order of the variable. bisecting things down finds this: https://gitweb.gentoo.org/proj/portage.git/commit/?id=328dd4712f88cbb8ef390ae9eb471afa1ef781d7
,
Feb 8 2018
i've moved this report upstream w/reproduction: https://bugs.gentoo.org/647076
,
Feb 15 2018
Issue 812526 has been merged into this issue.
,
Feb 15 2018
Thanks for pointing me to this bug. This issue affects not only build correctness but also build performance. I've been investigating recent build performance problem independently (go/cros-binhost-util) and found that binary packages are chosen from multiple binhosts almost randomly (precisely build timestamp order). Since Portage refuses to use binary packages from different binhosts depending in build time, such randomness triggers massive rebuild from source even with clean CrOS checkout. If we know the culprit commit that started to ignore binhost order, how about reverting it first? This issue heavily impacts our productivity.
,
Feb 15 2018
there's a patch attached to the upstream bug report if you want to give it a try to see if it improves things. i'll post a CL for that in the morning to our repo so we can at least get things back on track.
,
Feb 15 2018
Thanks Mike, I confirmed the patch posted in the upstream fixes the binary package utilization.
,
Feb 19 2018
,
Feb 26 2018
,
Feb 26 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/portage_tool/+/3f0c2efc08703f93b8473428b4e0fda0dc640fb8 commit 3f0c2efc08703f93b8473428b4e0fda0dc640fb8 Author: Mike Frysinger <vapier@chromium.org> Date: Mon Feb 26 22:10:06 2018 binarytree: clear build_time with binhosts Before the multi-binhost support in commit 328dd4712f88cbb8ef390a, portage would only consult one CPV. This allowed us to control the ordering based on the PORTAGE_BINHOST so we could have public full builders (including a Chromium browser) and we could have internal Chrome builds, and the last one would be preferred over the first. After that commit, the build time is mixed in, and portage will (reasonably) consider a newer build as better. The trouble is that our full builders run more often than our Chrome PFQ builders, so the Chromium build will have a newer time, and we install that. For now, lets clear the build_time setting with the binpkgs. This lets us retain the PORTAGE_BINHOST ordering until we can figure out a better way of managing this, or upstream gets a knob for us. We also don't bother inserting the new entry into the "metadata" variable because it isn't used after this logic, and the only code here that cares about it is the test for "is a local binpkg already available". https://bugs.gentoo.org/647076 BUG= chromium:809312 TEST=precq passes Change-Id: I08030f1951411e41ce2bcb510cd8241796a1258b Reviewed-on: https://chromium-review.googlesource.com/922807 Commit-Ready: Mike Frysinger <vapier@chromium.org> Tested-by: Mike Frysinger <vapier@chromium.org> Reviewed-by: Shuhei Takahashi <nya@chromium.org> [modify] https://crrev.com/3f0c2efc08703f93b8473428b4e0fda0dc640fb8/pym/portage/dbapi/bintree.py
,
Feb 26 2018
,
Feb 26 2018
,
Dec 13
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/portage_tool/+/b49bd17c9479bef966499ba9a323c9d0746a8a30 commit b49bd17c9479bef966499ba9a323c9d0746a8a30 Author: Mike Frysinger <vapier@chromium.org> Date: Thu Dec 13 21:14:09 2018 binarytree: clear build_time with binhosts Before the multi-binhost support in commit 328dd4712f88cbb8ef390a, portage would only consult one CPV. This allowed us to control the ordering based on the PORTAGE_BINHOST so we could have public full builders (including a Chromium browser) and we could have internal Chrome builds, and the last one would be preferred over the first. After that commit, the build time is mixed in, and portage will (reasonably) consider a newer build as better. The trouble is that our full builders run more often than our Chrome PFQ builders, so the Chromium build will have a newer time, and we install that. For now, lets clear the build_time setting with the binpkgs. This lets us retain the PORTAGE_BINHOST ordering until we can figure out a better way of managing this, or upstream gets a knob for us. We also don't bother inserting the new entry into the "metadata" variable because it isn't used after this logic, and the only code here that cares about it is the test for "is a local binpkg already available". https://bugs.gentoo.org/647076 BUG= chromium:809312 TEST=precq passes https://chromium-review.googlesource.com/922807 Change-Id: I6eef52c6c92ab177de19ec51ca209f1bb86835b0 Reviewed-on: https://chromium-review.googlesource.com/c/1377100 Commit-Queue: Allen Webb <allenwebb@google.com> Tested-by: Allen Webb <allenwebb@google.com> Reviewed-by: Mike Frysinger <vapier@chromium.org> [modify] https://crrev.com/b49bd17c9479bef966499ba9a323c9d0746a8a30/lib/portage/dbapi/bintree.py |
|||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||
Comment 1 by vapier@google.com
, Feb 6 2018