Older "firmwarebranch" builds failing to fetch binary prebuilts during WorkspaceInitSDK |
|||||||
Issue description
Some older firmwarebranch builds are failing in InitSDK because of SSL issues.
Example:
=== Start output for job ja-motoyafonts-1.0.0 (1m43.0s) ===
ja-motoyafonts-1.0.0: >>> Fetching (1 of 1) chromeos-base/ja-motoyafonts-1.0.0 from chromeos-overlay
ja-motoyafonts-1.0.0: !!! RESUMECOMMAND_GS does not contain the required ${FILE} parameter.
ja-motoyafonts-1.0.0: !!! Refer to the make.conf(5) man page for information about how to
ja-motoyafonts-1.0.0: !!! correctly specify FETCHCOMMAND and RESUMECOMMAND.
ja-motoyafonts-1.0.0: Failure: [Errno 1] _ssl.c:491: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed.
ja-motoyafonts-1.0.0: No digest file available and download failed.
ja-motoyafonts-1.0.0:
ja-motoyafonts-1.0.0: !!! Couldn't download 'ja-motoyafonts-1.0.0.tar.gz'. Aborting.
ja-motoyafonts-1.0.0: * Fetch failed for 'chromeos-base/ja-motoyafonts-1.0.0'
ja-motoyafonts-1.0.0: >>> Failed to emerge chromeos-base/ja-motoyafonts-1.0.0
=== Complete: job ja-motoyafonts-1.0.0 (1m43.0s) ===
,
Oct 9
vapier, you mentioned there is a hook we can override to download things for portage. How does that work? Maybe it's time to try it out.
,
Oct 11
This only appears to apply to the very oldest active firmware branch "firmware-skate-3824.129.B". Since it's only one branch of many, I'm lowering the priority, for now.
,
Oct 11
hmm, this error confuses me:
ja-motoyafonts-1.0.0: !!! RESUMECOMMAND_GS does not contain the required ${FILE} parameter.
looking at the firmware-skate-3824.129.B branch, it has FILE just fine:
https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/firmware-skate-3824.129.B/chromeos/config/make.conf.common-target#49
FETCHCOMMAND_GS="bash -c 'BOTO_CONFIG=/home/\${PORTAGE_USERNAME}/.boto gsutil cp \"${URI}\" \"${DISTDIR}/${FILE}\"'"
RESUMECOMMAND_GS="bash -c 'BOTO_CONFIG=/home/\${PORTAGE_USERNAME}/.boto gsutil cp \"${URI}\" \"${DISTDIR}/${FILE}\"'"
so i don't know what's going on with that.
focusing on the "how to make gsutil work in the chroot", we have a few options. the interface with portage is simply the FETCHCOMMAND_GS & RESUMECOMMAND_GS vars noted above. so we have to get it into the target env. /etc/make.conf.user has existed since the start of the project and only exists for users to tweak. so we could write into it like:
FETCHCOMMAND_GS='/mnt/host/workspace/chromite/bin/gs_fetch_binpkg --boto "/home/${PORTAGE_USERNAME}/.boto" "\${URI}" "\${DISTDIR}/\${FILE}"'
RESUMECOMMAND_GS="${FETCHCOMMAND_GS}"
that wouldn't help with non gs:// URIs though. those go through the $FETCHCOMMAND setting which just invokes `curl`. but if we had a similar wrapper in place, we could write out the FETCHCOMMAND variable to make.conf.user and point it to a workspace-injected tool.
another thought is that, if we're only seeing certificate failures, and the tools are all going through the system certificate store, we can bind mount that to something newer. maybe add to the workspace wrapper a bind mount from the host distro's /etc/ssl/certs/ca-certificates.crt to the chroot/etc/ssl/certs/ca-certificates.crt path ?
,
Oct 11
I like the idea of using gsutil from TOT branch, so that all network versioning problems are solved, not just the cert store. Handling all gs:// URLs is a really good start. I take it that there aren't similar hooks for http, https, or git?
,
Oct 11
all the other protos go through FETCHCOMMAND which currently runs `curl` which uses the system CA store for cert checking if pointing to the workspace's copy of gs_fetch_binpkg works for gsutil, and bind mounting the CA store works for everything else, that (hopefully) shouldn't be too painful
,
Oct 11
Curl is pretty darn compatible. What about authentication? Is there any value in using a wrapper in TOT that could invisibly use modern account info?
,
Oct 12
afaik, gsutil is the only tool that has ACLs. all other stuff should be direct anonymous access with curl.
,
Oct 12
Git and git bundles are authenticated for the internal repos.
,
Oct 12
the git repos should have been fetched during the manifest & chrome sync stages right ? there's really no other git repo that would normally get fetched inside the chroot.
,
Oct 12
I thought the ebuild could sometimes fetch it remotely. I know we try to use local copies, but I thought there were a few exceptions sometimes.
,
Oct 12
there could be, but we've moved away from that and discourage further usage
,
Oct 15
As shown here: go/firmwarebranch-build-status This is affecting the seven oldest firmware branches.
,
Oct 15
,
Oct 15
,
Nov 8
Issue 903481 has been merged into this issue. |
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by dgarr...@chromium.org
, Oct 9