New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 893285 link

Starred by 2 users

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug

Blocking:
issue 889286



Sign in to add a comment

Older "firmwarebranch" builds failing to fetch binary prebuilts during WorkspaceInitSDK

Project Member Reported by dgarr...@chromium.org, Oct 8

Issue description

Some older firmwarebranch builds are failing in InitSDK because of SSL issues.

Example:

=== Start output for job ja-motoyafonts-1.0.0 (1m43.0s) ===
ja-motoyafonts-1.0.0: >>> Fetching (1 of 1) chromeos-base/ja-motoyafonts-1.0.0 from chromeos-overlay
ja-motoyafonts-1.0.0: !!! RESUMECOMMAND_GS does not contain the required ${FILE} parameter.
ja-motoyafonts-1.0.0: !!! Refer to the make.conf(5) man page for information about how to
ja-motoyafonts-1.0.0: !!! correctly specify FETCHCOMMAND and RESUMECOMMAND.
ja-motoyafonts-1.0.0: Failure: [Errno 1] _ssl.c:491: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed.
ja-motoyafonts-1.0.0: No digest file available and download failed.
ja-motoyafonts-1.0.0: 
ja-motoyafonts-1.0.0: !!! Couldn't download 'ja-motoyafonts-1.0.0.tar.gz'. Aborting.
ja-motoyafonts-1.0.0:  * Fetch failed for 'chromeos-base/ja-motoyafonts-1.0.0'
ja-motoyafonts-1.0.0: >>> Failed to emerge chromeos-base/ja-motoyafonts-1.0.0
=== Complete: job ja-motoyafonts-1.0.0 (1m43.0s) ===

 
This is not a binary prebuilt, but the source code.

From the ebuild:

clean$cat ja-motoyafonts-1.0.0.ebuild
# Copyright (c) 2012 The Chromium OS Authors. All rights reserved.
# Distributed under the terms of the GNU General Public License v2
# $Header: $

inherit font

DESCRIPTION="Motoya licensed fonts (Motoya G04 Gothic and Mincho)"
HOMEPAGE="http://www.motoya.co.jp"
SRC_URI="gs://chromeos-localmirror-private/distfiles/${P}.tar.gz"

LICENSE="Proprietary"
SLOT="0"
KEYWORDS="~alpha amd64 arm ~hppa ~ia64 ~ppc ~ppc64 ~s390 ~sh x86 ~x86-fbsd"
IUSE=""

FONT_SUFFIX="ttc"
FONT_S="${S}"
FONTDIR="/usr/share/fonts/ja-motoyafonts"

# Only installs fonts
RESTRICT="strip binchecks mirror"

src_install() {
        # call src_install() in font.eclass.
        font_src_install
}
Cc: vapier@chromium.org
vapier, you mentioned there is a hook we can override to download things for portage.

How does that work? Maybe it's time to try it out.
Status: Assigned (was: Started)
This only appears to apply to the very oldest active firmware branch "firmware-skate-3824.129.B".

Since it's only one branch of many, I'm lowering the priority, for now.
hmm, this error confuses me:
ja-motoyafonts-1.0.0: !!! RESUMECOMMAND_GS does not contain the required ${FILE} parameter.

looking at the firmware-skate-3824.129.B branch, it has FILE just fine:
https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/firmware-skate-3824.129.B/chromeos/config/make.conf.common-target#49
FETCHCOMMAND_GS="bash -c 'BOTO_CONFIG=/home/\${PORTAGE_USERNAME}/.boto gsutil cp \"${URI}\" \"${DISTDIR}/${FILE}\"'"
RESUMECOMMAND_GS="bash -c 'BOTO_CONFIG=/home/\${PORTAGE_USERNAME}/.boto gsutil cp \"${URI}\" \"${DISTDIR}/${FILE}\"'"

so i don't know what's going on with that.

focusing on the "how to make gsutil work in the chroot", we have a few options.  the interface with portage is simply the FETCHCOMMAND_GS & RESUMECOMMAND_GS vars noted above.  so we have to get it into the target env.  /etc/make.conf.user has existed since the start of the project and only exists for users to tweak.  so we could write into it like:
FETCHCOMMAND_GS='/mnt/host/workspace/chromite/bin/gs_fetch_binpkg --boto "/home/${PORTAGE_USERNAME}/.boto" "\${URI}" "\${DISTDIR}/\${FILE}"'
RESUMECOMMAND_GS="${FETCHCOMMAND_GS}"

that wouldn't help with non gs:// URIs though.  those go through the $FETCHCOMMAND setting which just invokes `curl`.  but if we had a similar wrapper in place, we could write out the FETCHCOMMAND variable to make.conf.user and point it to a workspace-injected tool.

another thought is that, if we're only seeing certificate failures, and the tools are all going through the system certificate store, we can bind mount that to something newer.  maybe add to the workspace wrapper a bind mount from the host distro's /etc/ssl/certs/ca-certificates.crt to the chroot/etc/ssl/certs/ca-certificates.crt path ?
I like the idea of using gsutil from TOT branch, so that all network versioning problems are solved, not just the cert store.

Handling all gs:// URLs is a really good start.

I take it that there aren't similar hooks for http, https, or git?
all the other protos go through FETCHCOMMAND which currently runs `curl` which uses the system CA store for cert checking

if pointing to the workspace's copy of gs_fetch_binpkg works for gsutil, and bind mounting the CA store works for everything else, that (hopefully) shouldn't be too painful
Curl is pretty darn compatible.

What about authentication? Is there any value in using a wrapper in TOT that could invisibly use modern account info?
afaik, gsutil is the only tool that has ACLs.  all other stuff should be direct anonymous access with curl.
Git and git bundles are authenticated for the internal repos.
the git repos should have been fetched during the manifest & chrome sync stages right ?  there's really no other git repo that would normally get fetched inside the chroot.
I thought the ebuild could sometimes fetch it remotely. I know we try to use local copies, but I thought there were a few exceptions sometimes.
there could be, but we've moved away from that and discourage further usage
Labels: -Pri-3 Pri-2
As shown here: go/firmwarebranch-build-status

This is affecting the seven oldest firmware branches.
Blocking: 889286
Cc: pgeorgi@chromium.org martinroth@chromium.org
Cc: jean@chromium.org stevenh@chromium.org dgarr...@chromium.org
 Issue 903481  has been merged into this issue.

Sign in to add a comment