New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 811542 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Jul 6
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug
Build-Toolchain



Sign in to add a comment

cros-go eclass does not support splitting a repository into multiple projects with interdependencies?

Project Member Reported by ayatane@chromium.org, Feb 13 2018

Issue description

The dev-go/gcp package needs to be split into two packages to resolve dependency cycles.

The CL for the split package: https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/915127

However, I can't get the dev-go/grpc-credentials-oauth package to build.

The build seems to be looking for google.golang.org/grpc/credentials

google.golang.org/grpc/credentials is packaged in dev-go/grpc, but including dev-go/grpc in DEPEND and RDEPEND does not fix the issue.

(cr) ((8e3381b...)) ayatane@sharanohiar ~/trunk/src/scripts $ sudo emerge dev-go/grpc-credentials-oauth
Calculating dependencies... done!

>>> Emerging (1 of 1) dev-go/grpc-credentials-oauth-1.6.0-r1::chromiumos
 * Running stacked hooks for pre_pkg_setup
 *    sysroot_build_bin_dir ...                                                                                                                                                                                                                                                                                        [ ok ]
 * Running stacked hooks for post_pkg_setup
 *    python_eclass_hack ...                                                                                                                                                                                                                                                                                           [ ok ]
 * Running stacked hooks for pre_src_unpack
 *    python_multilib_setup ...                                                                                                                                                                                                                                                                                        [ ok ]
>>> Unpacking source...
GIT update -->
   repository:               https://chromium.googlesource.com/external/github.com/grpc/grpc-go.git
   at the commit:            d50734d1d6ca477a72646f3022216ec39639f4cd
   commit:                   v1.6.0
   branch:                   master
   storage directory:        "/var/cache/chromeos-cache/distfiles/host/egit-src/external/github.com/grpc/grpc-go"
   checkout type:            bare repository
Cloning into '/var/tmp/portage/dev-go/grpc-credentials-oauth-1.6.0-r1/work/grpc-credentials-oauth-1.6.0/src/google.golang.org/grpc'...
done.
Switched to a new branch 'tree-v1.6.0'
>>> Unpacked to /var/tmp/portage/dev-go/grpc-credentials-oauth-1.6.0-r1/work/grpc-credentials-oauth-1.6.0/src/google.golang.org/grpc
>>> Source unpacked in /var/tmp/portage/dev-go/grpc-credentials-oauth-1.6.0-r1/work
 * Running stacked hooks for post_src_unpack
 *    asan_init ...                                                                                                                                                                                                                                                                                                    [ ok ]
>>> Preparing source in /var/tmp/portage/dev-go/grpc-credentials-oauth-1.6.0-r1/work/grpc-credentials-oauth-1.6.0 ...
>>> Source prepared.
>>> Configuring source in /var/tmp/portage/dev-go/grpc-credentials-oauth-1.6.0-r1/work/grpc-credentials-oauth-1.6.0 ...
>>> Source configured.
>>> Compiling source in /var/tmp/portage/dev-go/grpc-credentials-oauth-1.6.0-r1/work/grpc-credentials-oauth-1.6.0 ...
>>> Source compiled.
>>> Test phase [not enabled]: dev-go/grpc-credentials-oauth-1.6.0-r1

>>> Install grpc-credentials-oauth-1.6.0-r1 into /var/tmp/portage/dev-go/grpc-credentials-oauth-1.6.0-r1/image/ category dev-go
 * Installing "google.golang.org/grpc/credentials/oauth"
[../../image/usr/lib/gopath/src/google.golang.org/grpc/credentials/oauth/oauth.go:31:2: no Go files in /var/tmp/portage/dev-go/grpc-credentials-oauth-1.6.0-r1/image/usr/lib/gopath/src/google.golang.org/grpc/credentials]
 * ERROR: dev-go/grpc-credentials-oauth-1.6.0-r1::chromiumos failed (install phase):
 *   Package has missing dependency: "google.golang.org/grpc/credentials/oauth"
 * 
 * Call stack:
 *     ebuild.sh, line  133:  Called src_install
 *   environment, line 3493:  Called cros-go_src_install
 *   environment, line  757:  Called die
 * The specific snippet of code:
 *               die "Package has missing dependency: \"${pkg}\"";
 * 
 * If you need support, post the output of `emerge --info '=dev-go/grpc-credentials-oauth-1.6.0-r1::chromiumos'`,
 * the complete build log and the output of `emerge -pqv '=dev-go/grpc-credentials-oauth-1.6.0-r1::chromiumos'`.
 * The complete build log is located at '/var/log/portage/dev-go:grpc-credentials-oauth-1.6.0-r1:20180213-001945.log'.
 * For convenience, a symlink to the build log is located at '/var/tmp/portage/dev-go/grpc-credentials-oauth-1.6.0-r1/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/dev-go/grpc-credentials-oauth-1.6.0-r1/temp/environment'.
 * Working directory: '/var/tmp/portage/dev-go/grpc-credentials-oauth-1.6.0-r1/work/grpc-credentials-oauth-1.6.0'
 * S: '/var/tmp/portage/dev-go/grpc-credentials-oauth-1.6.0-r1/work/grpc-credentials-oauth-1.6.0'

>>> Failed to emerge dev-go/grpc-credentials-oauth-1.6.0-r1, Log file:

>>>  '/var/log/portage/dev-go:grpc-credentials-oauth-1.6.0-r1:20180213-001945.log'

 * Messages for package dev-go/grpc-credentials-oauth-1.6.0-r1:

 * ERROR: dev-go/grpc-credentials-oauth-1.6.0-r1::chromiumos failed (install phase):
 *   Package has missing dependency: "google.golang.org/grpc/credentials/oauth"
 * 
 * Call stack:
 *     ebuild.sh, line  133:  Called src_install
 *   environment, line 3493:  Called cros-go_src_install
 *   environment, line  757:  Called die
 * The specific snippet of code:
 *               die "Package has missing dependency: \"${pkg}\"";
 * 
 * If you need support, post the output of `emerge --info '=dev-go/grpc-credentials-oauth-1.6.0-r1::chromiumos'`,
 * the complete build log and the output of `emerge -pqv '=dev-go/grpc-credentials-oauth-1.6.0-r1::chromiumos'`.
 * The complete build log is located at '/var/log/portage/dev-go:grpc-credentials-oauth-1.6.0-r1:20180213-001945.log'.
 * For convenience, a symlink to the build log is located at '/var/tmp/portage/dev-go/grpc-credentials-oauth-1.6.0-r1/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/dev-go/grpc-credentials-oauth-1.6.0-r1/temp/environment'.
 * Working directory: '/var/tmp/portage/dev-go/grpc-credentials-oauth-1.6.0-r1/work/grpc-credentials-oauth-1.6.0'
 * S: '/var/tmp/portage/dev-go/grpc-credentials-oauth-1.6.0-r1/work/grpc-credentials-oauth-1.6.0'
(cr) ((8e3381b...)) ayatane@sharanohiar ~/trunk/src/scripts $
 
Is it really for Infra>Platform>CIPD component? I don't see anything relating to CIPD here.
Components: -Infra>Platform>CIPD OS>Packages
Not sure, I couldn't find a more suitable component, and I don't know of a way to search components other than by autocompletion.  I have found a different component that I think is more suitable and probably right.

Comment 3 Deleted

Comment 4 Deleted

The problem is how Go resolves packages using GOPATH.

In this instance I can fix it by flipping GOPATH in the cros-go eclass

modified   eclass/cros-go.eclass
@@ -61,7 +61,7 @@ DEPEND="dev-lang/go"
 
 cros_go() {
 	local workspace="${CROS_GO_WORKSPACE:-${S}}"
-	GOPATH="${workspace}:${SYSROOT}/usr/lib/gopath" \
+	GOPATH="${SYSROOT}/usr/lib/gopath:${workspace}" \
 		$(tc-getGO) "$@" || die
 }

The patch makes it work when foo/bar/baz depends on foo/bar, but conversely breaks when foo/bar depends on foo/bar/baz.  It's either/or, can't have both.

This is an ugly problem, I don't have any good ideas at the moment.
Repro:

Make two hello world packages

p1/src/foo/bar/main.go
p2/src/foo/main.go

export GOPATH=p1:p2

go build foo/bar works, go build foo does not
There's a similar issue with internal packages.  Currently, there's no way to share an internal Go package with multiple co-resident Go packages packaged as separate ebuilds.

e.g.

google.golang.org/api/option requires google.golang.org/api/internal
google.golang.org/api/transport requires google.golang.org/api/internal

In the ebuild, each package can only access the internal package if it is under the same directory in GOPATH.  Thus, the only option is to package everything as one ebuild currently, which creates dependency cycles.
Maybe there should be a CROS_GO_WHITELIST or some such that allows copying source files into the destdir while doing dependency checking and then removing them afterward.

Of course there's the potential to "forget" a dependency, but since you need to whitelist explicitly it's straightforward to check the DEPENDS and make sure all whitelisted package are actually depended on.
I haven't looked at the dependency cycles yet, but the issue in #5 and #6 seems to be in the Go toolchain.
Do you know if this is intentional? Is there an upstream bug for it?
Even if upstream is not willing to change this behavior, we can do it in our toolchain.
That seems preferable to me over creating any CROS_GO_WHITELISTs.

It is wontfix I believe https://groups.google.com/a/google.com/forum/#!topic/go-nuts/6EvlF7pIZno

I could take this to real upstream, but I am sufficiently convinced that the problem is not trivial to fix.

I vaguely recall encountering an upstream discussion about this issue a year ago that was concluded wontfix, but I cannot find it.
Components: -OS>Packages Tools>ChromeOS-Toolchain
Owner: rahulchaudhry@chromium.org
Status: Assigned (was: Untriaged)
Project Member

Comment 13 by bugdroid1@chromium.org, Feb 21 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/9eb388ebf3e16a1e7c318877acdf9f3def4aff29

commit 9eb388ebf3e16a1e7c318877acdf9f3def4aff29
Author: Allen Li <ayatane@chromium.org>
Date: Wed Feb 21 04:31:33 2018

cros-go: Disable dependency checking to work around bug

BUG= chromium:749300 
BUG= chromium:811542 
TEST=None

Change-Id: I2c3aa51acdda2f8eac11e0b768927008b5ab8e09
Reviewed-on: https://chromium-review.googlesource.com/924601
Commit-Ready: Allen Li <ayatane@chromium.org>
Tested-by: Allen Li <ayatane@chromium.org>
Reviewed-by: Rahul Chaudhry <rahulchaudhry@chromium.org>

[modify] https://crrev.com/9eb388ebf3e16a1e7c318877acdf9f3def4aff29/eclass/cros-go.eclass

Project Member

Comment 14 by bugdroid1@chromium.org, Jul 5

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/9e1773bfe328a505b61f3cb5bc297d4d3b2e759d

commit 9e1773bfe328a505b61f3cb5bc297d4d3b2e759d
Author: Rahul Chaudhry <rahulchaudhry@chromium.org>
Date: Thu Jul 05 01:24:27 2018

cros-go.eclass: check for missing dependencies of installed packages.

Moved dependency checking to pkg_postinst stage.

BUG= chromium:749300 
BUG= chromium:811542 
TEST='sudo emerge dev-go/grpc-credentials-oauth' works.
TEST=Emerged dev-go/* (50 ebuilds).

Change-Id: Idd9182c40da5dedb9741c081dc2c7c58171e13a9
Reviewed-on: https://chromium-review.googlesource.com/1124668
Commit-Ready: Rahul Chaudhry <rahulchaudhry@chromium.org>
Tested-by: Rahul Chaudhry <rahulchaudhry@chromium.org>
Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
Reviewed-by: Rahul Chaudhry <rahulchaudhry@chromium.org>

[modify] https://crrev.com/9e1773bfe328a505b61f3cb5bc297d4d3b2e759d/eclass/cros-go.eclass

Status: Fixed (was: Assigned)

Sign in to add a comment