New issue
Advanced search Search tips

Issue 765897 link

Starred by 1 user

Issue metadata

Status: Archived
Owner:
Closed: Dec 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug

Blocking:
issue 753915



Sign in to add a comment

cros_workon reverse dep lookup is too expansive

Project Member Reported by vapier@chromium.org, Sep 16 2017

Issue description

after this CL:
  https://chromium-review.googlesource.com/609232

we suck up too many packages due to the use of --indirect.  is it really necessary to walk back the entire graph when the goal was to rebuild reverse deps ?  i don't think so.

example:
$ equery-eve -q depends --indirect bluez
net-libs/libpcap-1.5.3
 net-firewall/iptables-1.4.21-r4
  sys-apps/iproute2-4.9.0
   net-misc/dhcp-4.3.4

i can see rebuilding libpcap because it links against bluez, but i don't think we really need to go beyond that.  it's pretty uncommon that the changes will cascade that far, or that it happens on enough of a regular basis that devs want to eat the pain of all these pointless source rebuilds.
 

Comment 1 by derat@chromium.org, Sep 16 2017

Issue 765831 has been merged into this issue.
Hi Mike,

> is it really necessary to walk back the entire graph when the goal was to rebuild reverse deps ?  i don't think so.

Yes, that was our intention and I documented that at https://bugs.chromium.org/p/chromium/issues/detail?id=753915 and on https://bugs.chromium.org/p/chromium/issues/detail?id=732536#c29. To be extra safe, I publicly announced this change here: https://groups.google.com/a/chromium.org/forum/#!topic/chromium-os-dev/ueiuzrpLx6g

To reiterate, the issue is that occasionally package A output depends on B output and B output depends on C output: A->B->C. Using --indirect is the only way to catch A's change if C was being cros_workon'd. Without it, only B would be rebuilt. To get to the crux of the issue:

> it's pretty uncommon that the changes will cascade that far, or that it happens on enough of a regular basis that devs want to eat the pain of all these pointless source rebuilds.

It's this ambiguity that makes this necessary. Actually, Debian Reproducible builds has found that there are a lot of packages that have this spooky action at a distance far down the dependency tree: https://wiki.debian.org/BuildProfileSpec#Example

In addition, it's worth noting that this has been the default behavior for more than a month and the duped bug is the first report of a productivity problem.

Though it's impossible to prove, I suspect that at least some of the improvement in CQ failure rate in the past month are the result of this change: some failures are being caught on an eng environment before being sent to the CQ: we finally have correct cros_workon builds that mirror what is happening on CQ.

All that said, I would be in favor of a flag that lets someone disable the revdep calculation if they know what they're doing. Something like: --no_cros_workon_revdeps.

What do you think?

Comment 3 by derat@chromium.org, Sep 18 2017

Rebuilding chromeos-chrome from source is pretty painful (20-30 minutes unless Goma is set up, I believe) and usually not intentional, I suspect. It's my belief that most OS-side developers never want to build Chrome, while most UI developers use the Simple Chrome workflow when they're working on Chrome. I suspect that it'd be better if we could disable reverse deps calculation just for chromeos-chrome instead of adding an all-or-nothing flag.
Sure, that makes sense, too. Either one is fine with me. Mike?

Comment 5 by derat@chromium.org, Oct 25 2017

Mike, any thoughts about the question in #4? It'd be nice to be able to work on system_api without making my builds super-slow.
Status: Started (was: Available)
Cc: shapiroc@chromium.org
Project Member

Comment 8 by bugdroid1@chromium.org, Dec 13 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/crosutils/+/40c2117fbc8eb006096a1dcf2d55e05347c1a5f5

commit 40c2117fbc8eb006096a1dcf2d55e05347c1a5f5
Author: Jason D. Clinton <jclinton@chromium.org>
Date: Wed Dec 13 20:33:36 2017

build_packages: Skip Chrome build in revdeps of cros_workon packages

BUG= chromium:765897 
TEST=cros_workon chromeos-base/gestures; ./build_packages;
chromeos-chrome not included in rebuild list

Change-Id: I9287364d8b3527ffec1c9e55a068f2482e0a41d6
Reviewed-on: https://chromium-review.googlesource.com/823240
Commit-Ready: Jason Clinton <jclinton@chromium.org>
Tested-by: Jason Clinton <jclinton@chromium.org>
Reviewed-by: Dan Erat <derat@chromium.org>

[modify] https://crrev.com/40c2117fbc8eb006096a1dcf2d55e05347c1a5f5/build_packages

Status: Fixed (was: Started)
Status: Archived (was: Fixed)

Sign in to add a comment