+SDK folks 

My suspicion is purely based on the time of the regression.

"i see an sdk roll just before this
timing seems to line up
b3185b045957666477f0b5a39cebc3e3dbcfb4cc in chromiumos-overlay (not visible in gerrit)
don't see any chromite suspect in the time range"

seems unlikely that a toolchain/sdk roll would impact the uprev stage as the uprev logic is entirely in python (and git) and run outside of the chroot.  we execute uprev before initsdk by design.

yes, I really dont see how this is related to the toolchain...

is this related somehow? (I doubt it but double checking)

https://bugs.chromium.org/p/chromium/issues/detail?id=793540

The logs of uprev differs a lot between 'good' and 'bad' one
https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos%2Ffalco-paladin%2F17682%2F%2B%2Frecipes%2Fsteps%2FUprev%2F0%2Fstdout

https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos%2Ffalco-full-compile-paladin%2F11519%2F%2B%2Frecipes%2Fsteps%2FUprev%2F0%2Fstdout

I saw a "regen cache" message somewhere.  And there were some ebuilds that were upreved even though the 9999 ebuild was not touched. 

https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos%2Ffalco-paladin%2F17682%2F%2B%2Frecipes%2Fsteps%2FUprev%2F0%2Fstdout

12:31:34: INFO: Creating new stable ebuild /b/c/cbuild/repository/src/third_party/chromiumos-overlay/sys-libs/libcxxabi/libcxxabi-4.0.0-r30.ebuild
12:31:34: INFO: Creating new stable ebuild /b/c/cbuild/repository/src/third_party/chromiumos-overlay/sys-libs/gcc-libs/gcc-libs-4.9.2-r39.ebuild

So I suspect it is not related to toolchain changes but some sort of periodic refresh done by CQ?

i doubt coreboot-sdk is related as that's also inside the sdk only

those two logs are very diff because they're comparing very different runs ...
https://uberchromegw.corp.google.com/i/chromeos/builders/falco-paladin/builds/17682
https://uberchromegw.corp.google.com/i/chromeos/builders/falco-full-compile-paladin/builds/11519

notice the number of CLs in each run.

Compared the two auron_yuna-paladin builds

https://uberchromegw.corp.google.com/i/chromeos/builders/auron_yuna-paladin/builds/1899
(Uprev Stage: 7 mins, 34 secs)

06:42:43: INFO: RunCommand: /b/c/cbuild/repository/chromite/bin/cros_mark_as_stable --all '--boards=auron_yuna' '--overlays=...'
06:45:23: INFO: Skip: Determined that none of the ebuild autotest-private-tests-echoprivate rev_subdirs was touched ['client/site_tests/extension_EchoPrivate']


https://uberchromegw.corp.google.com/i/chromeos/builders/auron_yuna-paladin/builds/1900

(Uprev stage: 16 mins, 51 secs )

09:53:56: INFO: RunCommand: /b/c/cbuild/repository/chromite/bin/cros_mark_as_stable --all '--boards=auron_yuna' '--overlays=...'
10:01:56: INFO: Skip: Determined that none of the ebuild autotest-private-tests-echoprivate rev_subdirs was touched ['client/site_tests/extension_EchoPrivate']


builds/1899 took ~ 3 mins to setup and get the overlay->ebuilds map, while builds/1900 took ~ 8 mins to get the overlay -> ebuilds map before starting processing each overlay. Didn't see big changes in GCE performance. Were some big projects added to the overlays? 

Just noticed Uprev became slower since Jan 10th only in GCE builders (http://shortn/_i3Q4HHB16w, http://shortn/_dhZCS263cz), not in baremetal builders (http://shortn/_P0H1gWU3fD, http://shortn/_kLPQEjzLsa)

I wonder if GCE disks have changed behavior in some what that especially affects uprev.

Or that it's related to how much RAM the machines have? Though the GCE instances have way more ram.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/db884322454e003b5e93e45884b3f25c864281a5

commit db884322454e003b5e93e45884b3f25c864281a5
Author: Ningning Xia <nxia@chromium.org>
Date: Fri Feb 02 21:19:20 2018

Get Ebuilds for all overlays in parallel.

BUG= chromium:805077 ; chromium:803634
TEST=unit_tests

Change-Id: I9189933476e0f92a1375501a8bb28e28cdec2132
Reviewed-on: https://chromium-review.googlesource.com/890234
Commit-Ready: Ningning Xia <nxia@chromium.org>
Tested-by: Ningning Xia <nxia@chromium.org>
Reviewed-by: Ningning Xia <nxia@chromium.org>

[modify] https://crrev.com/db884322454e003b5e93e45884b3f25c864281a5/lib/portage_util_unittest.py
[modify] https://crrev.com/db884322454e003b5e93e45884b3f25c864281a5/scripts/cros_mark_as_stable_unittest.py
[modify] https://crrev.com/db884322454e003b5e93e45884b3f25c864281a5/scripts/cros_mark_as_stable.py
[modify] https://crrev.com/db884322454e003b5e93e45884b3f25c864281a5/lib/portage_util.py

Still don't know why uprev took longer time to finish since Jan 10th, but the improvement CL in #14 has made the "50%ile" uprev down to 440s. will have one more CL to improve the efficiency.

http://shortn/_HkTN7ApCV2

running things in parallel is good, and we want to do it, but i think there's a significant underlying issue that we don't have a handle on.  in the log Manoj highlighted, gcc-libs shouldn't have been uprevved in the first place.

I added some logs before, and I believe gcc-lib has always been uprevved as it satisfies http://shortn/_sQ9mE8TC4W ("not subdirs_to_rev and not self.cros_workon_vars.rev_subdirs"). We started noticing that as jclinton@ turned on the logging in https://chromium-review.googlesource.com/c/chromiumos/chromite/+/865441. 


I mentioned in #12, the slowness only happened to the GCE machines, not the bare-metal machines. There could be some performance regression in the GCE machines, I'm not sure what that is though. 

so we are incorrectly flagging some packages for uprevving when we shouldn't be ?

+ pmalani@/sjg@, should http://shortn/_sQ9mE8TC4W be applied to test ebuilds?

I meant only apply it to test ebuilds and avoid other ebuilds.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/419e4ebf068e91963b71da73770439085855fcaf

commit 419e4ebf068e91963b71da73770439085855fcaf
Author: Ningning Xia <nxia@google.com>
Date: Thu Feb 08 05:27:26 2018

Uprev overlays belonging to different repo in parallel.

BUG= chromium:805077 ;chromium:803634
TEST=unit_tests

Change-Id: I99378005c6f287f67907b51fbbcbf99ba01c909c
Reviewed-on: https://chromium-review.googlesource.com/902442
Commit-Ready: Ningning Xia <nxia@chromium.org>
Tested-by: Ningning Xia <nxia@chromium.org>
Reviewed-by: Ningning Xia <nxia@chromium.org>

[modify] https://crrev.com/419e4ebf068e91963b71da73770439085855fcaf/scripts/cros_mark_as_stable_unittest.py
[modify] https://crrev.com/419e4ebf068e91963b71da73770439085855fcaf/lib/git.py
[modify] https://crrev.com/419e4ebf068e91963b71da73770439085855fcaf/scripts/cros_mark_as_stable.py