New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 752196 link

Starred by 1 user
Status: Archived
Owner: ----
Closed: Mar 2018
Cc:
davidri...@chromium.org
vapier@chromium.org
█ bhthompson@chromium.org
nya@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug

Blocked on:
issue 659864


Sign in to add a comment

AndroidMetadata stage is slow

Project Member Reported by akes...@chromium.org, Aug 3 2017 
This stage is on the critical path (runs before BuildPackages) and is kind of slow (at least based on what it sounds like it is doing), taking a median of 2 minutes.

http://shortn/_Hq61VmkjAq

(also, this median time increased significantly around July 13)

Can this stage be made faster by doing a cheaper computation. Or can its results be extracted as a side effect of some other stage like BuildPackages that will run later, or can it be parallelized with a non-critical-path stage?

Comment 1 by nya@chromium.org, Aug 4 2017 

Looks like BuilderRun.DetermineAndroidPackage() is slow. It lists all portage dependencies from virtual/target-os.

One example log:
https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/no_vmtest_pre_cq/builds/88728/steps/AndroidMetadata/logs/stdio
10:40:23: INFO: RunCommand: /b/c/cbuild/repository/chromite/bin/cros_sdk -- emerge-cyan -p --cols --quiet --root /var/empty -e virtual/target-os in /b/c/cbuild/repository
10:43:24: INFO: Board cyan has Android version 4234098

Hmm BTW I guess we should run this after BuildPackage stage is finished -- otherwise AndroidMetadata stage is blamed for package dependency corruptions, which is confusing.

I think we can move AndroidMetadata stage after BuildPackage stage. Then dependency query should be faster by hot cache. Total build time might not be improved by this change, though.

Comment 2 by bhthompson@google.com, Aug 4 2017

Cc: vapier@chromium.org
+vapier whom may have optimization suggestions

The reason this is before BuildPackages is so that we still know what Android version is there even if BuildPackages fails on some unrelated packages. 

I do agree it is annoying that Portage level failures trigger breakage in AndroidMetadata instead of BuildPackages, my thought on resolving this previously was to allow the AndroidMetadata to pass anyway if it cannot get a version and allow BuildPackages to really fail later, but it is not clear this is optimal either. 

Maybe we should add better logging explaining why it might have failed here?

Comment 3 by nya@chromium.org, Aug 4 2017 

Thanks Bernie, makes sense.

Provided that running AndroidMetadata before BuildPackages does not produce confusing failures, I think it's fine to leave this stage as-is because I guess (AndroidMetadata duration) + (BuildPackages duration) does not change much by moving around AndroidMetadata stage as long as file cache is hot.

Comment 4 by vapier@chromium.org, Aug 4 2017 

two things:
- dep calc being slow seems like  issue 659864 
- the android stages prob should be run in parallel with SyncChromeStage

Comment 5 by akes...@chromium.org, Aug 9 2017

Blockedon: 659864
partially blocked on portage cache

Comment 6 by ayatane@chromium.org, Mar 14 2018

Status: Archived (was: Untriaged)
This bug is very old, is Untriaged, and has no owner.  If it is still relevant, reopen as Untriaged or open a new bug
►

Sign in to add a comment