New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 592715 link

Starred by 1 user

Issue metadata

Status: Verified
Owner:
Closed: Apr 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Feature

Blocked on:
issue 592808
issue 596970



Sign in to add a comment

Speed up culprit finding for compile failures using output nodes of failed edges in ninja.

Project Member Reported by st...@chromium.org, Mar 7 2016

Issue description

Background: currently we use dependency "analyze" to determine which targets to be built at a given revision/commit. This does reduce the number of targets to be built than a full build. However, it is still not fast enough (still over an hour on average).

With info from ninja, we could do more fine-grained rerun of compile. When a compile failure occurs, ninja knows which edges failed. An edge here could be a CXX or CC for compile from source files to object files, a LINK for link from object files to libs or executables, an ACTION for running python to generate code, etc. We could retrieve the output nodes (object files, libs/executables, generated code, etc) from ninja, and then feed them back into ninja to rerun the compile step. In this way, we could minimize the number of tasks to be executed by ninja, eg: if a CXX/CC compile fails, we don't have to compile all source files and run the LINK for the executable which runs much longer than a single CXX.
Experiments shows that compile time could be reduced from 1+ hour to ~10 minutes. (Time for bot_update and gclient sync are excluded here.)

And CXX/CC failures are the majority of all compile failures. (The statistic data are from April 2015 to Feb 2016 of UNIQUE* failures in the compile steps.)
Faiure types          | # of occurrence    |     Percentage
compile (CXX/CC)      |     1272           |      70.70%
link (LINK)	      |     191            |      10.62%
others(ACTION,etc)    |     336            |      18.68%
*In the same builder, first-time failures of compile step after a green build are treated as unique.


The main tasks to achieve this are:
1. Collect output nodes of failed edges from ninja on the main waterfall.
   Note: need to wait for the next release of ninja 1.7
2. Pass these info to the culprit-finding recipe findit/chromium/compile.
3. Check if the given nodes still exist at a revision/commit against which compile is to be re-run.
   Eg: a source file could be added or deleted by some revision in the regression range.
4. Re-run compile for the existing nodes only.
 
Project Member

Comment 1 by bugdroid1@chromium.org, Mar 10 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/6b6925d0339a8d4b67a634254860116b7361fcc0

commit 6b6925d0339a8d4b67a634254860116b7361fcc0
Author: stgao@chromium.org <stgao@chromium.org>
Date: Thu Mar 10 01:35:53 2016

[Findit] Check existence of given targets before running compile in recipe findit/chromium/compile.

A "target" here is actually a node in ninja's build graph, eg:
1. An executable target like browser_tests
2. An object file like obj/path/to/Source.o
3. An action like build/linux:gio_loader
4. An generated header file like gen/library_loaders/libgio.h
5. and so on

Because a "target" could be added or deleted by some commit in the regression range, we have to check whether it still exists before re-running compile at a different commit. Otherwise, compile will always fail.
Use 'ninja -t query TARGET1 [TARGET2 ...]' to test whether "target"s exist or not: ninja exits with 0 if all the targets exist; otherwise with 1.


BUG= 592715 ,  560991 

Review URL: https://codereview.chromium.org/1766863002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/build@299206 0039d316-1c4b-4281-b951-d872f2087c98

[modify] https://crrev.com/6b6925d0339a8d4b67a634254860116b7361fcc0/scripts/slave/recipe_modules/findit/__init__.py
[modify] https://crrev.com/6b6925d0339a8d4b67a634254860116b7361fcc0/scripts/slave/recipe_modules/findit/api.py
[add] https://crrev.com/6b6925d0339a8d4b67a634254860116b7361fcc0/scripts/slave/recipe_modules/findit/resources/check_target_existence.py
[add] https://crrev.com/6b6925d0339a8d4b67a634254860116b7361fcc0/scripts/slave/recipes/findit/chromium/compile.expected/compile_none_existing_targets.json
[modify] https://crrev.com/6b6925d0339a8d4b67a634254860116b7361fcc0/scripts/slave/recipes/findit/chromium/compile.expected/compile_specified_targets.json
[modify] https://crrev.com/6b6925d0339a8d4b67a634254860116b7361fcc0/scripts/slave/recipes/findit/chromium/compile.expected/failed_compile_upon_infra_failure.json
[modify] https://crrev.com/6b6925d0339a8d4b67a634254860116b7361fcc0/scripts/slave/recipes/findit/chromium/compile.py

Comment 2 by st...@chromium.org, Mar 15 2016

Blockedon: 592808

Comment 3 by st...@chromium.org, Mar 22 2016

Blockedon: 596970

Comment 4 by st...@chromium.org, Apr 22 2016

Components: Tools>Test>FindIt

Comment 5 by ojan@chromium.org, Mar 7 2017

Cc: -ojan@chromium.org

Comment 6 by st...@chromium.org, Apr 25 2017

Labels: -Findit
Status: Verified (was: Assigned)
This was done. The way we collected failed "target"s is to parse the stdout of the compile step.

Sign in to add a comment