New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 724686 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner: ----
Closed: May 2017
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug

Blocked on:
issue 726358



Sign in to add a comment

Don't build the translation_unit clang tool on codesearch-gen-chromium-linux each time

Project Member Reported by h...@chromium.org, May 19 2017

Issue description

The buidlbot builds the translation_unit clang tool, along with all of clang, on each build step, which is a lot of wasted time.

Either it should do incremental builds, or perhaps even better download a prebuilt binary of the tool.
 

Comment 2 by h...@chromium.org, May 19 2017

Taking a stab at this: https://codereview.chromium.org/2896663002

Comment 3 by dcheng@chromium.org, May 19 2017

Cc: dcheng@chromium.org
FWIW, I know update.py isn't really meant for development--it's more for packaging clang builds--but in practice, it's quite useful for iterating on the clang plugins. People who hack on the plugins have mostly been doing ninja -C third_party/llvm-build/Release+Asserts, but it would be kind of nice if update.py had an incremental mode.

Also, are builders for a given bot dedicated? Or can they be shared among several different builders?

Comment 4 by h...@chromium.org, May 19 2017

> it would be kind of nice if update.py had an incremental mode.

It used to be incremental actually. It seems this buildbot just nukes the build directory each time. And it seems also our script has become non-incremental(?), maybe because we clear the cmake cache too hard.

> Also, are builders for a given bot dedicated? Or can they be shared among several different builders?

If I understand the question correctly, no machines don't need to be dedicated to a specific buildbot. Machines can be shared by multiple buildbots.

In this case it seems that the buildbot is using a dedicated VM, though.

Comment 5 by emso@chromium.org, May 22 2017

Our goal is to have a dedicated VM per configuration (linux, android, windows, and so on). Eventually, we'll likely have more than one slave per config as we try to speed up the generation of Xrefs.
Project Member

Comment 6 by bugdroid1@chromium.org, May 25 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2aa36cd3f1ae44ee75d549304bdf568798528917

commit 2aa36cd3f1ae44ee75d549304bdf568798528917
Author: hans <hans@chromium.org>
Date: Thu May 25 05:58:00 2017

Package the translation_unit tool as part of building the clang package.

It will go in a separate tarball and be consumed by the codesearch buildbot.

Getting built by default in update.py also means our ToT bots will catch any
breakages due to upstream changes in clang.

BUG= 724686 

Review-Url: https://codereview.chromium.org/2896663002
Cr-Commit-Position: refs/heads/master@{#474576}

[add] https://crrev.com/2aa36cd3f1ae44ee75d549304bdf568798528917/build/download_translation_unit_tool.py
[modify] https://crrev.com/2aa36cd3f1ae44ee75d549304bdf568798528917/tools/clang/scripts/package.py
[modify] https://crrev.com/2aa36cd3f1ae44ee75d549304bdf568798528917/tools/clang/scripts/update.py
[modify] https://crrev.com/2aa36cd3f1ae44ee75d549304bdf568798528917/tools/clang/translation_unit/TranslationUnitGenerator.cpp

Comment 7 by h...@chromium.org, May 26 2017

Blockedon: 726358

Comment 8 by h...@chromium.org, May 26 2017

Status: Started (was: Available)
Patch for the buildbots: https://chromium-review.googlesource.com/c/517262/
Project Member

Comment 9 by bugdroid1@chromium.org, May 30 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/a3e30c591fbf8c2dafef2ecfe04bbb109a4df516

commit a3e30c591fbf8c2dafef2ecfe04bbb109a4df516
Author: Hans Wennborg <hans@chromium.org>
Date: Tue May 30 16:32:49 2017

Use a pre-built package of the translation_unit tool for codesearch

Also rename the function to reflect that it both downloads and runs the tool.

Bug:  724686 
Change-Id: I81aba1868860c9815c0fb9ee84d615a6ddfb6a9b
Reviewed-on: https://chromium-review.googlesource.com/517262
Reviewed-by: Shenghua Zhang <shenghuazhang@google.com>
Reviewed-by: Emma Söderberg <emso@chromium.org>
Commit-Queue: Hans Wennborg <hans@chromium.org>

[modify] https://crrev.com/a3e30c591fbf8c2dafef2ecfe04bbb109a4df516/scripts/slave/recipe_modules/codesearch/api.py
[modify] https://crrev.com/a3e30c591fbf8c2dafef2ecfe04bbb109a4df516/scripts/slave/recipe_modules/codesearch/example.expected/codesearch_gen_chromium_linux_test_basic.json
[rename] https://crrev.com/a3e30c591fbf8c2dafef2ecfe04bbb109a4df516/scripts/slave/recipe_modules/codesearch/tests/run_clang_tool.expected/run_translation_unit_clang_tool_failed.json
[rename] https://crrev.com/a3e30c591fbf8c2dafef2ecfe04bbb109a4df516/scripts/slave/recipe_modules/codesearch/tests/run_clang_tool.py
[modify] https://crrev.com/a3e30c591fbf8c2dafef2ecfe04bbb109a4df516/scripts/slave/recipes/chromium_codesearch.py
[modify] https://crrev.com/a3e30c591fbf8c2dafef2ecfe04bbb109a4df516/scripts/slave/recipes/chromium_codesearch.expected/full_codesearch_gen_chromium_android.json
[modify] https://crrev.com/a3e30c591fbf8c2dafef2ecfe04bbb109a4df516/scripts/slave/recipe_modules/codesearch/example.py
[rename] https://crrev.com/a3e30c591fbf8c2dafef2ecfe04bbb109a4df516/scripts/slave/recipe_modules/codesearch/tests/run_clang_tool.expected/basic.json
[modify] https://crrev.com/a3e30c591fbf8c2dafef2ecfe04bbb109a4df516/scripts/slave/recipes/chromium_codesearch.expected/full_codesearch_gen_chromium_linux.json
[modify] https://crrev.com/a3e30c591fbf8c2dafef2ecfe04bbb109a4df516/scripts/slave/recipes/chromium_codesearch.expected/full_codesearch_gen_chromium_chromiumos_fail.json
[modify] https://crrev.com/a3e30c591fbf8c2dafef2ecfe04bbb109a4df516/scripts/slave/recipes/chromium_codesearch.expected/full_codesearch_gen_chromium_chromiumos.json

Project Member

Comment 10 by bugdroid1@chromium.org, May 30 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/a3e30c591fbf8c2dafef2ecfe04bbb109a4df516

commit a3e30c591fbf8c2dafef2ecfe04bbb109a4df516
Author: Hans Wennborg <hans@chromium.org>
Date: Tue May 30 16:32:49 2017

Use a pre-built package of the translation_unit tool for codesearch

Also rename the function to reflect that it both downloads and runs the tool.

Bug:  724686 
Change-Id: I81aba1868860c9815c0fb9ee84d615a6ddfb6a9b
Reviewed-on: https://chromium-review.googlesource.com/517262
Reviewed-by: Shenghua Zhang <shenghuazhang@google.com>
Reviewed-by: Emma Söderberg <emso@chromium.org>
Commit-Queue: Hans Wennborg <hans@chromium.org>

[modify] https://crrev.com/a3e30c591fbf8c2dafef2ecfe04bbb109a4df516/scripts/slave/recipe_modules/codesearch/api.py
[modify] https://crrev.com/a3e30c591fbf8c2dafef2ecfe04bbb109a4df516/scripts/slave/recipe_modules/codesearch/example.expected/codesearch_gen_chromium_linux_test_basic.json
[rename] https://crrev.com/a3e30c591fbf8c2dafef2ecfe04bbb109a4df516/scripts/slave/recipe_modules/codesearch/tests/run_clang_tool.expected/run_translation_unit_clang_tool_failed.json
[rename] https://crrev.com/a3e30c591fbf8c2dafef2ecfe04bbb109a4df516/scripts/slave/recipe_modules/codesearch/tests/run_clang_tool.py
[modify] https://crrev.com/a3e30c591fbf8c2dafef2ecfe04bbb109a4df516/scripts/slave/recipes/chromium_codesearch.py
[modify] https://crrev.com/a3e30c591fbf8c2dafef2ecfe04bbb109a4df516/scripts/slave/recipes/chromium_codesearch.expected/full_codesearch_gen_chromium_android.json
[modify] https://crrev.com/a3e30c591fbf8c2dafef2ecfe04bbb109a4df516/scripts/slave/recipe_modules/codesearch/example.py
[rename] https://crrev.com/a3e30c591fbf8c2dafef2ecfe04bbb109a4df516/scripts/slave/recipe_modules/codesearch/tests/run_clang_tool.expected/basic.json
[modify] https://crrev.com/a3e30c591fbf8c2dafef2ecfe04bbb109a4df516/scripts/slave/recipes/chromium_codesearch.expected/full_codesearch_gen_chromium_linux.json
[modify] https://crrev.com/a3e30c591fbf8c2dafef2ecfe04bbb109a4df516/scripts/slave/recipes/chromium_codesearch.expected/full_codesearch_gen_chromium_chromiumos_fail.json
[modify] https://crrev.com/a3e30c591fbf8c2dafef2ecfe04bbb109a4df516/scripts/slave/recipes/chromium_codesearch.expected/full_codesearch_gen_chromium_chromiumos.json

Comment 11 by h...@chromium.org, May 30 2017

The first build is here:
https://build.chromium.org/p/chromium.infra.codesearch/builders/codesearch-gen-chromium-linux/builds/675

It failed because I had forgotten to copy the package from the staging bucket. I've fixed that now and will update the copying instructions.

Let's see how the next one goes.
Project Member

Comment 12 by bugdroid1@chromium.org, May 30 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6c96037e35179fdafa339ee44411c018702d82eb

commit 6c96037e35179fdafa339ee44411c018702d82eb
Author: hans <hans@chromium.org>
Date: Tue May 30 20:17:45 2017

Update the updating_clang document to include copying the translation_unit tool

BUG= 724686 

Review-Url: https://codereview.chromium.org/2909373002
Cr-Commit-Position: refs/heads/master@{#475651}

[modify] https://crrev.com/6c96037e35179fdafa339ee44411c018702d82eb/docs/updating_clang.md

Comment 13 by h...@chromium.org, May 31 2017

Status: Fixed (was: Started)
Bot is happy and ~30 minutes faster:
https://build.chromium.org/p/chromium.infra.codesearch/builders/codesearch-gen-chromium-linux/builds/676


Comment 14 by emso@chromium.org, May 31 2017

Super awesome! Thanks so much :)

Sign in to add a comment