Speed up generation of Xrefs |
|||
Issue descriptionSince we switched to show files directly grabbed from Git we are continuously seeing out-of-sync Xrefs. The reason for this is that Xrefs are computed with the old setup using a buildbot builder. This builder runs every 4 hours, creating and uploading an Xrefs index pack. This index pack is picked up by the Xrefs indexing pipeline which checks for updates every 4 hours and then takes about 2 hours. In between the builder and the Xrefs pipeline, there is a job moving the index pack from a bucket to internal storage, adding some additional time. We are seeing delays of up to 8-9 hours with the current setup. We should gather data on how these different pieces interact with the current scheduling and then investigate where we can speed things up. For instance, can we run builders in parallel to provide a new index pack more often, or/and can we increase the frequency of the Xrefs indexing pipeline.
,
Mar 2 2017
,
Mar 2 2017
Is https://crbug.com/697627 caused by out of date xrefs, or incorrect xrefs? The code in that bug didn't change recently.
,
Mar 3 2017
Sorry that merge was a mistake. Should be fixed now.
,
Mar 21 2017
Issue 703499 has been merged into this issue.
,
May 11 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/590e609111eaaa9c7fd94d919b090e36c1fa9151 commit 590e609111eaaa9c7fd94d919b090e36c1fa9151 Author: Emma Soederberg <emso@chromium.org> Date: Thu May 11 10:45:57 2017 Adds more slaves to the codesearch master Puts the current gen bots for Chromium/ChromeOS and Chromium/Android on separate slaves to reduce the latency of Xrefs. Adding a comment about the windows slave to use for Chromium/Windows. Bug:697786 Change-Id: Ie440485224a7e0d42e8ecd5533f5d228557cddad Reviewed-on: https://chromium-review.googlesource.com/502932 Reviewed-by: Scott Lee <ddoman@google.com> Commit-Queue: Emma Söderberg <emso@chromium.org> [modify] https://crrev.com/590e609111eaaa9c7fd94d919b090e36c1fa9151/masters/master.chromium.infra.codesearch/slaves.cfg
,
May 12 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/0e4aca1b687f97b725e8123c8c9708aa3b0aba48 commit 0e4aca1b687f97b725e8123c8c9708aa3b0aba48 Author: Emma Soederberg <emso@chromium.org> Date: Fri May 12 07:46:35 2017 Removes unnecessary steps from the chromium codesearch recipe The uploaded files are not used anymore. The grok index pack has been replaced by the kythe index pack, and the source tar ball has been replaced with indexing directly from the repository. Bug:697786 Change-Id: I87da2aaebc49e832271ac6ef4b20e4bbf5f914f5 Reviewed-on: https://chromium-review.googlesource.com/503601 Commit-Queue: Emma Söderberg <emso@chromium.org> Reviewed-by: Henrik Kjellander <kjellander@chromium.org> [modify] https://crrev.com/0e4aca1b687f97b725e8123c8c9708aa3b0aba48/scripts/slave/recipes/chromium_codesearch.expected/full_codesearch_gen_chromium_linux.json [modify] https://crrev.com/0e4aca1b687f97b725e8123c8c9708aa3b0aba48/scripts/slave/recipes/chromium_codesearch.py [modify] https://crrev.com/0e4aca1b687f97b725e8123c8c9708aa3b0aba48/scripts/slave/recipes/chromium_codesearch.expected/full_codesearch_gen_chromium_chromiumos_fail.json [modify] https://crrev.com/0e4aca1b687f97b725e8123c8c9708aa3b0aba48/scripts/slave/recipes/chromium_codesearch.expected/full_codesearch_gen_chromium_android.json [modify] https://crrev.com/0e4aca1b687f97b725e8123c8c9708aa3b0aba48/scripts/slave/recipes/chromium_codesearch.expected/full_codesearch_gen_chromium_chromiumos.json
,
May 12 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/master-manager/+/7c330b8e3c0578a9c7a40180c56279726219260b commit 7c330b8e3c0578a9c7a40180c56279726219260b Author: emso <emso@google.com> Date: Fri May 12 13:26:14 2017
,
May 17 2017
,
Nov 2 2017
|
|||
►
Sign in to add a comment |
|||
Comment 1 by emso@chromium.org
, Mar 2 2017