New issue
Advanced search Search tips

Issue 868114 link

Starred by 1 user

Issue metadata

Status: Started
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

Look into increasing sharding for slow tests [webkit_layout_tests, browser_tests, etc.] on the CQ.

Project Member Reported by erikc...@chromium.org, Jul 26

Issue description

Question: Can we increase the size of the fleet and/or increase the sharding parameter for slow tests to speed up the CQ? 

It's possible we're limited by non-parallelizable tasks [e.g. downloading isolate, startup/shutdown, etc.]. 
 
Cc: tikuta@chromium.org
tikuta@ has spent a bunch of time over the last couple quarters knocking down the longest poles. Talk to him for more.
Cc: jbudorick@chromium.org
Components: Infra>Client>Chrome
We do so on an as needed basis. It's a simple tuning parameter. Here's two recent examples:
https://chromium-review.googlesource.com/1143435
https://chromium-review.googlesource.com/1133278

The sharding level is definitely bound by the amount of overhead, which varies from test to test and across platforms.
Owner: tikuta@chromium.org
Status: Started (was: Assigned)
I thought I adjusted almost all tests not to run more than around 15 mins in
https://chromium-review.googlesource.com/c/chromium/src/+/1104277

But we can/want to decrease that to 10 mins or less?
http://shortn/_nZfp7PmMy8

I think there is some improvable space yet by simply increasing the number of shards.
If we can use more shards, let me increase the shard num for some slowest tests.

It's a pure function of tolerable total fleet overhead. For example increasing from 10 to 20 shards is significantly more costly in term of total fleet overhead than 1 to 2.

I'm working (sigh) on getting better data on this. John has the last word here.
Project Member

Comment 5 by bugdroid1@chromium.org, Aug 7

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/eb6d997be97d8267af540deb57ce2763f77f5460

commit eb6d997be97d8267af540deb57ce2763f77f5460
Author: Takuto Ikuta <tikuta@chromium.org>
Date: Tue Aug 07 00:46:43 2018

Increase shards for surface_sync_content_browsertests

This is for slow surface_sync_content_browsertests step like belows.
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/android-marshmallow-arm64-rel/54921
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/android-marshmallow-arm64-rel/54740
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/android-marshmallow-arm64-rel/54965

Overhead of the test is small, 6 seconds overhead vs 35 mins runtime.
https://chromium-swarm.appspot.com/task?id=3f2773404bdad410&refresh=10&show_raw=1

So overhead increase is negligible.

Bug: 868114
Change-Id: Iab6b396fcdf5a25ba50f299a0fa5e294ccea32ae
Reviewed-on: https://chromium-review.googlesource.com/1163642
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#581069}
[modify] https://crrev.com/eb6d997be97d8267af540deb57ce2763f77f5460/testing/buildbot/chromium.android.fyi.json
[modify] https://crrev.com/eb6d997be97d8267af540deb57ce2763f77f5460/testing/buildbot/chromium.android.json
[modify] https://crrev.com/eb6d997be97d8267af540deb57ce2763f77f5460/testing/buildbot/chromium.clang.json
[modify] https://crrev.com/eb6d997be97d8267af540deb57ce2763f77f5460/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/eb6d997be97d8267af540deb57ce2763f77f5460/testing/buildbot/chromium.memory.json
[modify] https://crrev.com/eb6d997be97d8267af540deb57ce2763f77f5460/testing/buildbot/test_suites.pyl

Project Member

Comment 7 by bugdroid1@chromium.org, Sep 5

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ea000e94b95e786748b5b6b860b2fc320ecca182

commit ea000e94b95e786748b5b6b860b2fc320ecca182
Author: Takuto Ikuta <tikuta@chromium.org>
Date: Wed Sep 05 04:31:54 2018

Increase shard for browser_tests on win

Currently browser_tests on win7_chromium_rel_ng sometimes took more than 10 mins.
Let me increase the number of shards for such tests.

e.g
753s https://ci.chromium.org/b/8936390613165886176
732s https://ci.chromium.org/b/8936406831776876544
706s https://ci.chromium.org/b/8936396862731996176
704s https://ci.chromium.org/b/8936363432077079248

Bug: 868114
Change-Id: I9a7eb07ce2c58f5b03e5f39152dba7ac6101f11b
Reviewed-on: https://chromium-review.googlesource.com/1203519
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#588764}
[modify] https://crrev.com/ea000e94b95e786748b5b6b860b2fc320ecca182/testing/buildbot/chromium.win.json
[modify] https://crrev.com/ea000e94b95e786748b5b6b860b2fc320ecca182/testing/buildbot/test_suite_exceptions.pyl

Project Member

Comment 8 by bugdroid1@chromium.org, Jan 9

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/f060055de7b1b7e537b5e490fd757ed98412c6a7

commit f060055de7b1b7e537b5e490fd757ed98412c6a7
Author: Takuto Ikuta <tikuta@google.com>
Date: Wed Jan 09 23:05:11 2019

Project Member

Comment 9 by bugdroid1@chromium.org, Jan 10

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c9c269a29ff911f91920ff108010a41f0b64e72b

commit c9c269a29ff911f91920ff108010a41f0b64e72b
Author: Takuto Ikuta <tikuta@chromium.org>
Date: Thu Jan 10 22:40:25 2019

Increase shards for slow tests of win10_chromium_x64_rel_ng builder

I'd like to increase shards that has more than 20 mins execution time in win10_chromium_x64_rel_ng builder.
Slow tests are detected from below query
http://shortn/_39HQP6rTpb

AFAIK, win10_chromium_x64_rel_ng test does not have large per shard overhead.
And current win10 capacity seems to able to handle this.
http://shortn/_caTg5Sxv0F

Bug: 868114
Change-Id: Ib0dbeefac8e7fe12ea3ef7fd7ca49c1de4f34598
Reviewed-on: https://chromium-review.googlesource.com/c/1391038
Reviewed-by: John Budorick <jbudorick@chromium.org>
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Cr-Commit-Position: refs/heads/master@{#621783}
[modify] https://crrev.com/c9c269a29ff911f91920ff108010a41f0b64e72b/testing/buildbot/chromium.win.json
[modify] https://crrev.com/c9c269a29ff911f91920ff108010a41f0b64e72b/testing/buildbot/test_suite_exceptions.pyl

Sign in to add a comment