New issue
Advanced search Search tips

Issue 921353 link

Starred by 1 user
Status: Assigned
Owner:
crouleau@chromium.org
Cc:
eyaich@chromium.org
jbudorick@chromium.org
jmad...@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

gtest_perf_tests all run against the same bot

Project Member Reported by crouleau@chromium.org, Jan 12 
From https://bugs.chromium.org/p/chromium/issues/detail?id=921039#c8 :

> Looking at the test shards, performance_test_suite uses all five shards including build202-m7 while the gtest perf tests all compete for build202-m7.


This is a regression in how perf tests do sharding. It used to be that we would hand-select specific shards to run gtest_perf_tests on, and in this way we would have the gtest perf tests not all run against the same bot. I think this was regressed by https://chromium-review.googlesource.com/c/chromium/src/+/1106549/ . OBBS I think stands for "One Build Bot Step". That changelist did also fix a lot of things though, so it's not like we can go back to that earlier state.

+Emily, do you have any suggestions on how we can get the gtest_pef_tests to not all run on the same bot?

Comment 1 by eyaich@chromium.org, Jan 14 

So all the shard mapping is managed in https://cs.chromium.org/chromium/src/testing/trigger_scripts/perf_device_trigger.py?q=perf_device_trigger&sq=package:chromium&g=0&l=1.  

Basically this test exists in its own isolate and shards=1.  Therefore, every isolate that indicates shards=1 will choose the same bot to run on. This hasn't been a problem since the non-telemetry tests have been negligible compared to the performance tests, but the problem here is just that they never get a chance to run due to the the logic to run tests with more shards first.

I don't think the solution of spreading out the gtests more will really solve anything since it wounds like they would get starved by the performance_test_suite either way.  I think this problem will be solved with capacity and device planning.

Our current sharding design moves away from the test indicating what shard it wants to run on, so I don't like the idea of sending in a flag to determine the shard. 

A hack would be to create a shard map for the gtests similar to that for performance_test_suite, but the semantics don't line up there since gtests are all separate vs the benchmark suite being all connected.
►

Sign in to add a comment