New issue
Advanced search Search tips

Issue 614705 link

Starred by 1 user
Status: Fixed
Owner:
serg...@chromium.org
Closed: May 2016
Cc:
jam@chromium.org
ojan@chromium.org
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

webkit_tests (with patch) flakily crash thousands of tests causing chromium-try-flakes fail to record them

Project Member Reported by serg...@chromium.org, May 25 2016 
In https://codereview.chromium.org/1974833002#ps160001, build https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/226881, webkit_tests (with patch) has crashed 14492 tests and then succeeded on the same CL in https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/227006. From the chromium-try-flakes perspective this is 14492 flaked tests. This is so many, that we've failing to record them in datastore and chromium-try-flakes is sending alerts.

This looks like the webkit_tests (with patch) is incorrectly crashing thousands of tests. Ojan or Jam, can you please file a separate bug for this and assign to an appropriate owner?

OTH, I feel we also need to add a reasonable limit to the number of failed tests from a step and report whole-step failure if this number is exceeded. I'll set this number based on Dremel analysis of typical number of flaky tests per step multiplied by two.

Comment 1 by serg...@chromium.org, May 25 2016 

After querying some data from Dremel, I've come to a conclusion that a lot of tests report very large number of failures: more than 50% report more than 1000 failures. This was unexpected as I was hoping to cap the number of tests with a number under 100 and cover majority of cases.

Script for analysis: https://plx.corp.google.com/script/#a=qo%7Ci=google%253A%253Ascript_77._402d20_52c8_4424_b4c0_fb3209bd1bb2.

The only idea that I have for now is that many runs fail all tests therefore leading to large numbers. I'll continue to investigate.

Comment 2 by serg...@chromium.org, May 25 2016 

Based on the distribution it looks like numbers of failed results are clustered around some values. This seems to confirm my theory that these values are the total number of tests in various test suites, thus all tests fail.

I was considering to look at the ratio of the failed tests compared to the total number of executed tests, but then we still need to cap the absolute number of failed tests in a step to stop datastore errors from happening. Therefore, I'll just take 50, which is also the maximum number of bugs we can file per day.

Comment 4 by serg...@chromium.org, May 25 2016

Labels: -Pri-0 Pri-1
This should have fixed chromium-try-flakes. I'll keep monitoring it for the next hour or so.

Comment 5 by serg...@chromium.org, May 27 2016

Status: Fixed (was: Assigned)
►

Sign in to add a comment