kojii@

It seems like you worked on this file at some point. Could you take a look at it and assign it to the right owner?

I just fixed writing-modes related property of all tests.

schenney@, do you have any ideas?

I don't understand what the issue is. The test fails "with patch" and does not have any history of failure in the webkit_tests set of flakiness metrics.

All of the linked flaky builds show a large number (almost 200) failing tests, so I very much doubt that this is the specific test causing problems. It seems like something far more systemic. Adding infra label to get an opinion on whether this is actionable, because I think it is WontFix.

We do not use Sheriff-Infra. Infra>Flakiness is the right component for flakiness infra issues (but not for flaky tests).

What makes you think that this is an infra failure though? Couldn't it be that someone has introduced a bug that caused hundreds of tests to be flaky? From the logs (https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/builds/323351/steps/webkit_tests%20%28with%20patch%29/logs/stdio) I see the following error messages:

20:57:03.022 31006 worker/5 fast/backgrounds/border-radius-split-background.html crashed, (stderr lines):
20:57:03.023 31006   [33583:1287:1026/205702:5285792177886:WARNING:vt_video_decode_accelerator_mac.cc(174)] Failed to create VTDecompressionSession: Error Domain=NSOSStatusErrorDomain Code=-8973 "The operation couldn???t be completed. (OSStatus error -8973.)" (codecOpenErr) (-8973)
20:57:03.023 31006   [33583:1287:1026/205702:5285793897711:WARNING:vt_video_decode_accelerator_mac.cc(209)] Failed to create hardware VideoToolbox session
20:57:03.023 31006   [33583:1287:1026/205702:5285883936236:ERROR:vt_video_encode_accelerator_mac.cc(540)]  VTCompressionSessionCreate failed: -12908
20:57:03.023 31006   [33584:1287:1026/205702:5286177549907:INFO:SkScan_AAAPath.cpp(1048)] frac_y = 24.250000

So it appears the renderer is flaky here and crashes occasionally. I am also surprised that the test was not retried after a CRASH: https://test-results.appspot.com/testfile?builder=mac_chromium_rel_ng&name=full_results.json&master=tryserver.chromium.mac&testtype=webkit_tests%20(with%20patch)&buildnumber=323351.

Shouldn't it be retried 4 times? Or do you have different retry logic for CRASH outcomes?

What I'm primarily interested in is the expected action on a bug like this, and follow up issues.

I don't know what the retry logic is for crashes.

It seems that this bug is WontFix because it's just one of many tests that failed together, in a transient manner, for reasons that seem entirely unrelated to the test itself. I looked at the list of tests that failed and saw no particular clustering of functionality, suggesting it was a bot problem of some kind. Hence involving infra.

I also think it's an error for the chromium-try-flakes app to be reporting a bug for every test that failed when a large number of tests all fail together. Maybe that cannot be fixed, but I would like it to be considered.

I have no idea on who to CC here.

Thanks for looking into this. I appreciate the discussion that you started.

IMHO, an action on a bug like this is to investigate the failure, try to understand why it happens and find a way to address the issue. You've concluded that this is an infra failure and I've looked into it. In particular I'm surprised that CRASHES are not retried 4 times like other failures, so the next action would be to figure out why this happens and whether we can change that.

I am also open to suggestions on how to improve chromium-try-flakes app, but I don't think that ignoring cases when many tests fail together is the right action even if it's a bot problem. Failures like these still impact developers and should be addressed by someone. One improvement that we can do however is to report all tests that failed together in a single bug rather than filing individual bugs for each test, see issue 576274 for this.

Whom to CC? Since this is about test launcher logic (retrying CRASHes), I'd recommend to ask wider team on blink-dev@. I've also CC'd ojan@ on this bug since he seems to be the right person to start such discussion. I'd also CC someone who knows renderer code to try to figure out why the renderer has crashed, but I don't know the right people on Blink either, so perhaps that is another question that should be asked on blink-dev@.

Finally, I think the underlying issue here is that tests launchers are not owned by anyone in particular and thus people keep bouncing bugs around without anyone willing to fix it. Ojan, do you know if crossover team is supposed to be working on improving test launchers or we still have no owners for them?

Re: retrying crashes ...

Normally crashes are retried.

In this case the test step as a whole exited early because we had too many crashes and timeouts (100 in total), and we don't bother to retry things in that case because the assumption is that the build is just broken. Hence the INVALID_TEST_RESULTS result.

As to diagnosing and clustering the flakes better, there is certainly ongoing work in that area (+tandrii@).

It would be interesting to know if these crashes showed up on other runs from other patches, or on the main waterfalls. I would hope nothing this broken actually got to trunk.

As an aside, it seems like something in skia is doing a crazy amount of logging to stderr. I hope that isn't still happening, because it would certainly be putting a lot of extra load on the system.

That's interesting. We used to print "TEST RESULTS WERE INVALID" in the step text when the results were invalid and file bugs for entire step in this case. Did this change recently or was it never working for webkit_tests?

Ah. Actually, I see that it's printed in the summary step, which we ignore since it's a duplicating the failure in the (with patch) step. Looks like this is a bug in chromium-try-flakes. Filed issue 660810.