Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Aye, I agree that I should prioritize investigating this -- I'll plan to start next week, and I'll update this bug with any important findings :-)

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 5 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 6 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 4 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Update: No progress on this in the last week, though I will continue to keep it assigned to me for now.

Possible next step: use remote desktop to try to reproduce the error on one of the buildslaves of https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng.

Detected 6 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 8 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 8 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 5 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 5 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 5 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 6 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 7 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 9 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 7 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 11 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 13 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

I'm upping the priority of this and tracking it as part of my work on burning down infra failures. 

qyearsley -- have you been able to make any progress? 

It looks like the master is having to kill the step after it hits the global timeout:

command timed out: 3600 seconds without output, attempting to kill
program finished with exit code 1
elapsedTime=7502.778000

qyearsley, as part of your work, can you also add a sane timeout to the step in the recipe?

@katthomas - the problem isn't that there's no sane timeout. The problem is that there's a hung process somewhere in the process tree that is keeping the step from exiting, and so we'll hit the timeout no matter what it is.

@qyearsley - Please figure out if you can look at this or if we need to find someone else. Six weeks with no update is not so good ...

@dpranke -- True, but I believe a step-defined timeout would result in this step being marked as red rather than purple, which would be a better signal. Is that correct?

I haven't been looking at this at all, and I suspect someone else will be able to solve this much faster than me; the reason why I haven't set the status as "available" is that it then gets put on the sheriff's triage list.

Would anybody who is CC'd on this bug be interested in investigating this?

This is a long standing issue (> 1 year), which is probably the same as  bug 522396  -- so it's important, but I haven't really been sure how best to start investigating :-/

In the meantime, does it still make sense to add a timeout so that the Windows bot fails faster if it's hung?

Yes, I think it's some variant of  bug 522396 , and we probably need to apply some of the same techniques to see if we can figure out what's going on.

@katthomas - yes, that might at least shift things to red and make it clearer that this is a dev-side issue. 

@qyearsley - I don't think "fail faster" is all that useful.

maybe we should schedule a meeting with some of the people that worked on 522396 to brainstorm ideas and find a victi^H^H^H^H^H^Howner.

This most certainly sounds like  bug 522396  is back.

The problem was that using subprocess on Windows while processing stderr/stdout is **really** hard to get right (mostly due to peculiar Windows behaviour). It is full of gotcha's which only show up randomly. 

For example, using threads is pretty much 100% unsafe, the following code will randomly lock up on Windows;
----------------------------
 p = subprocess.Popen(...., stdout=subprocess.PIPE)
 
 def reader(pipe):
   while True:
     data = pipe.read(x)
     if not data:
       break

 t1 = threading.Thread(run=reader, args=(p.stdout))

 t1.join()
----------------------------

The solution to this problem is;
 * Audit / reduce / refactor usage of subprocess throughout the whole Chrome Infra stack.
    - webkitpy bug - https://bugs.chromium.org/p/chromium/issues/detail?id=610613
 * Remove as many levels of wrapper scripts as possible. Every wrapper script adds another place for the problem to show up.

None of this is simple or easy to solve quickly. 

(BTW None of this is actually unique to Python - these problems are likely to turn up in any Windows command line program which tries to mix threading and stdout/stderr without a huge amount of care.)

Detected 6 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

@qyearsley, are you able to take this on?

Detected 4 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 4 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Re #113: I don't think I'll be able to take this on in the short term (see comment #109) - partly because I'm not really sure how to investigate this.

In #110, dpranke@ said that one possible next step would be to schedule a meeting to brainstorm ideas and review what was done before when investigating  bug 522396 . I just  scheduled a meeting for Monday afternoon next week.

In #111, tansell@ said that some issues are caused by using subprocess on Windows. Maybe one possible approach would be to focus on bug 610613 while keeping track of the flakiness level of the windows try bots?

katthomas@, does it make sense to assign this to you now to indicate that I'm not currently working on it? :-/ And hopefully next week can decide what to try next?

No, katthomas@ shouldn't work on this (unless she really wants to). Someone in blink-infra should own this. 

Yep, I wasn't trying to say that katthomas@ should work on this now. jparent@ suggested in an email thread that I might assign to katthomas@ to indicate whether I'm currently working on it. Now reset owner so this bug doesn't go back in the sheriff queue.

Should we disable this test in the meantime?

Thanks for scheduling that meeting--looking forward to hearing what comes out of it! #111 suggests some approaches you might want to discuss as well.

You mean disable the entire test suite (step)? 

That would cause us to lose a lot of coverage, so I would prefer not to do that. 

From what I can tell, the flake rate is low enough (only a couple of times a day or less) that I don't think we need to do that yet.

Detected 3 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

My tryjobs for https://codereview.chromium.org/2547813002/ (which is trying to fix other test flakiness) just failed two tryjobs back-to-back in webkit_tests. It looks like the execution of wdiff.exe hung, and was aborted by the recipe engine when it timed out. The two runs are here:

https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/342767
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/342695

The logs are attached. The end of one is:

05:11:07.673 1856 "E:\b\c\b\win\src\third_party\cygwin\bin\wdiff.exe --start-delete=##WDIFF_DEL## --end-delete=##WDIFF_END## --start-insert=##WDIFF_ADD## --end-insert=##WDIFF_END## E:\b\rr\tmprmdml5\w\layout-test-results\virtual/mojo-loading/http/tests/local/blob/send-data-blob-expected.txt E:\b\rr\tmprmdml5\w\layout-test-results\virtual/mojo-loading/http/tests/local/blob/send-data-blob-actual.txt" took 4713.74s


--------------------------------------------------------------------------------
started: Fri Dec  2 03:35:50 2016
ended: Fri Dec  2 05:11:07 2016
duration: 1 hrs, 35 mins, 17 secs
status: FAILURE
status reason: return code was 1.


I thought maruel's subprocess42.py was supposed to solve problems like this. Can the webkit test runner be modified to use it?

I would upgrade this to P0 but it isn't worth raising an alarm over a weekend. Next Monday, please treat this as P0, and upgrade as necessary. This urgently needs to be fixed.

Deleted: webkit-tests-logs.zip 2.1 MB

webkit-tests-logs.zip 2.1 MB Download

I suggested assigning to Katie to re-triage when qyearsley@ said he would not be able to work on it.  A P1 (or P0 as kbr suggests), needs to be treated accordingly, and having an owner means it is slated for work in the near term. Keeping a bug owned, and high priority, but without a work plan, is not proper bug hygiene.

qyearsley@, jeffcarp@, tansell@, djd@, mcgreevy@ - y'all are "blink infra", we will need one of you to own this.  Update this bug after your meeting next week with an outline of next steps and clear owner.  If no one feels able to own, please assign back to me as owner so I can escalate.

Detected 9 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 4 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Another instance of the test harness hanging at the end of the run:
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/343629

Thanks for posting --

Today I started a doc for notes while investigating this bug:
https://docs.google.com/document/d/127hqUX_EeCpeIiUyNRVYEBRGdZgq9N-xRgANWvnZ0FE/edit

So, interesting thing ...

Historically, this step, when it hung, hung at the very end of the test run, when we were shutting everything down.

From the few logs I looked at on Thursday and Friday, we're hanging elsewhere, apparently often in a call out to wdiff.exe. So this doesn't look like the old subprocess hang (at least some times).

We should look at however many logs we can from recent failures to see if they all seem wdiff-related or if there are other failures as well. We should also try to log into the machines while a test run is hanging (before it is killed), to see if we see any error dialogs, capture process trees, etc.

Snapshot of procexp taken from https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/344178

Looks like wdiff is hanging around for too long.

Deleted: Screen Shot 2016-12-05 at 4.50.19 PM.png 91.1 KB

	Screen Shot 2016-12-05 at 4.50.19 PM.png 91.1 KB View Download

So I killed wdiff and it looks like the rest of the test started moving again.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/88fb1e82a1866e975526c93ca86482ee4e602547

commit 88fb1e82a1866e975526c93ca86482ee4e602547
Author: qyearsley <qyearsley@chromium.org>
Date: Tue Dec 06 03:44:03 2016

Disable wdiff on Windows.

The purpose of this is to see how many of the Windows hanging instances
are related to wdiff, and also perhaps separate the wdiff-related hangs
from any others.

BUG= 637478 

Review-Url: https://codereview.chromium.org/2548303003
Cr-Commit-Position: refs/heads/master@{#436512}

[modify] https://crrev.com/88fb1e82a1866e975526c93ca86482ee4e602547/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/win.py

Detected 29 new flakes for test/step "webkit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJAsSBUZsYWtlIhl3ZWJraXRfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

29 new flakes seems like a lot, but most of them are actually flakes on mac and linux that are not the same as this bug. According to the queries in the (google.com-only) doc in #127, it looks like there were no hangs today (Dec 6 PST) after wdiff was disabled on win.

Also: Didn't successfully SSH to one of the hosts today, will try tomorrow.

(apparently repeating what qyearsley@ just wrote ...)

The clustering of flakes leaves much to be desired here. 

I can't easily tell what the "29 new flakes" are, since the link in #132 points to a page w/ the 50 most recent failures.

That, said, it looks like the most recent failures on *windows* as of the time of me writing this are 

http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/344351
http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/344259

both of which are from before when we disabled wdiff. 

The newer failures:

http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/builds/348398 (try run at 2016-12-06 05:19:16 UTC)

to

http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/350468 (try run at 2016-12-06 07:31:54 UTC)

all appear to be a different issue (with a bunch of tests failing). 

It looks like the mac/linux failures were caused by r436518 (which, being a "reland", landed without try jobs), which was later reverted in r436542. So, these aren't really "flakes".

I've filed  bug 671874  to follow up on my comments in #135. So, this bug should stay focused on windows-specific hangs.

There have been no win_chromium_rel_ng hangs reported by chromium-try-flakes since 436512 landed. Given that we were seeing multiple failures per day before, I'm inclined to call this bug fixed. We can re-open it if need be if we see more.

Quentin, can you file a separate bug to determine whether we should re-enable wdiff (and figure out why it was hanging), re-implement the wdiff functionality some other way (e.g., in pure python code), or just remove it altogether?

Sounds good; filed  bug 637478  for the wdiff issue (on an initial look, it appears that maybe it can just be removed).

I'm happy to close this for now and start a new issue for future hanging flakiness, although I don't think that 
hanging flakiness on Windows try bots is fully solved yet. Timeout rate and recent timeouts for the past few days include:

build_date	timeout_count	total_count	percent_timeout
2016-12-04	4	81	4.94
2016-12-05	9	264	3.41
2016-12-06	2	248	0.81
2016-12-07	5	297	1.68
2016-12-08	1	84	1.19

build_number	host_name	step_duration_s	build_date
346482	vm616-m4	6181.698	2016-12-08
346074	vm664-m4	8093.161	2016-12-07
345609	vm324-m4	4244.884	2016-12-07
345454	vm1210-m4	8293.897	2016-12-07
345444	vm846-m4	7770.655	2016-12-07
345343	vm329-m4	4450.391	2016-12-07
345296	vm325-m4	4098.992	2016-12-06
344971	vm325-m4	4016.317	2016-12-06
344351	vm716-m4	5736.754	2016-12-05
344178	vm156-m4	6195.165	2016-12-05

e.g.
https://luci-milo.appspot.com/buildbot/tryserver.chromium.win/win_chromium_rel_ng/346482
https://luci-milo.appspot.com/buildbot/tryserver.chromium.win/win_chromium_rel_ng/346074

The plan of checking up on suspicious builds on win_chromium_rel_ng and SSHing and
looking at the process tree still might be helpful, and the plan of auditing our use of
subprocess (bug 610613) still seems like a good idea.

I'll file a new bug and link to this one when I see another hang.

Correction: I filed  bug 672651  for the wdiff issue.

Quick follow-up: the number of layout test steps that take over an hour has fallen to < 1%, which appear to have different causes.

The two most recent builds where the step took > 1 hr both appear to have an issue where the render crashes for every test (separate issue, not sure yet whether this is going to be a recurring important issue:
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/349170/
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/350158

Deleted: Screenshot from 2016-12-16 10:10:43.png 13.3 KB

	Screenshot from 2016-12-16 10:10:43.png 13.3 KB View Download

On the timeout issue, I think it still makes sense to have a recipe-defined timeout for three reasons:

- The step might hang again in the future for a different reason
- The step will be marked red when it hangs
- The step will hang for less time, CQ will wait for less time (@dpranke -- do you think this isn't useful because the difference is only 10 minutes?)

I know there was a discussion about builder-specific timeouts that caused you to come to the conclusion that this change was too complex at the moment. Given that, and the fact that the difference in timeouts (50 minutes vs. 60) is very small, we could still do a recipe-defined timeout that is not builder specific and just have that equal to the global timeout (60 min). This could also be beneficial in the case where the global timeout increases, but we want the timeout for this step to remain the same.

@katthomas - I think I've lost the thread here. Are you suggesting that we should have per-step timeouts for no output that are not also per-builder and that override the global per-build timeout? I feel like that's probably not worth it at this point.