From a random sample, this batch is mostly LayoutTableRowDeathTest things

LayoutTableRowDeathTest.CrashIfSettingUnsetRowIndex
https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/builds/267802

Retrying 1 test (retry #3)
[ RUN      ] LayoutTableRowDeathTest.CrashIfSettingUnsetRowIndex

[WARNING] ../../testing/gtest/src/gtest-death-test.cc:834:: Death tests use fork(), which is unsafe particularly in a threaded context. For this test, Google Test detected 5 threads.
[3707/3707] LayoutTableRowDeathTest.CrashIfSettingUnsetRowIndex (TIMED OUT)
1 test timed out:
    LayoutTableRowDeathTest.CrashIfSettingUnsetRowIndex (../../third_party/WebKit/Source/core/layout/LayoutTableRowTest.cpp:72)



LayoutTableCellDeathTest.CrashIfColumnOverflowOnSetting
LayoutTableCellDeathTest.CrashIfSettingUnsetColumnIndex
LayoutTableRowDeathTest.CrashIfSettingUnsetRowIndex
LayoutTableRowDeathTest.CrashIfRowOverflowOnSetting
https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/builds/269140
4 tests timed out:
    LayoutTableCellDeathTest.CrashIfColumnOverflowOnSetting (../../third_party/WebKit/Source/core/layout/LayoutTableCellTest.cpp:67)
    LayoutTableCellDeathTest.CrashIfSettingUnsetColumnIndex (../../third_party/WebKit/Source/core/layout/LayoutTableCellTest.cpp:72)
    LayoutTableRowDeathTest.CrashIfRowOverflowOnSetting (../../third_party/WebKit/Source/core/layout/LayoutTableRowTest.cpp:67)
    LayoutTableRowDeathTest.CrashIfSettingUnsetRowIndex (../../third_party/WebKit/Source/core/layout/LayoutTableRowTest.cpp:72)

Actually I'll leave Sheriff-Chromium on for now. These tests might need to be disabled, but with the strange detection of these tests coming from just "webkit_unit_tests" there might be some other underlying problem.

@suzyh who fixed similar death tests in  issue 631021 .

Note that instead of disabling you might want to look into
testing::FLAGS_gtest_death_test_style = "threadsafe";

(not sure what it does, but it's used in our codebase...)

I also saw the thread-safeness warning for other death tests in webkit_unit_tests, but they are not reported flaky. Not sure if the warning applies for real multi-thread tests only.

This bug reminds me  bug 619153 . Could this be the case that the renderer crashed but is not detected by the main process?

I 'fixed' the death test in  issue 631021  by deleting it, and did so as an owner of the code being tested. I am uncomfortable making a call on a similar test in another team's area. Perhaps robhogan will know who should own this?

I'd consider mstensho the defacto owner of tables. He's on A/L till later in the month though. These tests don't look OK to remove though.

@wangxianzhu - this does appear to be related to  bug 619153  as it is only the Mac trybots that are flaking.

Detected 3 new flakes for test/step "webkit_unit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyKQsSBUZsYWtlIh53ZWJraXRfdW5pdF90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

Detected 7 new flakes for test/step "webkit_unit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyKQsSBUZsYWtlIh53ZWJraXRfdW5pdF90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_unit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyKQsSBUZsYWtlIh53ZWJraXRfdW5pdF90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webkit_unit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyKQsSBUZsYWtlIh53ZWJraXRfdW5pdF90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

I think dgrogan is the actual table code owner. :)

Detected 8 new flakes for test/step "webkit_unit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyKQsSBUZsYWtlIh53ZWJraXRfdW5pdF90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

This is still flaking and making the tree red, I'm going to disable them all until someone takes action here. It's been 11 days since this was reported.

This is all layout team now, the tests were disabled.

https://codereview.chromium.org/2237573003/

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2c361a3f7ff085f19d9a14c1ad35f4ebce7e25c6

commit 2c361a3f7ff085f19d9a14c1ad35f4ebce7e25c6
Author: esprehn <esprehn@chromium.org>
Date: Wed Aug 10 21:30:46 2016

Disable flaky LayoutTableCellDeathTest and LayoutTableRowDeathTest.

BUG= 632816 
TBR=dgrogan@chromium.org

Review-Url: https://codereview.chromium.org/2237573003
Cr-Commit-Position: refs/heads/master@{#411149}

[modify] https://crrev.com/2c361a3f7ff085f19d9a14c1ad35f4ebce7e25c6/third_party/WebKit/Source/core/layout/LayoutTableCellTest.cpp
[modify] https://crrev.com/2c361a3f7ff085f19d9a14c1ad35f4ebce7e25c6/third_party/WebKit/Source/core/layout/LayoutTableRowTest.cpp

Detected 4 new flakes for test/step "webkit_unit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyKQsSBUZsYWtlIh53ZWJraXRfdW5pdF90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

Back from OOO and starting to look at this.

wangxianzhu: this is similar to  issue 619153  in that it started at about the same time, the OS is mac and there are test timeouts instead of failures. But it's dissimilar in that this issue is flaky whereas 619153 was deterministic (I think) and this issue persists even after your low-level fix. Do you have any pointers on how to determine if the issues have similar causes? (I'd like to piggy-back on your research rather than redo it.)

cbiesinger: Instead of digging in blindly to the failures I could bisect to see where the flakes started. Do you know if there is a tool that will help me bisect webkit_unit_tests for flakiness or am I completely on my own? Completely on my own means manually keeping track of and checking out revisions manually and rebuilding webkit_unit_tests at each checkout. I'm asking you because last I looked, the bisect tools were in various states of disrepair but you seem to be adept at their use.

I never had to do this myself... I was going to point you to https://www.chromium.org/developers/bisecting-bugs but that doesn't seem to be working. You can always use git bisect -- https://git-scm.com/docs/git-bisect. It may not be too bad (remember gclient sync for each revision)

Other than that, I'm not aware of anything that would work for you. Conceivably we archive the test binary somewhere... but I wouldn't know if that's true or where that is.

(lowering priority as the tests have been disabled. we should still figure out what caused it and fix it though!)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/999f2f8ad6a65f52d6ad3e36b8ab03224b7af5d4

commit 999f2f8ad6a65f52d6ad3e36b8ab03224b7af5d4
Author: dgrogan <dgrogan@chromium.org>
Date: Wed Aug 24 22:05:57 2016

Re-enable LayoutTable{Cell,Row}DeathTest.* on non-Mac

They only flake on mac_chromium_rel_ng. I couldn't reproduce locally.

BUG= 632816 

Review-Url: https://codereview.chromium.org/2277893003
Cr-Commit-Position: refs/heads/master@{#414166}

[modify] https://crrev.com/999f2f8ad6a65f52d6ad3e36b8ab03224b7af5d4/third_party/WebKit/Source/core/layout/LayoutTableCellTest.cpp
[modify] https://crrev.com/999f2f8ad6a65f52d6ad3e36b8ab03224b7af5d4/third_party/WebKit/Source/core/layout/LayoutTableRowTest.cpp

These tests are now re-enabled on non-mac non-android.

Detected 3 new flakes for test/step "webkit_unit_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyKQsSBUZsYWtlIh53ZWJraXRfdW5pdF90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

I didn't see any flaky timeout of the test in the runs reported in #24. Did I miss anything?

I don't think you missed anything, it was All/WebViewTest.PreferredSizeDirtyLayout/1 that flaked. chromium-try-flakes probably isn't sophisticated enough to differentiate between tests inside of webkit_unit_tests.

The LayoutTable{Cell,Row}DeathTest.* seem all stable in the past months.

The timeout on mac may be caused by  bug 644303 , so I think we can close this bug now.

Woah. Déjà vu - I triaged this back in #c1 and I'm Sheriff again :p

https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/324061 seems to be All/WebViewTest.PreferredSizeDirtyLayout/1 (I'm not sure that's  bug 644303 ). But let's close this out so that chromium-try-flakes files a new bug.