components_perftests failing on chromium.perf/win-10-perf |
|||
Issue descriptionFiled by sheriff-o-matic@appspot.gserviceaccount.com on behalf of charliea@chromium.org components_perftests failing on chromium.perf/win-10-perf Builders failed on: - win-10-perf: https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/win-10-perf It looks like the "components_perftests" step is failing on the Win 10 perf bot about 20% of the time. Looking at the failures (example at https://chrome-swarming.appspot.com/task?id=4227561b67c4a710&refresh=10&show_raw=1), I see two tests that are failing: - ProtoDBPerfTest.InsertSingleDBAlternating_Individual_100b - ProtoDBPerfTest.InsertMultipleDBsAlternating_Individual_100b It seems like this has probably been happening since at least as far back as 11/26 based on https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/win-10-perf?limit=400 Going to go ahead and disable these due to flakiness and assign to altimin@, who wrote the tests originally, for investigation.
,
Jan 3
Quick correction: it looks like the failing test was ProtoDBPerfTest.InsertSingleDBAlternating_Individual_1000b (the large one), not 100b (the medium one). However, one of the medium ones did time out (ProtoDBPerfTest.InsertMultipleDBsAlternating_Individual_100b). Quoting my chat with thildebr@ on chat: "something about disk IO in Windows is causing some wild flakiness. I never had the time to dig into exactly what was causing the problem or how much it varied locally on a Windows test machine. Disabling the tests on Windows until we know why it varies so much from the others is probably an OK solution" Based on that, I'm going to go ahead and disable the tests that have failed on Windows in the last ~200 runs. Also swapping out thildebr@, who doesn't work on Chrome any longer, for nyquist@, who does
,
Jan 3
The following tests have timed out: ProtoDBPerfTest.InsertMultipleDBsAlternating_Individual_1000b ProtoDBPerfTest.InsertMultipleDBsAlternating_Individual_100b ProtoDBPerfTest.InsertSingleDBAlternating_Individual_1000b ProtoDBPerfTest.InsertSingleDBAlternating_Individual_100b I'll go ahead and disable these on Windows. Another thing worth noting: when these tests occasionally fail, they tend to fail in groups. It's more common to see three or four of these tests fail together than it should be if their probabilities of failing are independent from one another. This makes me think that thildebr@'s theory about slow conditions on the test machine (IO?) being responsible seem likely.
,
Jan 3
Yeah, I'm not sure what to do with this honestly, other than disabling the tests on Windows. Thanks for doing that! robliao: Do you have any idea why this is a problem typically only on Windows? ssid: FYI
,
Jan 3
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/d441e02ee44c186f7cbc878dfb46a6e8547844e9 commit d441e02ee44c186f7cbc878dfb46a6e8547844e9 Author: Charlie Andrews <charliea@chromium.org> Date: Thu Jan 03 19:35:32 2019 Disable flaky proto DB perf tests on Windows TBR=nyquist@chromium.org Bug: 918874 Change-Id: I45eaab1c4f4ba92d6a90f2826b92fa3c3eb37774 Reviewed-on: https://chromium-review.googlesource.com/c/1394823 Reviewed-by: Charlie Andrews <charliea@chromium.org> Reviewed-by: Tommy Nyquist <nyquist@chromium.org> Commit-Queue: Charlie Andrews <charliea@chromium.org> Cr-Commit-Position: refs/heads/master@{#619719} [modify] https://crrev.com/d441e02ee44c186f7cbc878dfb46a6e8547844e9/components/leveldb_proto/proto_database_perftest.cc
,
Jan 15
For this bug, it would be instructive to get a dump of the stack when it timed out. If you can get that, that would certainly shed light on what's going on here. |
|||
►
Sign in to add a comment |
|||
Comment 1 by charliea@google.com
, Jan 3