Note that the one 'success' recently is actually just the analyze step saying nothing needs to be done. I wish that status had a different label/color (like 'analyzed'/white instead of 'success'/green).
This seems to have started with https://build.chromium.org/p/tryserver.chromium.win/builders/win_optional_gpu_tests_rel/builds/13087 , but the actual stack trace doesn't seem obviously related (though I haven't rulled that out). It seems interesting that this bot started failing here but none of the waterfall bots seem to indicate any issue.
This also blocks V8 roll. Any ETA? So far we manually remove the bot from our rolls. Why does it break while CI seems to be ok? That's quite a bad state.
Ok that's quite strange!
I'm looking into this now, but I can't imagine why it's failing on gpu tests, but not other tests...
What's special on the GPU bots? Is it possible to run them on my windows workstation? I tried to run TabCaptureApiPixelTest.EndToEndWithoutRemoting, but I got the message "Skipping this CPU-intensive test on this platform/build.", so I assume it didn't actually run.
I also tried to run TabCaptureApiPixelTest.EndToEndThroughWebRTC, but it failed a CHECK in path_service.cc(323): Check failed: key_start >= iter->key_end || key_end <= iter->key_start. path provider collision
Here's the command I used:
>ninja -C out/GnDebug browser_tests -j 512 && out\GnDebug\browser_tests.exe --enable_gpu --gtest_also_run_disabled_tests --gtest_filter=*TabCaptureApiPixelTest* --single_process --user-data-dir=C:\chrome-testing-profile
I also tried running CastStreamingApiTestWithPixelOutput.EndToEnd, but it seemed to hang without doing anything...
C:\src\chrome\src>out\GnDebug\browser_tests.exe --enable-gpu --gtest_also_run_disabled_tests --gtest_filter=*CastStreamingApiTestWithPixelOutput*EndToEnd* --no-xvfb --single_process --test-launcher-bot-mode --test-launcher-jobs=1 --user-data-dir=C:\chrome-testing-profile
I think I've found the root cause of the issue:
The main builders have moved to "is_clang = true" on Windows, while the GPU builders still set "is_clang = false". The CL in question seems to have an interaction with the MSVC compiler that causes it to generate crashy code, so this was only seen on the GPU bots where we still use MSVC. Looking at the code where it crashes, my guess is that something about taking a reference to a "static constexpr" may be hitting an MSVC compatibility issue.
From reading the CL where Clang was enabled by default for Windows (https://chromium.googlesource.com/chromium/src/+/d2c91228a51bdf37ae3b2e501fb53c0528f1629c), it sounds like we want MSVC to keep working (at least in the near future). This is why the clang builders were updated to now run with "is_clang = false". However, these builders didn't catch this error as they just build, and don't run tests.
Given this, we probably want to make the CL above work with MSVC - although maybe this isn't necessary now that Clang seems to be sticking?
On the other side of things, we probably want to update the GPU builders to match the main builders, and use is_clang = true.
+hans@, the above change causes runtime issues when built with MSVC, but not Clang. It was caught by the GPU bots, which are still running with "is_clang = false" (we probably need to update these).
Do you know if it's still important to keep our MSVC builds working at this point? If so, would it be worth adding a test bot to run the binaries built by the "is_clang=false" builder?
> Do you know if it's still important to keep our MSVC builds working at this point?
Yes, we should keep it working for at least a few releases.
> If so, would it be worth adding a test bot to run the binaries built by the "is_clang=false" builder?
We do run tests on these bots (despite the name, they're now msvc builds): https://build.chromium.org/p/chromium.fyi/console?category=win%20clang Do they not run the binaries you're talking about here?
> We do run tests on these bots (despite the name, they're now msvc builds): https://build.chromium.org/p/chromium.fyi/console?category=win%20clang Do they not run the binaries you're talking about here?
Thanks for pointing these out! Looks like they did catch the bug as well as the GPU bots.
Great catch Eric!
The main reason the GPU FYI bots are still building with MSVC is that they need to build and run the drawElements (dEQP) tests for the ANGLE team, and these use exceptions, which are not supported by the Clang team.
There's work underway to split these tests off onto their own builder and only use MSVC there, so that the majority of the GPU FYI tests would run with Clang like the rest of the bots.
machenbach@: per c#4: win_optional_gpu_tests_rel would have caught this bug, but it only runs against a subset of CLs because we don't have enough hardware to run it against all of them. This CL didn't run that trybot, but per https://bugs.chromium.org/p/chromium/issues/detail?id=545324#c23 , re-lands should.
> The main reason the GPU FYI bots are still building with MSVC is that they need to build and run the drawElements (dEQP) tests for the ANGLE team, and these use exceptions, which are not supported by the Clang team.
I don't think that phrasing is quite right. "Aren't supported in chrome builds" is more like it I think. If we were to roll, say, chrome's pinned MSVC to a version that has broken exception support, we wouldn't want to revert that roll for breaking gpu fyi bots either.
> There's work underway to split these tests off onto their own builder and only use MSVC there
As said elsewhere, that's cool as long as that bot doesn't build any chromium code. We'll eventually move MSVC to community-supported for chrome builds, which means chrome engs won't be responsible for keeping it buildable in that config. (Subteams like v8 and skia could decide if they wanted to keep supporting building in MSVC, but they should have the option of deciding against it.)
The crash here has blink on the stack, so I'm wondering if you expect blink to be buildable with MSVC on the GPU FYI bots. If so, that's a problem.
No, we only expect the dEQP tests and ANGLE to be buildable with MSVC. This Blink crash happened because all of the tests on the Windows bots on the chromium.gpu.fyi waterfall, not just the dEQP tests, have been switched to use MSVC. We are requesting more machines from the Labs team so that we can build just the dEQP tests with MSVC and everything else on this waterfall with Clang.
I'm a little confused about what action I should take here...
Do I need to make my CL work with MSVC long term? Or should I put in a temporary fix that I can change back once blink isn't being built with MSVC?
meade@: hans@ pointed out that Chromium's going to support building with MSVC for a few more releases -- this is a ~9 month timeframe, so please consider that when adjusting your CL.
Thanks for the info folks! I have the (adjusted) re-land for the problematic CL here in case anyone is interested: https://chromium-review.googlesource.com/c/609660
I'm running the win_optional_gpu_tests_rel bot to make sure it works.
Comment 1 by bunge...@chromium.org
, Aug 7 2017