Occasionally the GPU process that is launched by content_shell.exe while running webkit_layout_tests will fail to terminate cleanly, preventing Swarming from deleting the temporary directory and failing the entire shard. This causes an un-actionable CQ failure which causes the entire run to be retried.

Here's one example:
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/win7_chromium_rel_ng/24116

Specifically, this shard:
https://chromium-swarm.appspot.com/task?id=3e518c9492128210&refresh=10&show_raw=1

Here's a log excerpt:

14:10:23.253 1680 Testing completed. Exit status: 0
Failed to delete e:\b\s\w\ir (4 files remaining).
  Maybe the test has a subprocess outliving it.
  Sleeping 2 seconds.
Failed to delete e:\b\s\w\ir (4 files remaining).
  Maybe the test has a subprocess outliving it.
  Sleeping 4 seconds.
Failed to delete e:\b\s\w\ir. The following files remain:
- \\?\e:\b\s\w\ir
- \\?\e:\b\s\w\ir\out
- \\?\e:\b\s\w\ir\out\Release
- \\?\e:\b\s\w\ir\out\Release\content_shell.exe
Enumerating processes:
- pid 4248; Handles: 2; Exe: None; Cmd: "e:\b\s\w\ir\out\Release\content_shell.exe" --type=gpu-process --field-trial-handle=812,3443643823849636651,14475209876073705687,131072 --enable-features=OutOfBlinkCORS --disable-gpu-sandbox --disable-gpu-rasterization --disable-skia-runtime-opts --enable-logging --run-web-tests --enable-crash-reporter --crash-dumps-dir="e:\b\s\w\ir\out\Release\crash-dumps\reports" --register-font-files="e:\b\s\w\ir\out\Release\/test_fonts/Ahem.ttf" --gpu-preferences=KAAAAAAAAACAAwBgAQAAAAAAAAAAAGAAEAAAAAAAAAAAAAAAAAAAACgAAAAEAAAAIAAAAAAAAAAoAAAAAAAAADAAAAAAAAAAOAAAAAAAAAAQAAAAAAAAAAAAAAAKAAAAEAAAAAAAAAAAAAAACwAAABAAAAAAAAAAAQAAAAoAAAAQAAAAAAAAAAEAAAALAAAA --use-gl=swiftshader --run-web-tests --enable-crash-reporter --crash-dumps-dir="e:\b\s\w\ir\out\Release\crash-dumps\reports" --register-font-files="e:\b\s\w\ir\out\Release\/test_fonts/Ahem.ttf" --enable-logging --service-request-channel-token=6229271957777897514 --mojo-platform-channel-handle=1920 /prefetch:2

There's no good reason for the GPU process to fail to terminate when the overall harness does.

Not sure but could this behavior be related to the switch to Swiftshader as the software renderer? Could one of Swiftshader's worker threads be processing some bad draw call sent by an earlier layout test, thereby preventing the GPU process from exiting?

Is there more instrumentation that could be added to figure out why this is happening?

I don't know what percentage of CQ jobs are affected by this particular failure mode, but right now I'm seeing 80% of my own runs on win7_chromium_rel_ng fail with layout test flakes so am marking this as P1.