This error doesn't happen to all the try jobs though, see https://ci.chromium.org/p/chromium/builders/luci.chromium.findit/findit_variable/3209

This seems like a P2 at least.

robertma@, can you take a quick look at this, and unassign if it's more work than it seems?

I can't reproduce the problem locally (by running the same command). I'll try to download the isolated input.

Here's the log on Swarming:

Command: /b/s/w/ir/.swarming_module_cache/vpython/73deba/bin/python ../../third_party/blink/tools/run_web_tests.py --seed 4 --no-show-results --zero-tests-executed-ok --clobber-old-results --exit-after-n-failures 5000 --exit-after-n-crashes-or-timeouts 100 --debug-rwt-logging --results-directory /b/s/w/io8m8gbK/layout-test-results --additional-expectations ../../third_party/WebKit/LayoutTests/MSANExpectations --time-out-ms 66000 --enable-sanitizer --gtest_filter=http/tests/devtools/service-workers/service-worker-v8-cache.js --gtest_repeat=20 --test-launcher-retry-limit=0 --gtest_also_run_disabled_tests --write-full-results-to /b/s/w/io8m8gbK/output.json

10:57:57.467 24095 Using port 'linux-trusty'
10:57:57.467 24095 Test configuration: <trusty, x86_64, release>
10:57:57.467 24095 View the test results at file:///b/s/w/io8m8gbK/layout-test-results/results.html
10:57:57.467 24095 Using random order with seed: 4
10:57:57.467 24095 Baseline search path: linux -> win -> generic
10:57:57.467 24095 Using Release build
10:57:57.467 24095 Pixel tests enabled
10:57:57.467 24095 Regular timeout: 66000, slow test timeout: 330000
10:57:57.467 24095 Command line: /b/s/w/ir/out/Release/content_shell --run-web-tests --ignore-certificate-errors-spki-list=Nxvaj3+bY3oVrTc+Jp7m3E3sB1n3lXtnMDCyBsqEXiY= --user-data-dir --enable-crash-reporter --crash-dumps-dir=/b/s/w/ir/out/Release/crash-dumps -
10:57:57.467 24095 
10:57:57.467 24095 Collecting tests ...
10:57:57.467 24095 Collecting tests ...
10:57:57.472 24095 chunk slice [1:2] of 1 is 0 tests
10:57:57.472 24095 Parsing expectations ...
10:57:57.477 24095 reading additional_expectations from path '../../third_party/WebKit/LayoutTests/MSANExpectations'

KeyError raised: <blinkpy.web_tests.models.test_expectations.TestExpectationLine object at 0x7fb3bf47c950>
Traceback (most recent call last):
  File "/b/s/w/ir/third_party/blink/tools/blinkpy/web_tests/run_webkit_tests.py", line 66, in main
    return run(port, options, args, stderr, stdout).exit_code
  File "/b/s/w/ir/third_party/blink/tools/blinkpy/web_tests/run_webkit_tests.py", line 592, in run
    run_details = _run_tests(port, options, args, printer)
  File "/b/s/w/ir/third_party/blink/tools/blinkpy/web_tests/run_webkit_tests.py", line 583, in _run_tests
    return manager.run(args)
  File "/b/s/w/ir/third_party/blink/tools/blinkpy/web_tests/controllers/manager.py", line 129, in run
    self._expectations.remove_tests_from_expectations(tests_in_other_chunks)
  File "/b/s/w/ir/third_party/blink/tools/blinkpy/web_tests/models/test_expectations.py", line 1228, in remove_tests_from_expectations
    self._model.remove_expectation_line(line)
  File "/b/s/w/ir/third_party/blink/tools/blinkpy/web_tests/models/test_expectations.py", line 731, in remove_expectation_line
    del self._test_to_expectation_line[test]
KeyError: <blinkpy.web_tests.models.test_expectations.TestExpectationLine object at 0x7fb3bf47c950>

Weird, I tried the following steps to reproduce the Swarming task locally:

python isolateserver.py download -I https://isolateserver.appspot.com --namespace default-gzip -s 24332ae101442cc52a786fae3ec2ebf8b37ae0d7 --target ~/tmp
cd ~/tmp/out/Release
../../third_party/blink/tools/run_web_tests.py --seed 4 --no-show-results --zero-tests-executed-ok --clobber-old-results --exit-after-n-failures 5000 --exit-after-n-crashes-or-timeouts 100 --debug-rwt-logging  --additional-expectations ../../third_party/WebKit/LayoutTests/MSANExpectations --time-out-ms 66000 --enable-sanitizer --gtest_filter=http/tests/devtools/service-workers/service-worker-v8-cache.js --gtest_repeat=20 --test-launcher-retry-limit=0 --gtest_also_run_disabled_tests

But still no luck. Everything seems to work fine locally.

chanli@ stgao@ any other difference the Swarming job might have?

Sorry I cannot think of anything at my end... I remembered you mentioned that you'll retry the swarming task, did you mean to retry it locally? have you tried to retry the task directly from the task page?

FYI, I retried one of the failed try jobs yesterday and got the same error again.

Yeah retrying the task on Swarming can reproduce the error reliably, but so far  can't reproduce it locally.

Gentle ping. Is there anything I could do at my end?

chanli@ have you seen more examples of this? Do you have a rough estimate of how often it happens?

I haven't seen this failure happening over the weekends. Is there any change/fix to resolve this? 

No... I still don't know why it happened (and can't be reproduced locally) in the first place. Let's wait one more week to see if it happens again.

Haven't seen this for a while, marked it as wontFix for now.