This is affecting tryjobs right now, pretty heavily.

https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_rel_ng/builds/286288 is another example build.

Several of the purple builds on https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_rel_ng?numbuilds=200 are caused by this bug.

Stip will take a look.

Mirrored waterfall bots are https://build.chromium.org/p/chromium.win/builders/Win%20x64%20Builder and https://build.chromium.org/p/chromium.win/builders/Win%207%20Tests%20x64%20%281%29

First build that failed seems to be https://build.chromium.org/p/chromium.win/builders/Win%207%20Tests%20x64%20%281%29/builds/17012

Suspicious, but doesn't look like it directly contributes: https://chromium.googlesource.com/chromium/src/+/77d78ea609d82f419d35911ad4d6cedef30ef00a

Hey, uh:

https://build.chromium.org/p/chromium.win/builders/Win%207%20Tests%20x64%20%281%29/builds/17012/steps/browser_tests%20on%20Windows-7-SP1/logs/failed%20to%20process%20gtest%20output%20JSON

Traceback (most recent call last):
  File "E:\b\rr\tmpcsltmh\rw\checkout\scripts/slave\recipe_modules\swarming\resources\collect_gtest_task.py", line 183, in main
    merged = merge_shard_results(task_output_dir)
  File "E:\b\rr\tmpcsltmh\rw\checkout\scripts/slave\recipe_modules\swarming\resources\collect_gtest_task.py", line 67, in merge_shard_results
    json_data = load_shard_json(output_dir, index)
  File "E:\b\rr\tmpcsltmh\rw\checkout\scripts/slave\recipe_modules\swarming\resources\collect_gtest_task.py", line 104, in load_shard_json
    return json.load(f)
  File "E:\b\depot_tools\python276_bin\lib\json\__init__.py", line 290, in load
    **kw)
  File "E:\b\depot_tools\python276_bin\lib\json\__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "E:\b\depot_tools\python276_bin\lib\json\decoder.py", line 365, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "E:\b\depot_tools\python276_bin\lib\json\decoder.py", line 381, in raw_decode
    obj, end = self.scan_once(s, idx)
MemoryError

known separate bug: bug 634418

Oh, maybe this is just the same bug....

stip, can you confirm the memory error is causing all these failures?

Yes, because the memory error happens in collect_gtest_task.py, the recipe can't properly consume the gtest data. Because of this, it bails out.