Here's the traceback:

Traceback (most recent call last):
  File "E:\b\rr\tmpat4ts3\rw\checkout\scripts\slave\.recipe_deps\recipe_engine\recipe_engine\run.py", line 298, in _new_run
    recipe_result = recipe_script.run(api, self.properties, self.environ)
  File "E:\b\rr\tmpat4ts3\rw\checkout\scripts\slave\.recipe_deps\recipe_engine\recipe_engine\loader.py", line 108, in run
    self.run_steps, properties, environ, self.PROPERTIES, api=api)
  File "E:\b\rr\tmpat4ts3\rw\checkout\scripts\slave\.recipe_deps\recipe_engine\recipe_engine\loader.py", line 636, in invoke_with_properties
    arg_names, **additional_args)
  File "E:\b\rr\tmpat4ts3\rw\checkout\scripts\slave\.recipe_deps\recipe_engine\recipe_engine\loader.py", line 599, in _invoke_with_properties
    return callable_obj(*props, **additional_args)
  File "E:\b\rr\tmpat4ts3\rw\checkout\scripts/slave\recipes\chromium.py", line 51, in RunSteps
    api.chromium_tests.main_waterfall_steps()
  File "E:\b\rr\tmpat4ts3\rw\checkout\scripts\slave\.recipe_deps\recipe_engine\recipe_engine\recipe_api.py", line 765, in _inner
    return func(*a, **kw)
  File "E:\b\rr\tmpat4ts3\rw\checkout\scripts/slave\recipe_modules\chromium_tests\api.py", line 1119, in main_waterfall_steps
    test_runner()
  File "E:\b\rr\tmpat4ts3\rw\checkout\scripts/slave\recipe_modules\chromium_tests\api.py", line 317, in test_runner
    failed_ts = self.m.test_utils.run_tests(self.m, tl, suffix)
  File "E:\b\rr\tmpat4ts3\rw\checkout\scripts\slave\.recipe_deps\recipe_engine\recipe_engine\recipe_api.py", line 765, in _inner
    return func(*a, **kw)
  File "E:\b\rr\tmpat4ts3\rw\checkout\scripts/slave\recipe_modules\test_utils\api.py", line 189, in run_tests
    failed_tests.extend(group.run(caller_api, suffix))
  File "E:\b\rr\tmpat4ts3\rw\checkout\scripts/slave\recipe_modules\test_utils\api.py", line 505, in run
    (' (%s)' % suffix) if suffix else ''))
  File "E:\b\rr\tmpat4ts3\rw\checkout\scripts\slave\.recipe_deps\recipe_engine\recipe_engine\recipe_api.py", line 765, in _inner
    return func(*a, **kw)
  File "E:\b\rr\tmpat4ts3\rw\checkout\scripts/slave\recipe_modules\swarming\api.py", line 1153, in get_states
    task_ids, result.json.output['states'])}
TypeError: 'NoneType' object has no attribute '__getitem__'

Link: https://logs.chromium.org/v/?s=chrome%2Fbb%2Fchromium.perf%2FWin_7_Perf%2F3375%2F%2B%2Frecipes%2Fsteps%2FUncaught_Exception%2F0%2Flogs%2Fexception%2F0

Looks like last line in test_utils is 

https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/test_utils/api.py?q=recipe_modules/test_utils/api.py&sq=package:chromium&dr&l=505

And the last line in swarming is

https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/swarming/api.py?q=recipe_modules/swarming/api.py&sq=package:chromium&dr&l=1052

However, I think the actual failure here is

https://logs.chromium.org/v/?s=chrome%2Fbb%2Fchromium.perf%2FWin_7_Perf%2F3376%2F%2B%2Frecipes%2Fsteps%2Fcollect_tasks%2F0%2Fstdout

Which says 

1664 2018-10-01 15:51:52.348 E: Request to https://chrome-swarming.appspot.com/_ah/api/swarming/v1/tasks/get_states?task_id=4049258c2635f010&task_id=404925485c564210&task_id=404925954f643b10&task_id=4049257793c3c810&task_id=4049251a3c000010&task_id=4049258159c7d010&task_id=404925a825c00110&task_id=40492532b7ccda10&limit=200 failed with HTTP status code 403: 403 Client Error: Forbidden for url: https://chrome-swarming.appspot.com/_ah/api/swarming/v1/tasks/get_states?task_id=4049258c2635f010&task_id=404925485c564210&task_id=404925954f643b10&task_id=4049257793c3c810&task_id=4049251a3c000010&task_id=4049258159c7d010&task_id=404925a825c00110&task_id=40492532b7ccda10&limit=200 - Access is denied.
Failed to access https://chrome-swarming.appspot.com/_ah/api/swarming/v1/tasks/get_states?task_id=4049258c2635f010&task_id=404925485c564210&task_id=404925954f643b10&task_id=4049257793c3c810&task_id=4049251a3c000010&task_id=4049258159c7d010&task_id=404925a825c00110&task_id=40492532b7ccda10&limit=200
WARNING:root:get_states cmd had non-zero return code: 1

(I'm not sure why it's getting a 403 though).

maruel/vadimsh did something change with swarming auth recently?

Last green build: https://ci.chromium.org/buildbot/chromium.perf/Win%207%20Perf/3088
First failing build: https://ci.chromium.org/buildbot/chromium.perf/Win%207%20Perf/3089

Here's the list of tools/build changes it picked up between the two runs:
https://chromium.googlesource.com/chromium/tools/build/+log/ed2d9b438b04571d3fd252b0229e5051ffd85af8..65728173256a3e6503d4501f8dc227b02673dffc

Prob due to re4b14a8eca3a9a7a5eaa38c9e9790295656c72db given the collect error mentioned in #3

That is almost definitely due to my CL. I can revert. Is it expected that perf bots are not able to make API calls to chrome-swarming?

I'm reverting my CL.

https://ci.chromium.org/buildbot/chromium.perf/Win%207%20Perf/3394 seems to be running. Marking as fixed for now. Will ensure this won't happen again when the CL relands.

Any idea why exactly it broke? Seems strange a refactoring in recipes to cause new HTT 403 errors.

https://chromium.googlesource.com/chromium/tools/build/+/e4b14a8eca3a9a7a5eaa38c9e9790295656c72db isn't a refactoring. It's enabling a new feature which does a new type of query to swarming. It looks like it should have been passing https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/swarming/api.py?type=cs&q=auth-service-account-json&sq=package:chromium&g=0&l=730, which it wasn't. The new code isn't directly calling the swarming recipe module.