This is quite severe as we can't rely on those trybots at all anymore then. Raising prio as the resulting breakages require a lot of manual work.

[I've deleted my previous comment because it contained incorrect timestamps].

I looked into when this started occurring.  It looks like the last time that webkit_layout_tests (without patch)  ran without exhibiting this error was:

2017-05-10 2:30 (UTC) See: https://luci-milo.appspot.com/buildbot/tryserver.chromium.linux/linux_chromium_rel_ng/449337

The first time it failed was:

2017-05-10 2:42 (UTC)  See: https://luci-milo.appspot.com/buildbot/tryserver.chromium.linux/linux_chromium_rel_ng/449349

Based on when they were committed, these two changes look suspicious (97a5415 is more suspicious than 92c859f):


commit 97a54150ec0d3fb00517ff70071b9d367b303fd9
Author:     recipe-roller <recipe-roller@chromium.org>
AuthorDate: 2017-05-09 19:12:24 -0700
Commit:     Commit Bot <commit-bot@chromium.org>
CommitDate: 2017-05-10 02:40:48 +0000

    Roll recipe dependencies (nontrivial).

    This is an automated CL created by the recipe roller. This CL rolls recipe
    changes from upstream projects (e.g. depot_tools) into downstream projects
    (e.g. tools/build).


    Please review the expectation changes, and LGTM+CQ.


    More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug.
    depot_tools:
      https://crrev.com/6dadfbfcf77356b3cf4b972de93c0ba1f9472fe4 git-cl-upload: Set all reviewers and ccs in a single batch api call (agable@chromium.org)
    recipe_engine:
      https://crrev.com/c93ed98c3c3e6c9be82e88496e55cb1f3dd13372 [recipe_modules/step] do not set cwd if it is start_dir. (iannucci@chromium.org)


    R=iannucci@chromium.org, phajdan.jr@chromium.org
    TBR=phajdan.jr@chromium.org

    Recipe-Tryjob-Bypass-Reason: Autoroller
    Bugdroid-Send-Email: False
    Change-Id: Iff407ed4f89e75667fada8472f3c94d426e68700
    Reviewed-on: https://chromium-review.googlesource.com/501369
    Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
    Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

commit 92c859f50299571b9ed2c2d30e5ef9b3c17f9cca
Author:     Dan Jacques <dnj@chromium.org>
AuthorDate: 2017-05-09 19:11:28 -0700
Commit:     Commit Bot <commit-bot@chromium.org>
CommitDate: 2017-05-10 02:17:50 +0000

    remote_run: Roll fixed Kitchen.

    Roll fixed Kitchen to properly propagate return codes.

    Include "chromium.swarm" waterfall in canary list.

    BUG= chromium:694911 
    TEST=canary
      - This version has had a pass on canary waterfalls without issue.
    TBR=nodir@chromium.org

    Change-Id: If96ce0debb4dc66c3689c15577aa63a7ed0d380b
    Reviewed-on: https://chromium-review.googlesource.com/500891
    Reviewed-by: Daniel Jacques <dnj@chromium.org>
    Commit-Queue: Daniel Jacques <dnj@chromium.org>

97a54150ec0d3fb00517ff70071b9d367b303fd9 only changes the test expectations, it shouldn't have any impact on the behavior of things.

Though I see that exception.. that looks mighty bogus...

So the exception is being thrown from here: https://github.com/luci/recipes-py/blob/master/recipe_engine/recipe_api.py#L185

However I can't figure out what might cause the _step_stack to become empty during normal operation (and it seems extremely unlikely to be linked to the CLs you found)

Doesn't 97a54150ec0d3fb00517ff70071b9d367b303fd9 also bump the revision of ttps://chromium.googlesource.com/external/github.com/luci/recipes-py.git , which causes it to pick up:
https://chromium.googlesource.com/external/github.com/luci/recipes-py.git/+/c93ed98c3c3e6c9be82e88496e55cb1f3dd13372

I'm not sure whether that CL is innocuous or not.

Yes, that's the commit I was referring to; that CL only changes what shows up in the test expectation files, it shouldn't affect any behavior.

Specifically, before that CL a current working directory of START_DIR could show up in the expectation files; however, this is also the default value for the cwd. That CL makes it so that cwd of START_DIR no longer shows up in the expectations (that is: if the currently-set value is equivalent to the default, don't show it).

CWD doesn't have anything to do with the step stack though.

The change seems to only provide 'cwd' when it isn't equal to self.m.path['start_dir'] -- If something is depending on 'cwd' existing then it could be throwing an exception in a strange place?

Ah, ok I think I found a clue. Take a look at the stack trace here: https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Ftryserver.chromium.linux%2Flinux_chromium_rel_ng%2F449349%2F%2B%2Frecipes%2Fsteps%2FUncaught_Exception%2F0%2Flogs%2Fexception%2F0

Note the line:

  File "/b/rr/tmpe9asQl/rw/checkout/scripts/slave/.recipe_deps/depot_tools/recipes/recipe_modules/tryserver/api.py", line 223, in set_invalid_test_results_tryjob_result
    self._set_failure_type('INVALID_TEST_RESULTS')

I suspect that this stacktrace is actually a secondary error while trying to set an 'invalid test results' setting. 

Still no idea why though.

@tansell: the cwd in that context goes straight into the recipe engine, which treats None and 'start_dir' as interchangeable (with the exception of how they're represented in the expectations files). I don't believe that there's an opportunity for any other user code to inspect the value (though I've been surprised by the code in these recipes before...)

BTW The stack trace is pretty useless as it is occuring inside a finally statement which is clobbering the real stack trace?

Yeah, so at the time the exception is thrown from the engine, we're already here: https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/test_utils/api.py?q=test_utils/api.py+package:%5Echromium$&l=167

That said, that exception from the engine is pretty awful (but I don't think it's the cause here)

Ok, so I think what's happening is that the `webkit_layout_tests (without patch)` step is partially starting, after the _step_stack is drained[1]. Normally the next step starts and is immediately appended to the _step_stack[2]. However in this case it makes it into open_step[3], and then probably raises an exception while rendering the placeholders[4]. There's an indiscriminate `finally` somewhere (not sure where) on the stack which calls back into the engine (which is in an inconsistent state). _step_stack is empty at this point, so... 'splosion.

[1]: https://cs.chromium.org/chromium/infra/recipes-py/recipe_engine/run.py?q=recipe_engine/run.py&dr&l=262
[2]: https://cs.chromium.org/chromium/infra/recipes-py/recipe_engine/run.py?q=recipe_engine/run.py&dr&l=265
[3]: https://cs.chromium.org/chromium/infra/recipes-py/recipe_engine/run.py?q=recipe_engine/run.py&dr&l=264
[4]: https://cs.chromium.org/chromium/infra/recipes-py/recipe_engine/step_runner.py?dr&l=175

I suspect that the actual issue is that one of the Placeholders (test inputs, probably?) is bailing while processing https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Ftryserver.chromium.linux%2Flinux_chromium_rel_ng%2F449349%2F%2B%2Frecipes%2Fsteps%2Fwebkit_layout_tests__without_patch_%2F0%2Fstdout

I'll work on a patch to re-order this so that the engine isn't able to get into this invalid state, but it may be worth looking into whatever the inputs to that step are supposed to be doing.

Note: this bug was never assigned an owner, so it slipped through the cracks even though it is P0.  This should probably be owned by someone who is awake until it is resolved.  Assigning to tansell for now, becuase I have to run, but I'll let him decide who to hand it off to if necessary.

There are no src side changes which fall into the region;



commit 0dc3090c6e647245895ee6762dd6a07521382308
Author:     chrome-release-bot <chrome-release-bot@chromium.org>
AuthorDate: 2017-05-09 20:02:06 -0700
Commit:     chrome-release-bot <chrome-release-bot@chromium.org>
CommitDate: 2017-05-10 03:02:13 +0000   <---- This change is too late...

    Updating trunk VERSION from 3095.0 to 3096.0
    
    Cr-Commit-Position: refs/heads/master@{#470445}

commit 67970ee192706eed7c080811732bab47383d636c
Author:     dpranke <dpranke@chromium.org>
AuthorDate: 2017-05-09 17:36:40 -0700
Commit:     Commit bot <commit-bot@chromium.org>
CommitDate: 2017-05-10 00:36:40 +0000   <---- This change is too early....

    Revert of Ensure that builders that use goma strip absolute paths by default. (patchset #1 id:1 of https://codereview.chromium.org/2873513003/ )
    
    Reason for revert:
    Speculatively reverting in case this is causing goma crashes (see crbug.com/720074)

I have a CL up which should reveal the original exception: https://codereview.chromium.org/2885023004/  (even comes with a reproduction of the bug in the form of a test)

This code in the engine is really groady though and needs some serious refactoring/cleanup.

The CL above landed; it's now being rolled to depot_tools and build.

Roll  https://chromium-review.googlesource.com/c/505396/ has landed as 77009791cb6e257f8decb1e19a29058023893f78

Ok, it's merged to build now (https://chromium-review.googlesource.com/c/505396/). Do we have a definite repro for this, or do we just have to wait?

I've manually triggered https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/457125 (whitespace change to md5.h) ... hopefully that catches it.

Re-created bad v8 roll https://chromium-review.googlesource.com/c/507128/
with it luck it compiles and actually triggers webkit tests

OOps, wrong link above. Correct link: https://chromium-review.googlesource.com/505624

This gets weirder: https://build.chromium.org/p/tryserver.chromium.mac/builders/ios-device/builds/212857
had problem applying the patch, but ignored it and continued with compilation.

This builder has been this weird at least since 20:00 PST yesterday: 
https://build.chromium.org/p/tryserver.chromium.mac/builders/ios-device/builds/212677

and even earlier https://luci-milo.appspot.com/buildbot/tryserver.chromium.mac/ios-device/212488

Filed  http://crbug.com/723572  for ios-device bug 

I wouldn't try that V8 roll for testing as it was a flaky test that failed. Also, we need to trigger webkit_tests and DEPS changes don't do that. A CL with a simple behavioral change to a layout test or it's expectation should do. E.g. just change any expectation file so that it'd fail. Analyze.py will also trigger layout tests in that case.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/050b9f74f78395ca1f5b15502e55a9534c61b4ce

commit 050b9f74f78395ca1f5b15502e55a9534c61b4ce
Author: Michael Achenbach <machenbach@chromium.org>
Date: Wed May 17 12:18:38 2017

Log masked exception in test utils

Bug:  721466 
Change-Id: I54568a864a3f3d7ea8d1312cde8454c29d3af6b9
Reviewed-on: https://chromium-review.googlesource.com/507208
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/050b9f74f78395ca1f5b15502e55a9534c61b4ce/scripts/slave/recipe_modules/test_utils/api.py

Waiting for http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/457243 to show actual error

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/c0087564f11b6a1b68d5027cf093c31f4abf2186

commit c0087564f11b6a1b68d5027cf093c31f4abf2186
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Wed May 17 12:57:09 2017

Add debug marker for ease text search for debugging info.

TBR=machenbach@chromium.org
Bug:  721466 
Change-Id: I7b0a19e6ce069306a5c50e0e8cc4694d5b495999
Reviewed-on: https://chromium-review.googlesource.com/507210
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/c0087564f11b6a1b68d5027cf093c31f4abf2186/scripts/slave/recipe_modules/test_utils/api.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/6693edfe68b50da94228077969ef3fe1af3becf4

commit 6693edfe68b50da94228077969ef3fe1af3becf4
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Wed May 17 13:57:16 2017

Turtles all the way down = aka debugging chromium_tests.

Log masked exception in test utils

R=machenbach@chromium.org
TBR=phajdan.jr@chromium.org
Bug:  721466 
Change-Id: Ieb19b27703f96ca3b7a6fff2574a70b85a1a95d6
Reviewed-on: https://chromium-review.googlesource.com/507487
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>

[modify] https://crrev.com/6693edfe68b50da94228077969ef3fe1af3becf4/scripts/slave/recipe_modules/chromium_tests/steps.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/c69c5e6e2a2d11156e8adf2dab4a9052d63ccf1b

commit c69c5e6e2a2d11156e8adf2dab4a9052d63ccf1b
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Wed May 17 14:01:26 2017

Turtles all the way down = aka debugging chromium_tests 2.

Log masked exception in swarming utils.

R=machenbach@chromium.org
TBR=phajdan.jr@chromium.org
Bug:  721466 
Change-Id: I8016376395517100f95c3a0143052ec6f11eb65f
Reviewed-on: https://chromium-review.googlesource.com/507308
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/c69c5e6e2a2d11156e8adf2dab4a9052d63ccf1b/scripts/slave/recipe_modules/swarming/api.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/c69c5e6e2a2d11156e8adf2dab4a9052d63ccf1b

commit c69c5e6e2a2d11156e8adf2dab4a9052d63ccf1b
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Wed May 17 14:01:26 2017

Turtles all the way down = aka debugging chromium_tests 2.

Log masked exception in swarming utils.

R=machenbach@chromium.org
TBR=phajdan.jr@chromium.org
Bug:  721466 
Change-Id: I8016376395517100f95c3a0143052ec6f11eb65f
Reviewed-on: https://chromium-review.googlesource.com/507308
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/c69c5e6e2a2d11156e8adf2dab4a9052d63ccf1b/scripts/slave/recipe_modules/swarming/api.py

Preliminary analysis: Some of this is caused or at least made hard to track down by using the following pattern:

try:
  step that fails
finally:
  logic that relies on step.active_result

There are exceptions that break the step entirely it seems and doesn't make it possible to use active_result to access those steps. The finally clause then fails as well with a different exception, masking the fist.

We have several layers of this pattern, making it really hard to know what the real exception is. We landed a few hacky, temporary CLs that attempt to print the real exception.

+jbudorick, who according to the logs fought a lot with this as well lately...

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/d9baad869a80389fbe6796415ae5596ccf0bbdd3

commit d9baad869a80389fbe6796415ae5596ccf0bbdd3
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Wed May 17 14:11:24 2017

Remove debug output during recipe training.

Follow up on Turtles all the way down = aka debugging chromium_tests
landed https://chromium-review.googlesource.com/c/507487

Log masked exception in swarming utils.

R=machenbach@chromium.org
TBR=phajdan.jr@chromium.org
Bug:  721466 
Change-Id: If6d8f915dd44dada86940bb4d9f319db375b4d67
Reviewed-on: https://chromium-review.googlesource.com/507507
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/d9baad869a80389fbe6796415ae5596ccf0bbdd3/scripts/slave/recipe_modules/test_utils/api.py
[modify] https://crrev.com/d9baad869a80389fbe6796415ae5596ccf0bbdd3/scripts/slave/recipe_modules/chromium_tests/steps.py

Looks like y'all are on the right track here (especially if you've gotten to using turtles in your commit messages... https://chromium.googlesource.com/chromium/tools/build/+/8b006d9f979747fe5989c327703bf23aa5fd67be). It may be useful to rework the finally blocks to either not use active_result if it isn't present or to eat that ValueError & allow the previous one to propagate rather than simply printing out the exception text.

We got closer: https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Ftryserver.chromium.linux%2Flinux_chromium_rel_ng%2F457330%2F%2B%2Frecipes%2Fsteps%2Fwebkit_layout_tests__without_patch_%2F0%2Fstdout 


DEBUG_DEBUG_DEBUG marker
Traceback (most recent call last):
  File "/b/rr/tmpdWeUc7/rw/checkout/scripts/slave/recipe_modules/chromium_tests/steps.py", line 1218, in post_run
    api.swarming.collect_task(self._tasks[suffix])
  File "/b/rr/tmpdWeUc7/rw/checkout/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/recipe_api.py", line 579, in _inner
    return func(*a, **kw)
  File "/b/rr/tmpdWeUc7/rw/checkout/scripts/slave/recipe_modules/swarming/api.py", line 676, in collect_task
    return task.collect_step(task, **kwargs)
  File "/b/rr/tmpdWeUc7/rw/checkout/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/recipe_api.py", line 579, in _inner
    return func(*a, **kw)
  File "/b/rr/tmpdWeUc7/rw/checkout/scripts/slave/recipe_modules/swarming/api.py", line 947, in _isolated_script_collect_step
    self.m.step.active_result.presentation.logs['no_isolated_results_exc'] = [str(e)]
  File "/b/rr/tmpdWeUc7/rw/checkout/scripts/slave/.recipe_deps/recipe_engine/recipe_modules/step/api.py", line 74, in active_result
    return self.step_client.previous_step_result()
  File "/b/rr/tmpdWeUc7/rw/checkout/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/recipe_api.py", line 187, in previous_step_result
    'No steps have been run yet, and you are asking for a previous step '
ValueError: No steps have been run yet, and you are asking for a previous step result.
THIS IS DEBUG_DEBUG_DEBUG MARKER
Traceback (most recent call last):
  File "/b/rr/tmpdWeUc7/rw/checkout/scripts/slave/recipe_modules/test_utils/api.py", line 157, in determine_new_failures
    self.run_tests(caller_api, failing_tests, 'without patch')
  File "/b/rr/tmpdWeUc7/rw/checkout/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/recipe_api.py", line 579, in _inner
    return func(*a, **kw)
  File "/b/rr/tmpdWeUc7/rw/checkout/scripts/slave/recipe_modules/test_utils/api.py", line 109, in run_tests
    t.post_run(caller_api, suffix)
  File "/b/rr/tmpdWeUc7/rw/checkout/scripts/slave/recipe_modules/chromium_tests/steps.py", line 1555, in post_run
    super(SwarmingIsolatedScriptTest, self).post_run(api, suffix)
  File "/b/rr/tmpdWeUc7/rw/checkout/scripts/slave/recipe_modules/chromium_tests/steps.py", line 1224, in post_run
    valid, failures = self.validate_task_results(api, api.step.active_result)
  File "/b/rr/tmpdWeUc7/rw/checkout/scripts/slave/.recipe_deps/recipe_engine/recipe_modules/step/api.py", line 74, in active_result
    return self.step_client.previous_step_result()
  File "/b/rr/tmpdWeUc7/rw/checkout/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/recipe_api.py", line 187, in previous_step_result
    'No steps have been run yet, and you are asking for a previous step '
ValueError: No steps have been run yet, and you are asking for a previous step result.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/fa05f0c918ae03a5b0595b44e7bf0b378404d0b0

commit fa05f0c918ae03a5b0595b44e7bf0b378404d0b0
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Wed May 17 14:49:49 2017

Turtles all the way down = aka debugging chromium_tests 3.

Log masked exception in swarming utils.

R=machenbach@chromium.org
TBR=phajdan.jr@chromium.org
Bug:  721466 
Change-Id: I42a47c46114f8040f537838940a65f76ca45feb6
Reviewed-on: https://chromium-review.googlesource.com/507229
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/fa05f0c918ae03a5b0595b44e7bf0b378404d0b0/scripts/slave/recipe_modules/swarming/api.py

And yet closer from https://luci-milo.appspot.com/buildbot/tryserver.chromium.linux/linux_chromium_rel_ng/457412 

DEBUG_DEBUG_DEBUG marker
Traceback (most recent call last):
  File "/b/rr/tmp7U28Qm/rw/checkout/scripts/slave/recipe_modules/swarming/api.py", line 940, in _isolated_script_collect_step
    task, step_test_data=step_test_data, **kwargs)
  File "/b/rr/tmp7U28Qm/rw/checkout/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/recipe_api.py", line 579, in _inner
    return func(*a, **kw)
  File "/b/rr/tmp7U28Qm/rw/checkout/scripts/slave/recipe_modules/swarming/api.py", line 788, in _default_collect_step
    **kwargs)
  File "/b/rr/tmp7U28Qm/rw/checkout/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/recipe_api.py", line 579, in _inner
    return func(*a, **kw)
  File "/b/rr/tmp7U28Qm/rw/checkout/scripts/slave/recipe_modules/build/api.py", line 51, in python
    **kwargs)
  File "/b/rr/tmp7U28Qm/rw/checkout/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/recipe_api.py", line 579, in _inner
    return func(*a, **kw)
  File "/b/rr/tmp7U28Qm/rw/checkout/scripts/slave/.recipe_deps/recipe_engine/recipe_modules/python/api.py", line 54, in __call__
    return self.m.step(name, cmd + list(args or []), **kwargs)
  File "/b/rr/tmp7U28Qm/rw/checkout/scripts/slave/.recipe_deps/recipe_engine/recipe_modules/step/api.py", line 212, in __call__
    return self.run_from_dict(schema.as_jsonish())
  File "/b/rr/tmp7U28Qm/rw/checkout/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/recipe_api.py", line 619, in _inner
    return func(*a, **kw)
  File "/b/rr/tmp7U28Qm/rw/checkout/scripts/slave/.recipe_deps/recipe_engine/recipe_modules/step/api.py", line 217, in run_from_dict
    return self.step_client.run_step(dct)
  File "/b/rr/tmp7U28Qm/rw/checkout/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/recipe_api.py", line 201, in run_step
    return self._engine.run_step(StepConfig.create(**step_dict))
  File "/b/rr/tmp7U28Qm/rw/checkout/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/run.py", line 264, in run_step
    open_step = self._step_runner.open_step(step_config)
  File "/b/rr/tmp7U28Qm/rw/checkout/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/step_runner.py", line 191, in open_step
    self._print_step(step_stream, rendered_step, step_env)
  File "/b/rr/tmp7U28Qm/rw/checkout/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/step_runner.py", line 295, in _print_step
    stream.output_iter(step_stream, gen_step_prelude())
  File "/b/rr/tmp7U28Qm/rw/checkout/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/stream.py", line 43, in output_iter
    for text in it:
  File "/b/rr/tmp7U28Qm/rw/checkout/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/step_runner.py", line 287, in gen_step_prelude
    yield 'in dir %s:' % (step.config.cwd or os.getcwd())
OSError: [Errno 2] No such file or directory

So, after 5 hours of going deeeper, the problem seems to be that last command cwd no longer exists.

After looking at codesearch for instances of chdir [1], I see recipe engine calling chdir only once at the invocation [2]. I therefore conclude that perhaps the directory was somehow removed.


[1] https://cs.chromium.org/search/?q=chdir+file:%5Einfra/recipes-py/recipe_engine/+package:%5Echromium$&type=cs
[2] https://cs.chromium.org/chromium/infra/recipes-py/recipe_engine/run.py?type=cs&q=os.chdir+file:%5Einfra/recipes-py/recipe_engine/+package:%5Echromium$&l=691

Previous step [1] which is

 [trigger] webkit_layout_tests (without patch) ( 5 ms )

has been run in PWD: /b/build/slave/remote_run/build
but with explicit CWD, and this explains why it succeeded. PWD likely didn't exist already. So, what I have to find is first step which might have removed PWD and ideally which was run with CWD=PWD  

[1] https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Ftryserver.chromium.linux%2Flinux_chromium_rel_ng%2F457412%2F%2B%2Frecipes%2Fsteps%2Fs__trigger__webkit_layout_tests__without_patch_%2F0%2Fstdout

So, at step 41 the CWD=PWD=/b/build/slave/remote_run/build still existed in build https://luci-milo.appspot.com/buildbot/tryserver.chromium.linux/linux_chromium_rel_ng/457412


Given that this is the omly swarmified webkit tests builder, i suspect the removal of the dir happens in one of the steps related to swarmified tests. 

I must catch the flight now, so it's to the next trooper to figure it out :(

can we spec revert iannucci's CWD CL? this is pretty adversely affecting linux_chromium_rel_ng...

that's what I just suggested too in https://codereview.chromium.org/2875543002/

ah, perfect.

reverting

#51: note that the cwd for collection doesn't match the cwd for triggering: https://codesearch.chromium.org/chromium/build/scripts/slave/recipe_modules/swarming/api.py?rcl=6685fe37889919448bfa3bbf3d93e15026680e0a&l=781

A better comparison point for cwd existence is likely the (with patch) collection step.

Since reverting it is a pain (and doesn't actually FIX the problem, which is that start_dir is gone), I'm landing a patch (https://codereview.chromium.org/2885293003) which will not cause the step_runner to bail when the cwd is missing, but should help us pinpoint what's going on (maybe the cwd comes back at some point? I'm not sure that's possible).

In the PM for this, we should discuss spec reverts and P0s.

Please continue to test the effects of your changes by kicking off linux_chromium_rel_ng on https://chromium-review.googlesource.com/c/505624/

The output of our output hacks can be seen in the stdout of the last webkit test step.

Bingo: https://luci-milo.appspot.com/buildbot/tryserver.chromium.linux/linux_chromium_rel_ng/457759

Something in the recipe is, in fact, removing the current working directory out from under the engine.

The last step where the current working directory exists seems to be https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Ftryserver.chromium.linux%2Flinux_chromium_rel_ng%2F457759%2F%2B%2Frecipes%2Fsteps%2Fwebkit_layout_tests__with_patch_%2F0%2Fstdout

Yep, and in that step: 

root: [WARNING] Output directory exists '/b/rr/tmppcbhwN/w'
root: [INFO] Removing /b/rr/tmppcbhwN/w

This logic seems like it was added to merge-layout-test-results here: https://chromium.googlesource.com/chromium/src/+/d2908c0%5E%21

So some possible fixes:
  * change merge-layout-test-results to only delete the CONTENTS of its output directory, not the output directory itself
  * run merge-layout-test-results in a different directory than its output directory
  * give merge-layout-test-results a subdirectory to use for its output

I will work on a cl to delete only the contents of the output directory in merge-layout-test-results.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/a3b6bc4351f05643d535938e6e7f1bfaf1bfdfc7

commit a3b6bc4351f05643d535938e6e7f1bfaf1bfdfc7
Author: Dirk Pranke <dpranke@chromium.org>
Date: Thu May 18 00:29:32 2017

Turn off swarming for the linux layout tests temporarily.

It appears that when we run the layout tests under swarming,
the script that merges the JSON results deletes the current
working directory, which is the main writable directory used
by remote_run. This is causing failures and other weirdness.
For now, I'm turning off swarming, so that we can then properly
debug things and figure out what the right thing to do is.

R=iannucci@chromium.org
TBR=tansell@chromium.org, mcgreevy@chromium.org
BUG= 721466 

Change-Id: Ib6ee007cf162335116b564214e2a0ee901860593
Reviewed-on: https://chromium-review.googlesource.com/508158
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Michael McGreevy <mcgreevy@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Tim 'mithro' Ansell <tansell@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@chromium.org>

[delete] https://crrev.com/9ec76523bb3629ab4874cd834c44dfa992929fc7/scripts/slave/recipes/chromium_trybot.expected/add_swarming_layout_tests_via_manual_diff_inspection_that_fails.json
[modify] https://crrev.com/a3b6bc4351f05643d535938e6e7f1bfaf1bfdfc7/scripts/slave/recipe_modules/chromium_tests/api.py
[modify] https://crrev.com/a3b6bc4351f05643d535938e6e7f1bfaf1bfdfc7/scripts/slave/recipes/chromium_trybot.py
[modify] https://crrev.com/a3b6bc4351f05643d535938e6e7f1bfaf1bfdfc7/scripts/slave/recipe_modules/chromium_tests/tests/api/trybot_steps.expected/blink_linux.json
[delete] https://crrev.com/9ec76523bb3629ab4874cd834c44dfa992929fc7/scripts/slave/recipes/chromium_trybot.expected/add_swarming_layout_tests_via_manual_diff_inspection.json

As per the commit in comment #67, we've turned off swarming for the layout tests. This should eliminate the problems we were seeing, so I'm downgrading this to P1.

I'm also reassigning this to tansell@, who is going to work on a postmortem w/ iannucci@ (and others as needed).

Among other things, it appears that the "api contract" between the recipe engine, the call out to collect the results of a swarming task, the call to the custom merge script to merge the results, and then the follow-up "archive layout tests" script is unclear and different pieces are expecting different things. 

This is unsurprising; we've known this for some time. However, I think we need to get to the bottom of this and document it somewhere now, rather than continuing to kick this down the road. 

It looks like there is probably also some refactoring and cleanup needed across the recipe_engine, the chromium_tests steps code, and the swarming module to make sure everything is consistent.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/74b36c2301966a939024a8f240a7896c5d9b0b87

commit 74b36c2301966a939024a8f240a7896c5d9b0b87
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Thu May 18 09:27:45 2017

Revert "Turtles all the way down = aka debugging chromium_tests 3."

This reverts commit fa05f0c918ae03a5b0595b44e7bf0b378404d0b0.

Reason for revert: no longer needed

Original change's description:
> Turtles all the way down = aka debugging chromium_tests 3.
> 
> Log masked exception in swarming utils.
> 
> R=​machenbach@chromium.org
> TBR=phajdan.jr@chromium.org
> Bug:  721466 
> Change-Id: I42a47c46114f8040f537838940a65f76ca45feb6
> Reviewed-on: https://chromium-review.googlesource.com/507229
> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
> Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
> 

TBR=phajdan.jr@chromium.org,machenbach@chromium.org,tandrii@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Bug:  721466 

Change-Id: I76eacde5b6b7e852daccc8039e0947619eeb36ad
Reviewed-on: https://chromium-review.googlesource.com/508688
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/74b36c2301966a939024a8f240a7896c5d9b0b87/scripts/slave/recipe_modules/swarming/api.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/df8a4659dcb826adfeb1f1534f124db33cda13e8

commit df8a4659dcb826adfeb1f1534f124db33cda13e8
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Thu May 18 12:04:56 2017

Revert "Remove debug output during recipe training."

This reverts commit d9baad869a80389fbe6796415ae5596ccf0bbdd3.

Reason for revert: no longer needed

Original change's description:
> Remove debug output during recipe training.
> 
> Follow up on Turtles all the way down = aka debugging chromium_tests
> landed https://chromium-review.googlesource.com/c/507487
> 
> Log masked exception in swarming utils.
> 
> R=​machenbach@chromium.org
> TBR=phajdan.jr@chromium.org
> Bug:  721466 
> Change-Id: If6d8f915dd44dada86940bb4d9f319db375b4d67
> Reviewed-on: https://chromium-review.googlesource.com/507507
> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
> 

TBR=phajdan.jr@chromium.org,machenbach@chromium.org,tandrii@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Bug:  721466 

Change-Id: I3cab10fcf169262b5f5106543400222aa81cd908
Reviewed-on: https://chromium-review.googlesource.com/508690
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/df8a4659dcb826adfeb1f1534f124db33cda13e8/scripts/slave/recipe_modules/test_utils/api.py
[modify] https://crrev.com/df8a4659dcb826adfeb1f1534f124db33cda13e8/scripts/slave/recipe_modules/chromium_tests/steps.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/589e5267097f76224e0ac2e2ea554aff635d979f

commit 589e5267097f76224e0ac2e2ea554aff635d979f
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Thu May 18 12:05:36 2017

Revert "Turtles all the way down = aka debugging chromium_tests 2."

This reverts commit c69c5e6e2a2d11156e8adf2dab4a9052d63ccf1b.

Reason for revert: not needed any more.

Original change's description:
> Turtles all the way down = aka debugging chromium_tests 2.
> 
> Log masked exception in swarming utils.
> 
> R=​machenbach@chromium.org
> TBR=phajdan.jr@chromium.org
> Bug:  721466 
> Change-Id: I8016376395517100f95c3a0143052ec6f11eb65f
> Reviewed-on: https://chromium-review.googlesource.com/507308
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
> 

TBR=phajdan.jr@chromium.org,machenbach@chromium.org,tandrii@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Bug:  721466 

Change-Id: I231b9c43cb04903f015f4d7ae9a6ac3403bc1785
Reviewed-on: https://chromium-review.googlesource.com/508691
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/589e5267097f76224e0ac2e2ea554aff635d979f/scripts/slave/recipe_modules/swarming/api.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/af07a8d4dd8b3525570758f551c386dd680e0bf2

commit af07a8d4dd8b3525570758f551c386dd680e0bf2
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Thu May 18 12:05:56 2017

Revert "Turtles all the way down = aka debugging chromium_tests."

This reverts commit 6693edfe68b50da94228077969ef3fe1af3becf4.

Reason for revert: not needed any more.

Original change's description:
> Turtles all the way down = aka debugging chromium_tests.
> 
> Log masked exception in test utils
> 
> R=​machenbach@chromium.org
> TBR=phajdan.jr@chromium.org
> Bug:  721466 
> Change-Id: Ieb19b27703f96ca3b7a6fff2574a70b85a1a95d6
> Reviewed-on: https://chromium-review.googlesource.com/507487
> Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> 

TBR=phajdan.jr@chromium.org,machenbach@chromium.org,tandrii@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Bug:  721466 

Change-Id: I1b51598136ac86724b3b1be6c2e21ccb21cbeffd
Reviewed-on: https://chromium-review.googlesource.com/508692
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/af07a8d4dd8b3525570758f551c386dd680e0bf2/scripts/slave/recipe_modules/chromium_tests/steps.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/81ae4cac09270557f27b650e4a19239d16ae9fde

commit 81ae4cac09270557f27b650e4a19239d16ae9fde
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Thu May 18 12:06:26 2017

Revert "Add debug marker for ease text search for debugging info."

This reverts commit c0087564f11b6a1b68d5027cf093c31f4abf2186.

Reason for revert: not needed any more.

Original change's description:
> Add debug marker for ease text search for debugging info.
> 
> TBR=machenbach@chromium.org
> Bug:  721466 
> Change-Id: I7b0a19e6ce069306a5c50e0e8cc4694d5b495999
> Reviewed-on: https://chromium-review.googlesource.com/507210
> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
> Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
> 

TBR=machenbach@chromium.org,tandrii@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Bug:  721466 

Change-Id: I1fd6e7806776a2e2a15761df486d7a059284f435
Reviewed-on: https://chromium-review.googlesource.com/508693
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/81ae4cac09270557f27b650e4a19239d16ae9fde/scripts/slave/recipe_modules/test_utils/api.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/208399a7f37deab314244d1c990570c0fd630a6a

commit 208399a7f37deab314244d1c990570c0fd630a6a
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Thu May 18 12:13:00 2017

Revert "Log masked exception in test utils"

This reverts commit 050b9f74f78395ca1f5b15502e55a9534c61b4ce.

Reason for revert: not needed any more.

Original change's description:
> Log masked exception in test utils
> 
> Bug:  721466 
> Change-Id: I54568a864a3f3d7ea8d1312cde8454c29d3af6b9
> Reviewed-on: https://chromium-review.googlesource.com/507208
> Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
> 

TBR=machenbach@chromium.org,tandrii@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Bug:  721466 

Change-Id: If0dedfb65a8c5daad19044abaf06dd335f21cc6c
Reviewed-on: https://chromium-review.googlesource.com/508694
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/208399a7f37deab314244d1c990570c0fd630a6a/scripts/slave/recipe_modules/test_utils/api.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0f2a25964b96c4333d57da6d7fb38afef4bd02fc

commit 0f2a25964b96c4333d57da6d7fb38afef4bd02fc
Author: tansell <tansell@chromium.org>
Date: Mon May 29 22:41:06 2017

webkitpy: Remove contents of directory.

Adding a function which removes the contents of a directory but *not*
the directory itself.

This is used by the merge-layout-test-results script when generating
output into an existing directory.

BUG= 721466 , 524758 

Review-Url: https://codereview.chromium.org/2896423004
Cr-Commit-Position: refs/heads/master@{#475391}

[modify] https://crrev.com/0f2a25964b96c4333d57da6d7fb38afef4bd02fc/third_party/WebKit/Tools/Scripts/merge-layout-test-results
[modify] https://crrev.com/0f2a25964b96c4333d57da6d7fb38afef4bd02fc/third_party/WebKit/Tools/Scripts/webkitpy/common/system/filesystem.py
[modify] https://crrev.com/0f2a25964b96c4333d57da6d7fb38afef4bd02fc/third_party/WebKit/Tools/Scripts/webkitpy/common/system/filesystem_mock.py
[modify] https://crrev.com/0f2a25964b96c4333d57da6d7fb38afef4bd02fc/third_party/WebKit/Tools/Scripts/webkitpy/common/system/filesystem_unittest.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/97068bd221ac64536763a715b123aa490f52d949

commit 97068bd221ac64536763a715b123aa490f52d949
Author: Tim 'mithro' Ansell <tansell@chromium.org>
Date: Tue May 30 00:56:13 2017

Revert "Turn off swarming for the linux layout tests temporarily."

This reverts commit a3b6bc4351f05643d535938e6e7f1bfaf1bfdfc7.

Reason for revert: Merge script no longer removes the directory.

Original change's description:
> Turn off swarming for the linux layout tests temporarily.
> 
> It appears that when we run the layout tests under swarming,
> the script that merges the JSON results deletes the current
> working directory, which is the main writable directory used
> by remote_run. This is causing failures and other weirdness.
> For now, I'm turning off swarming, so that we can then properly
> debug things and figure out what the right thing to do is.
> 
> R=​iannucci@chromium.org
> TBR=tansell@chromium.org, mcgreevy@chromium.org
> BUG= 721466 
> 
> Change-Id: Ib6ee007cf162335116b564214e2a0ee901860593
> Reviewed-on: https://chromium-review.googlesource.com/508158
> Reviewed-by: Dirk Pranke <dpranke@chromium.org>
> Reviewed-by: Michael McGreevy <mcgreevy@chromium.org>
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Reviewed-by: Tim 'mithro' Ansell <tansell@chromium.org>
> Commit-Queue: Dirk Pranke <dpranke@chromium.org>
> 

TBR=iannucci@chromium.org,dpranke@chromium.org,tansell@chromium.org,mcgreevy@chromium.org
# Not skipping CQ checks because original CL landed > 1 day ago.
BUG= 721466 

Change-Id: I5055a7d9ad988da2c93f596d3a378e5503d084ef
Reviewed-on: https://chromium-review.googlesource.com/517468
Reviewed-by: Tim 'mithro' Ansell <tansell@chromium.org>
Reviewed-by: Michael McGreevy <mcgreevy@chromium.org>
Commit-Queue: Tim 'mithro' Ansell <tansell@chromium.org>

[add] https://crrev.com/97068bd221ac64536763a715b123aa490f52d949/scripts/slave/recipes/chromium_trybot.expected/add_swarming_layout_tests_via_manual_diff_inspection_that_fails.json
[modify] https://crrev.com/97068bd221ac64536763a715b123aa490f52d949/scripts/slave/recipe_modules/chromium_tests/api.py
[modify] https://crrev.com/97068bd221ac64536763a715b123aa490f52d949/scripts/slave/recipes/chromium_trybot.py
[modify] https://crrev.com/97068bd221ac64536763a715b123aa490f52d949/scripts/slave/recipe_modules/chromium_tests/tests/api/trybot_steps.expected/blink_linux.json
[add] https://crrev.com/97068bd221ac64536763a715b123aa490f52d949/scripts/slave/recipes/chromium_trybot.expected/add_swarming_layout_tests_via_manual_diff_inspection.json

For everyone's information;

 * A temporary fix which only removes the contents of the directory, but not the directory itself has been landed. This has allowed us to roll layout tests on swarming back out again.
 * A proper fix which moves the layout test output into a subdirectory is almost ready and will be landed soon.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/3ac80500e4b47981ca0d262093af31a4db882ad6

commit 3ac80500e4b47981ca0d262093af31a4db882ad6
Author: Tim 'mithro' Ansell <tansell@chromium.org>
Date: Wed Sep 20 20:48:27 2017

recipies_modules/swarming: Improve exception information.

Add traceback information to exception output to make it possible to
track down where an error is coming from.

BUG= 721466 

Change-Id: I1063f3da1d38000ad2b5c250ce4d2eb4db26c396
Reviewed-on: https://chromium-review.googlesource.com/439664
Commit-Queue: Tim 'mithro' Ansell <tansell@chromium.org>
Reviewed-by: Paweł Hajdan Jr. <phajdan.jr@chromium.org>

[modify] https://crrev.com/3ac80500e4b47981ca0d262093af31a4db882ad6/scripts/slave/recipe_modules/swarming/api.py
[modify] https://crrev.com/3ac80500e4b47981ca0d262093af31a4db882ad6/scripts/slave/recipe_modules/disk/api.py
[add] https://crrev.com/3ac80500e4b47981ca0d262093af31a4db882ad6/scripts/slave/recipe_modules/traceback/OWNERS
[modify] https://crrev.com/3ac80500e4b47981ca0d262093af31a4db882ad6/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_corrupt_json_isolated_script_test.json
[modify] https://crrev.com/3ac80500e4b47981ca0d262093af31a4db882ad6/scripts/slave/recipe_modules/ios/examples/full.expected/expired.json
[add] https://crrev.com/3ac80500e4b47981ca0d262093af31a4db882ad6/scripts/slave/recipe_modules/traceback/test_api.py
[modify] https://crrev.com/3ac80500e4b47981ca0d262093af31a4db882ad6/scripts/slave/recipe_modules/disk/__init__.py
[add] https://crrev.com/3ac80500e4b47981ca0d262093af31a4db882ad6/scripts/slave/recipe_modules/traceback/examples/full.expected/linux.json
[modify] https://crrev.com/3ac80500e4b47981ca0d262093af31a4db882ad6/scripts/slave/recipe_modules/swarming/examples/full.expected/swarming_expired_new.json
[modify] https://crrev.com/3ac80500e4b47981ca0d262093af31a4db882ad6/scripts/slave/recipe_modules/chromium_tests/tests/steps/json_results_handler.expected/invalid.json
[modify] https://crrev.com/3ac80500e4b47981ca0d262093af31a4db882ad6/scripts/slave/recipes/chromium.expected/dynamic_swarmed_passed_isolated_script_test_with_swarming_failure.json
[modify] https://crrev.com/3ac80500e4b47981ca0d262093af31a4db882ad6/scripts/slave/recipe_modules/disk/examples/full.expected/space_usage_doesnt_fail_build.json
[modify] https://crrev.com/3ac80500e4b47981ca0d262093af31a4db882ad6/scripts/slave/recipe_modules/chromium_tests/__init__.py
[modify] https://crrev.com/3ac80500e4b47981ca0d262093af31a4db882ad6/scripts/slave/recipe_modules/swarming/examples/full.expected/swarming_expired_old.json
[modify] https://crrev.com/3ac80500e4b47981ca0d262093af31a4db882ad6/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_isolated_chartjson_test_harness_failure.json
[modify] https://crrev.com/3ac80500e4b47981ca0d262093af31a4db882ad6/scripts/slave/recipe_modules/swarming/examples/full.expected/gtest_with_null_shard.json
[modify] https://crrev.com/3ac80500e4b47981ca0d262093af31a4db882ad6/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_invalid_json_isolated_script_test.json
[add] https://crrev.com/3ac80500e4b47981ca0d262093af31a4db882ad6/scripts/slave/recipe_modules/traceback/examples/full.py
[modify] https://crrev.com/3ac80500e4b47981ca0d262093af31a4db882ad6/scripts/slave/recipes/chromium.expected/dynamic_local_isolated_script_test_with_corrupt_json_results.json
[modify] https://crrev.com/3ac80500e4b47981ca0d262093af31a4db882ad6/scripts/slave/recipe_modules/chromium_tests/steps.py
[add] https://crrev.com/3ac80500e4b47981ca0d262093af31a4db882ad6/scripts/slave/recipe_modules/traceback/examples/full.expected/win.json
[add] https://crrev.com/3ac80500e4b47981ca0d262093af31a4db882ad6/scripts/slave/recipe_modules/traceback/api.py
[add] https://crrev.com/3ac80500e4b47981ca0d262093af31a4db882ad6/scripts/slave/recipe_modules/traceback/__init__.py
[modify] https://crrev.com/3ac80500e4b47981ca0d262093af31a4db882ad6/scripts/slave/recipe_modules/disk/examples/full.expected/space_usage_fails_build.json
[modify] https://crrev.com/3ac80500e4b47981ca0d262093af31a4db882ad6/scripts/slave/recipes/chromium.expected/dynamic_isolated_script_test_harness_failure_no_json.json
[add] https://crrev.com/3ac80500e4b47981ca0d262093af31a4db882ad6/scripts/slave/recipe_modules/traceback/examples/full.expected/mac.json
[modify] https://crrev.com/3ac80500e4b47981ca0d262093af31a4db882ad6/scripts/slave/recipe_modules/chromium_tests/tests/steps/swarming_isolated_script_test.expected/chartjson_no_results_failure.json
[modify] https://crrev.com/3ac80500e4b47981ca0d262093af31a4db882ad6/scripts/slave/README.recipes.md
[modify] https://crrev.com/3ac80500e4b47981ca0d262093af31a4db882ad6/scripts/slave/recipe_modules/swarming/examples/full.expected/isolated_script_with_null_shard.json
[modify] https://crrev.com/3ac80500e4b47981ca0d262093af31a4db882ad6/scripts/slave/recipe_modules/chromium_tests/tests/steps/json_results_handler.py
[modify] https://crrev.com/3ac80500e4b47981ca0d262093af31a4db882ad6/scripts/slave/recipe_modules/swarming/__init__.py