+Vadim, git blame says that you last changed this api, do you have any guidance to debug this?

Vadim to PTAL/triage.

Does this happen reliably?

I don't know what can be deleting the file from global temp. It might be better not to use it if files are expected to live longer than few seconds. 

Consider using some named directory under api.path['slave_build'] (clearing it before starting the tests). That way we also don't leak files into global temp if recipes crashes or otherwise doesn't cleanup after itself.

Related issue  https://crbug.com/361343 .

The files are expected to live for the life of a bisect job, which can be anywhere from 10 minutes to 24 hours.

I will follow the suggestion of using a named location and clearing it before starting the runs.

Creating a named directory under api.m.path['slave_build']:
https://codereview.chromium.org/2479943003/
That didn't work.  (Still could not read the file from the new location)
https://build.chromium.org/p/tryserver.chromium.perf/builders/staging_win_perf_bisect/builds/36/steps/Compare%20samples/logs/stdio

Then we tried making a copy of the file and explicitly closing the file handle as per Ned's hunch that windows fails to open files in some situations where the file handle is still open by a different process. https://codereview.chromium.org/2481883002/ That failed as well.

The problem with the last was that the code that would copy the file failed to open the new file for writing, apparently the directory that would contain it had already been removed: https://build.chromium.org/p/tryserver.chromium.perf/builders/staging_win_perf_bisect/builds/41

+Pawel, it seems that remote_run uses something called RobustTemp, and I was wondering if something similar could be clearing these paths? The files that get removed from under us are currently on paths like: C:\b\rr\tmpqelviw\w\bisect_results\8c914fa3-4b6e-456a-80d8-13e79b9d5e79\results-chart.json  both the bisect_results and the uuid named directory are explicitly created by bisect for this purpose.

+Robbie, you've given me solid pointers before, do you have any guidance in this case?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/7420f0cb19fbb27d7a519248891f540e5a031f46

commit 7420f0cb19fbb27d7a519248891f540e5a031f46
Author: robertocn <robertocn@chromium.org>
Date: Tue Nov 08 01:11:09 2016

Moving code that clears the bisect_results directory to the right place.

This might have been the cause of the problems in the bug. The results
directory getting cleared once for every revision.

TBR=dtu@chromium.org
BUG= 662180 

Review-Url: https://codereview.chromium.org/2489473002

[modify] https://crrev.com/7420f0cb19fbb27d7a519248891f540e5a031f46/scripts/slave/recipe_modules/auto_bisect_staging/api.py
[modify] https://crrev.com/7420f0cb19fbb27d7a519248891f540e5a031f46/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/bad_config.json
[modify] https://crrev.com/7420f0cb19fbb27d7a519248891f540e5a031f46/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/basic_bisect_other_direction.json
[modify] https://crrev.com/7420f0cb19fbb27d7a519248891f540e5a031f46/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/basic_linux_bisect.json
[modify] https://crrev.com/7420f0cb19fbb27d7a519248891f540e5a031f46/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/basic_mac_bisect.json
[modify] https://crrev.com/7420f0cb19fbb27d7a519248891f540e5a031f46/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/basic_win32_bisect.json
[modify] https://crrev.com/7420f0cb19fbb27d7a519248891f540e5a031f46/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/basic_win64_bisect.json
[modify] https://crrev.com/7420f0cb19fbb27d7a519248891f540e5a031f46/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/failed_build.json
[modify] https://crrev.com/7420f0cb19fbb27d7a519248891f540e5a031f46/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/failed_buildbucket_get.json
[modify] https://crrev.com/7420f0cb19fbb27d7a519248891f540e5a031f46/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/failed_reference_range.json
[modify] https://crrev.com/7420f0cb19fbb27d7a519248891f540e5a031f46/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/failed_revision.json
[modify] https://crrev.com/7420f0cb19fbb27d7a519248891f540e5a031f46/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/no_repro.json
[modify] https://crrev.com/7420f0cb19fbb27d7a519248891f540e5a031f46/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/no_values.json
[modify] https://crrev.com/7420f0cb19fbb27d7a519248891f540e5a031f46/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/retest_bisect.json
[modify] https://crrev.com/7420f0cb19fbb27d7a519248891f540e5a031f46/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/return_code.json
[modify] https://crrev.com/7420f0cb19fbb27d7a519248891f540e5a031f46/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/v8_roll_bisect.json
[modify] https://crrev.com/7420f0cb19fbb27d7a519248891f540e5a031f46/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/v8_roll_bisect_bis.json
[modify] https://crrev.com/7420f0cb19fbb27d7a519248891f540e5a031f46/scripts/slave/recipes/bisection/android_bisect_staging.expected/failure_in_ref_range.json
[modify] https://crrev.com/7420f0cb19fbb27d7a519248891f540e5a031f46/scripts/slave/recipes/bisection/android_bisect_staging.expected/local_basic_recipe_basic_device.json
[modify] https://crrev.com/7420f0cb19fbb27d7a519248891f540e5a031f46/scripts/slave/recipes/bisection/android_bisect_staging.expected/local_basic_recipe_disconnected_device.json
[modify] https://crrev.com/7420f0cb19fbb27d7a519248891f540e5a031f46/scripts/slave/recipes/bisection/android_bisect_staging.expected/local_basic_recipe_failed_device.json
[modify] https://crrev.com/7420f0cb19fbb27d7a519248891f540e5a031f46/scripts/slave/recipes/bisection/android_bisect_staging.expected/local_basic_recipe_no_device.json

It seems the named directory for the results was getting cleared in the wrong place.

This seems to be working now.