New issue
Advanced search Search tips

Issue 800899 link

Starred by 1 user

Issue metadata

Status: Verified
Owner:
Closed: Jan 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 2
Type: Bug

Blocked on:
issue 800413

Blocking:
issue 801046



Sign in to add a comment

"swarming.py reproduce" fails because of nonexistent directories on disk

Project Member Reported by kbr@chromium.org, Jan 10 2018

Issue description

To reproduce, on Windows, run:

python tools\swarming_client\swarming.py reproduce -S chromium-swarm.appspot.com 3af44789a4596410

The test downloads and runs successfully, but the harness throws an exception because it attempts to write a JSON file to a nonexistent directory:

[1/1] gpu_tests.maps_integration_test.MapsIntegrationTest.Maps_maps passed 12.0930s
INFO:root:Closing browser (pid=16360) ...
[16360:18556:0110/115100.598:INFO:chrome_cryptauth_service.cc(233)] Profile is not authenticated yet; waiting before starting CryptAuth managers.
[16360:18556:0110/115100.741:WARNING:url_request_context_getter.cc(43)] URLRequestContextGetter leaking due to no owning thread.
INFO:root:Successfully shut down browser cooperatively
INFO:root:Browser is closed.
1 test passed in 21.4s, 0 skipped, 0 failures.
INFO:root:Try printing formatted exception: <type 'exceptions.IOError'> [Errno 2] No such file or directory: 'out\\output.json' <traceback object at 0x00000000047AA408>

Traceback (most recent call last):
  <module> at C:\Users\kbr\src\chrome\src\work\third_party\catapult\telemetry\telemetry\testing\run_browser_tests.py:359
    ret_code = RunTests(sys.argv[1:])
  RunTests at C:\Users\kbr\src\chrome\src\work\third_party\catapult\telemetry\telemetry\testing\run_browser_tests.py:328
    ret, _, _ = runner.run()
  run at C:\Users\kbr\src\chrome\src\work\third_party\catapult\third_party\typ\typ\runner.py:189
    self._write(self.args.write_full_results_to, full_results)
  _write at C:\Users\kbr\src\chrome\src\work\third_party\catapult\third_party\typ\typ\runner.py:656
    self.host.write_text_file(path, json.dumps(obj, indent=2) + '\n')
  write_text_file at C:\Users\kbr\src\chrome\src\work\third_party\catapult\third_party\typ\typ\host.py:179
    return self._write(path, contents, mode='w')
  _write at C:\Users\kbr\src\chrome\src\work\third_party\catapult\third_party\typ\typ\host.py:185
    with open(path, mode) as f:
IOError: [Errno 2] No such file or directory: 'out\\output.json'

Locals:
  contents : '{\n  "version": 3, \n  "interrupted": false, \n  "path_delimiter": ".", \n  "seconds_since_epoch": 1515613860, \n  "num_failures_by_type": {\n    "FAIL": 0, \n    "PASS": 1, \n    "SKIP": 0\n  }, \n  "tests": {\n    "gpu_tests": {\n      "maps_integration_test": {\n        "MapsIntegrationTest": {\n          "Maps_maps": {\n            "actual": "PASS", \n            "times": [\n              12.093\n            ], \n            "expected": "PASS"\n          }\n        }\n      }\n    }\n  }\n}\n'
  mode     : 'w'
  path     : 'out\\output.json'


Is it feasible to fix this? I realize that this would probably add harness-specific knowledge to "swarming.py reproduce", which is undesirable.

 
Project Member

Comment 1 by bugdroid1@chromium.org, Jan 10 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-py.git/+/7efbe3ad731c875a86d40d134ecc9d0a91dd226e

commit 7efbe3ad731c875a86d40d134ecc9d0a91dd226e
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Wed Jan 10 21:05:55 2018

client: create the output directory directory when ${ISOLATED_OUTDIR} is used

Otherwise some test will be surprised by the fact that the output directory
doesn't exist and needs to be created. It is always created in advance in a
Swarming task.

R=iannucci@chromium.org

Bug:  800899 
Change-Id: Iff330cab2b1042845177e3789b3bc38a18f05166
Reviewed-on: https://chromium-review.googlesource.com/860628
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org>

[modify] https://crrev.com/7efbe3ad731c875a86d40d134ecc9d0a91dd226e/client/swarming.py

Comment 2 by kbr@chromium.org, Jan 10 2018

Still failing with top-of-tree swarming_client:

python tools\swarming_client\swarming.py reproduce -S chromium-swarm.appspot.com 3af44789a4596410

[1/1] gpu_tests.maps_integration_test.MapsIntegrationTest.Maps_maps passed 11.7250s
INFO:root:Closing browser (pid=19236) ...
[19236:968:0110/132139.632:INFO:chrome_cryptauth_service.cc(233)] Profile is not authenticated yet; waiting before starting CryptAuth managers.
[19236:968:0110/132139.704:WARNING:url_request_context_getter.cc(43)] URLRequestContextGetter leaking due to no owning thread.
INFO:root:Successfully shut down browser cooperatively
INFO:root:Browser is closed.
1 test passed in 20.8s, 0 skipped, 0 failures.
INFO:root:Try printing formatted exception: <type 'exceptions.IOError'> [Errno 2] No such file or directory: 'out\\output.json' <traceback object at 0x000000000467A408>

Traceback (most recent call last):
  <module> at C:\Users\kbr\src\chrome\src\work\third_party\catapult\telemetry\telemetry\testing\run_browser_tests.py:359
    ret_code = RunTests(sys.argv[1:])
  RunTests at C:\Users\kbr\src\chrome\src\work\third_party\catapult\telemetry\telemetry\testing\run_browser_tests.py:328
    ret, _, _ = runner.run()
  run at C:\Users\kbr\src\chrome\src\work\third_party\catapult\third_party\typ\typ\runner.py:189
    self._write(self.args.write_full_results_to, full_results)
  _write at C:\Users\kbr\src\chrome\src\work\third_party\catapult\third_party\typ\typ\runner.py:656
    self.host.write_text_file(path, json.dumps(obj, indent=2) + '\n')
  write_text_file at C:\Users\kbr\src\chrome\src\work\third_party\catapult\third_party\typ\typ\host.py:179
    return self._write(path, contents, mode='w')
  _write at C:\Users\kbr\src\chrome\src\work\third_party\catapult\third_party\typ\typ\host.py:185
    with open(path, mode) as f:
IOError: [Errno 2] No such file or directory: 'out\\output.json'

Locals:
  contents : '{\n  "version": 3, \n  "interrupted": false, \n  "path_delimiter": ".", \n  "seconds_since_epoch": 1515619299, \n  "num_failures_by_type": {\n    "FAIL": 0, \n    "PASS": 1, \n    "SKIP": 0\n  }, \n  "tests": {\n    "gpu_tests": {\n      "maps_integration_test": {\n        "MapsIntegrationTest": {\n          "Maps_maps": {\n            "actual": "PASS", \n            "times": [\n              11.725\n            ], \n            "expected": "PASS"\n          }\n        }\n      }\n    }\n  }\n}\n'
  mode     : 'w'
  path     : 'out\\output.json'

Comment 3 by mar...@chromium.org, Jan 10 2018

Duh, it's because the path is relative. Fixing (again).
Project Member

Comment 4 by bugdroid1@chromium.org, Jan 11 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-py.git/+/3ef1ca5462c02220be2a203aab42bd873936f8c9

commit 3ef1ca5462c02220be2a203aab42bd873936f8c9
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Thu Jan 11 02:20:47 2018

client: fix incorrect 'fix' in 7efbe3ad731c875.

Add unit test to stop 'trying' and actually 'fixing'.
This enables correct 'swarming.py reproduce' behavior.

R=iannucci@chromium.org

Bug:  800899 
Change-Id: Ie2897c3ad5dafdb8232a814ac331737680e485cb
Reviewed-on: https://chromium-review.googlesource.com/861007
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org>

[modify] https://crrev.com/3ef1ca5462c02220be2a203aab42bd873936f8c9/client/swarming.py
[modify] https://crrev.com/3ef1ca5462c02220be2a203aab42bd873936f8c9/client/tests/swarming_test.py

Comment 5 by kbr@chromium.org, Jan 11 2018

Status: Verified (was: Assigned)
Thanks for the fix. Verified.

I'm afraid there is one more bug. Will file separately.

Comment 6 by kbr@chromium.org, Jan 11 2018

Blocking: 801046
Project Member

Comment 7 by bugdroid1@chromium.org, Jan 11 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/72a97881dd6b6a6e0147d1cad984bdada22201c0

commit 72a97881dd6b6a6e0147d1cad984bdada22201c0
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Thu Jan 11 15:15:45 2018

Roll src/tools/swarming_client/ 36e0979a4..88229872d (3 commits)

https://chromium.googlesource.com/infra/luci/client-py.git/+log/36e0979a4f2f..88229872dd17

$ git log 36e0979a4..88229872d --date=short --no-merges --format='%ad %ae %s'
2018-01-10 maruel client: fix incorrect 'fix' in 7efbe3ad731c875.
2018-01-10 maruel client: create the output directory directory when ${ISOLATED_OUTDIR} is used
2018-01-10 maruel client: use subproces42 instead of subprocess

Created with:
  roll-dep src/tools/swarming_client

R=kbr@chromium.org

Bug:  800413 ,  800899 
Change-Id: Ic9372e8ba63fb5be66902bc9a4c6658f9ea307df
Reviewed-on: https://chromium-review.googlesource.com/861123
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org>
Cr-Commit-Position: refs/heads/master@{#528627}
[modify] https://crrev.com/72a97881dd6b6a6e0147d1cad984bdada22201c0/DEPS

Sign in to add a comment