New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 800413 link

Starred by 1 user

Issue metadata

Status: Verified
Owner:
Closed: Jan 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug

Blocking:
issue 800899



Sign in to add a comment

"swarming.py reproduce" crashes

Project Member Reported by kbr@chromium.org, Jan 9 2018

Issue description

Trying to reproduce this locally:
https://chromium-swarm.appspot.com/task?id=3af44789a4596410&refresh=10&show_raw=1

Running:
python tools\swarming_client\swarming.py reproduce -S chromium-swarm.appspot.com 3af44789a4596410

Throws exception:
Traceback (most recent call last):
  File "tools\swarming_client\swarming.py", line 1772, in <module>
    sys.exit(main(sys.argv[1:]))
  File "tools\swarming_client\swarming.py", line 1764, in main
    return dispatcher.execute(OptionParserSwarming(version=__version__), args)
  File "C:\Users\kbr\src\chrome\src\tools\swarming_client\third_party\depot_tools\subcommand.py", line 251, in execute
    return command(parser, args[1:])
  File "tools\swarming_client\swarming.py", line 1582, in CMDreproduce
    env_prefixes = properties['env']
KeyError: 'env'
Sending the crash report ... done.
Report URL: https://chromium-swarm.appspot.com/restricted/ereporter2/errors/6117571914366976
Process exited due to exception
'env'


There's a bug in the code which handles "env_prefixes". It fetches properties['env'] instead of properties['env_prefixes'].

 

Comment 1 by kbr@chromium.org, Jan 9 2018

Cc: zmo@chromium.org

Comment 2 by kbr@chromium.org, Jan 9 2018

Note: after patching in this fix locally it still crashes:

Traceback (most recent call last):
  File "tools\swarming_client\swarming.py", line 1772, in <module>
    sys.exit(main(sys.argv[1:]))
  File "tools\swarming_client\swarming.py", line 1764, in main
    return dispatcher.execute(OptionParserSwarming(version=__version__), args)
  File "C:\Users\kbr\src\chrome\src\tools\swarming_client\third_party\depot_tools\subcommand.py", line 251, in execute
    return command(parser, args[1:])
  File "tools\swarming_client\swarming.py", line 1584, in CMDreproduce
    for key, paths in env_prefixes.iteritems():
AttributeError: 'list' object has no attribute 'iteritems'
Sending the crash report ... done.
Report URL: https://chromium-swarm.appspot.com/restricted/ereporter2/errors/5637427017809920
Process exited due to exception
'list' object has no attribute 'iteritems'

Cc: -mar...@chromium.org
Labels: -OS-Windows
Owner: mar...@chromium.org
Status: Assigned (was: Untriaged)
yeah, fixing.
Project Member

Comment 4 by bugdroid1@chromium.org, Jan 9 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-py.git/+/7f12a449ca611184098f6fc4bf9b8c09fae1eb62

commit 7f12a449ca611184098f6fc4bf9b8c09fae1eb62
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Tue Jan 09 19:11:12 2018

client: fix swarming.py reproduce with env_prefixes.

- Remove encoding, it's unnecessary now that subprocess42 is SUPER SMART. :)
- Add a small unit test.

R=iannucci@chromium.org

Bug:  800413 
Change-Id: I4199c66146efb61bcb591714bae9350838da2f13
Reviewed-on: https://chromium-review.googlesource.com/857937
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org>

[modify] https://crrev.com/7f12a449ca611184098f6fc4bf9b8c09fae1eb62/client/swarming.py
[modify] https://crrev.com/7f12a449ca611184098f6fc4bf9b8c09fae1eb62/client/tests/swarming_test.py

Comment 5 by kbr@chromium.org, Jan 9 2018

Thanks for this fix but it's still not working for me.

python tools\swarming_client\swarming.py reproduce -S chromium-swarm.appspot.com 3af44789a4596410

265 files remaining...
227 files remaining...
189 files remaining...
166 files remaining...
148 files remaining...
99 files remaining...
Traceback (most recent call last):
  File "tools\swarming_client\swarming.py", line 1786, in <module>
    sys.exit(main(sys.argv[1:]))
  File "tools\swarming_client\swarming.py", line 1778, in main
    return dispatcher.execute(OptionParserSwarming(version=__version__), args)
  File "C:\Users\kbr\src\chrome\src\tools\swarming_client\third_party\depot_tools\subcommand.py", line 251, in execute
    return command(parser, args[1:])
  File "tools\swarming_client\swarming.py", line 1651, in CMDreproduce
    return subprocess.call(command + extra_args, env=env, cwd=workdir)
  File "C:\python_27_amd64\files\lib\subprocess.py", line 493, in call
    return Popen(*popenargs, **kwargs).wait()
  File "C:\python_27_amd64\files\lib\subprocess.py", line 679, in __init__
    errread, errwrite)
  File "C:\python_27_amd64\files\lib\subprocess.py", line 896, in _execute_child
    startupinfo)
TypeError: environment can only contain strings
Sending the crash report ... done.
Report URL: https://chromium-swarm.appspot.com/restricted/ereporter2/errors/5673908839120896
Process exited due to exception
environment can only contain strings

Output of "set" in my Command Prompt (Google internal only in case it has sensitive information):
https://drive.google.com/file/d/1aaqTnIXb1s0lNw8RY-xPBC-cSzDHIWow/view?usp=sharing

Project Member

Comment 6 by bugdroid1@chromium.org, Jan 10 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-py.git/+/0795e6510d23a472da796420c93c1e63dbaf2923

commit 0795e6510d23a472da796420c93c1e63dbaf2923
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Wed Jan 10 19:28:23 2018

client: use subproces42 instead of subprocess

subprocess has a lot of issues, one being lack of unicode support.

The rest of the code uses subprocess42 but for an unknown reason, 'swarming.py
reproduce' did not. Obviously, 'M-A from 2015' is at fault.

R=iannucci@chromium.org

Bug:  800413 
Change-Id: I3f8b6e3e07bee4156ba217a6e930371b62d0bb3d
Reviewed-on: https://chromium-review.googlesource.com/860260
Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/0795e6510d23a472da796420c93c1e63dbaf2923/client/swarming.py
[modify] https://crrev.com/0795e6510d23a472da796420c93c1e63dbaf2923/client/tests/swarming_test.py

Comment 7 by kbr@chromium.org, Jan 10 2018

Status: Verified (was: Assigned)
Thanks for the fixes. Confirming that swarming.py reproduce now runs the test, though it doesn't pass because some of the arguments cause the test to try to write to a nonexistent directory:

[1/1] gpu_tests.maps_integration_test.MapsIntegrationTest.Maps_maps passed 12.0930s
INFO:root:Closing browser (pid=16360) ...
[16360:18556:0110/115100.598:INFO:chrome_cryptauth_service.cc(233)] Profile is not authenticated yet; waiting before starting CryptAuth managers.
[16360:18556:0110/115100.741:WARNING:url_request_context_getter.cc(43)] URLRequestContextGetter leaking due to no owning thread.
INFO:root:Successfully shut down browser cooperatively
INFO:root:Browser is closed.
1 test passed in 21.4s, 0 skipped, 0 failures.
INFO:root:Try printing formatted exception: <type 'exceptions.IOError'> [Errno 2] No such file or directory: 'out\\output.json' <traceback object at 0x00000000047AA408>

Traceback (most recent call last):
  <module> at C:\Users\kbr\src\chrome\src\work\third_party\catapult\telemetry\telemetry\testing\run_browser_tests.py:359
    ret_code = RunTests(sys.argv[1:])
  RunTests at C:\Users\kbr\src\chrome\src\work\third_party\catapult\telemetry\telemetry\testing\run_browser_tests.py:328
    ret, _, _ = runner.run()
  run at C:\Users\kbr\src\chrome\src\work\third_party\catapult\third_party\typ\typ\runner.py:189
    self._write(self.args.write_full_results_to, full_results)
  _write at C:\Users\kbr\src\chrome\src\work\third_party\catapult\third_party\typ\typ\runner.py:656
    self.host.write_text_file(path, json.dumps(obj, indent=2) + '\n')
  write_text_file at C:\Users\kbr\src\chrome\src\work\third_party\catapult\third_party\typ\typ\host.py:179
    return self._write(path, contents, mode='w')
  _write at C:\Users\kbr\src\chrome\src\work\third_party\catapult\third_party\typ\typ\host.py:185
    with open(path, mode) as f:
IOError: [Errno 2] No such file or directory: 'out\\output.json'

Locals:
  contents : '{\n  "version": 3, \n  "interrupted": false, \n  "path_delimiter": ".", \n  "seconds_since_epoch": 1515613860, \n  "num_failures_by_type": {\n    "FAIL": 0, \n    "PASS": 1, \n    "SKIP": 0\n  }, \n  "tests": {\n    "gpu_tests": {\n      "maps_integration_test": {\n        "MapsIntegrationTest": {\n          "Maps_maps": {\n            "actual": "PASS", \n            "times": [\n              12.093\n            ], \n            "expected": "PASS"\n          }\n        }\n      }\n    }\n  }\n}\n'
  mode     : 'w'
  path     : 'out\\output.json'



Closing as Verified -- if we think "swarming.py reproduce" needs more features I'll file a follow-on bug.

Comment 8 by kbr@chromium.org, Jan 10 2018

Blocking: 800899
Project Member

Comment 9 by bugdroid1@chromium.org, Jan 11 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/72a97881dd6b6a6e0147d1cad984bdada22201c0

commit 72a97881dd6b6a6e0147d1cad984bdada22201c0
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Thu Jan 11 15:15:45 2018

Roll src/tools/swarming_client/ 36e0979a4..88229872d (3 commits)

https://chromium.googlesource.com/infra/luci/client-py.git/+log/36e0979a4f2f..88229872dd17

$ git log 36e0979a4..88229872d --date=short --no-merges --format='%ad %ae %s'
2018-01-10 maruel client: fix incorrect 'fix' in 7efbe3ad731c875.
2018-01-10 maruel client: create the output directory directory when ${ISOLATED_OUTDIR} is used
2018-01-10 maruel client: use subproces42 instead of subprocess

Created with:
  roll-dep src/tools/swarming_client

R=kbr@chromium.org

Bug:  800413 ,  800899 
Change-Id: Ic9372e8ba63fb5be66902bc9a4c6658f9ea307df
Reviewed-on: https://chromium-review.googlesource.com/861123
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org>
Cr-Commit-Position: refs/heads/master@{#528627}
[modify] https://crrev.com/72a97881dd6b6a6e0147d1cad984bdada22201c0/DEPS

Sign in to add a comment