New issue
Advanced search Search tips

Issue 781421 link

Starred by 3 users

Issue metadata

Status: Fixed
Owner:
Closed: Nov 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 1
Type: Bug-Regression



Sign in to add a comment

All shards of telemetry_perf_unittests failing with internal Swarming failure

Project Member Reported by kbr@chromium.org, Nov 3 2017

Issue description

In this tryjob:
https://ci.chromium.org/buildbot/tryserver.chromium.win/win7_chromium_rel_ng/34953

of this CL which should be unrelated:
https://chromium-review.googlesource.com/c/chromium/src/+/754019

All shards of telemetry_perf_unittests failed with an internal Swarming failure.

The shard exited with code 0:
https://chromium-swarm.appspot.com/task?id=399c909198029f10&refresh=10&show_raw=1

so there doesn't seem a good reason why it failed. Can someone help?

Note I'm not seeing evidence of this elsewhere on the bot:
https://ci.chromium.org/buildbot/tryserver.chromium.win/win7_chromium_rel_ng/?limit=200

so it might be my patch, but I don't see how.

Thanks.

 
It looks like the collect step is failing with 

Traceback (most recent call last):
  File "C:\b\c\b\win\src\tools\swarming_client\swarming.py", line 1727, in <module>
    sys.exit(main(sys.argv[1:]))
  File "C:\b\c\b\win\src\tools\swarming_client\swarming.py", line 1719, in main
    return dispatcher.execute(OptionParserSwarming(version=__version__), args)
  File "C:\b\c\b\win\src\tools\swarming_client\third_party\depot_tools\subcommand.py", line 251, in execute
    return command(parser, args[1:])
  File "C:\b\c\b\win\src\tools\swarming_client\swarming.py", line 1324, in CMDcollect
    options.perf)
  File "C:\b\c\b\win\src\tools\swarming_client\swarming.py", line 768, in collect
    "console" in task_output_stdout).encode(
  File "C:\b\c\b\win\src\tools\swarming_client\swarming.py", line 717, in decorate_shard_output
    metadata.get('output', '').rstrip(),
AttributeError: 'dict' object has no attribute 'rstrip'
Cc: mar...@chromium.org
https://apis-explorer.appspot.com/apis-explorer/?base=https://chromium-swarm.appspot.com/_ah/api#p/swarming/v1/swarming.task.result?task_id=399c909198029f10&_h=1& shows what I believe the swarming.py script is doing to get the metadata object which is being referenced in the stack trace. It doesn't have an output element though.
The CL in #4 looks to be very related to the exception we're seeing.
That CL landed this morning, and https://ci.chromium.org/buildbot/tryserver.chromium.win/win7_chromium_rel_ng/34953 started at 1:30 PM. So it shouldn't be failing like that. I don't think that CL has been rolled out. Do we need to roll the swarming client in chromium src?
Owner: martiniss@chromium.org
Status: Assigned (was: Untriaged)
I'll do a roll.
https://chromium-review.googlesource.com/c/chromium/src/+/753138 is the roll. Should I NOTRY it? I don't know how much this is affecting the bot. The bot doesn't seem to be all purple.
No.
Project Member

Comment 11 by bugdroid1@chromium.org, Nov 3 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-py.git/+/1c1c8f97645c9608fac1f74401eddf162b7df2e0

commit 1c1c8f97645c9608fac1f74401eddf162b7df2e0
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Fri Nov 03 22:39:06 2017

Fix handling of empty /stdout result.

If /stdout returns an empty dict (which happens if there's no stdout),
result['output'] becomes empty dict too, which later trips .rstrip(...) call.

BUG= 781421 
R=martiniss@chromium.org

Change-Id: I9b0086b6686bdc5073fbe30452a2d72532119469
Reviewed-on: https://chromium-review.googlesource.com/754284
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/1c1c8f97645c9608fac1f74401eddf162b7df2e0/client/swarming.py

Project Member

Comment 12 by bugdroid1@chromium.org, Nov 4 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a075effafacc793a2902cf534fed98f0d4ffc5d6

commit a075effafacc793a2902cf534fed98f0d4ffc5d6
Author: Stephen Martinis <martiniss@chromium.org>
Date: Sat Nov 04 01:31:13 2017

Roll src/tools/swarming_client/ 5da404cf3..6fd3c7b6e (2 commits)

https://chromium.googlesource.com/infra/luci/client-py.git/+log/5da404cf35b6..6fd3c7b6eb7c

$ git log 5da404cf3..6fd3c7b6e --date=short --no-merges --format='%ad %ae %s'
2017-11-03 vadimsh Fix handling of empty /stdout result.
2017-11-03 maruel swarming.py: fix regression in adaf484266, improve decorated ouput

Created with:
  roll-dep src/tools/swarming_client
R=vadimsh@chromium.org

Bug:  781421 
Change-Id: I8240c90e86a092e5deaf4d6b737ac95fea962c15
Reviewed-on: https://chromium-review.googlesource.com/754293
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Cr-Commit-Position: refs/heads/master@{#514006}
[modify] https://crrev.com/a075effafacc793a2902cf534fed98f0d4ffc5d6/DEPS

Status: Fixed (was: Assigned)
This seems to have been fixed, AFAIK.

Sign in to add a comment