New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 731095 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Jun 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Feature

Blocked on:
issue 767026



Sign in to add a comment

Change swarming collect recipe to not print stdout

Project Member Reported by mar...@chromium.org, Jun 8 2017

Issue description

When a Swarming task's stdout is collected, it is effectively output twice, as the annotator reprints it another time. In practice we do not need the output anymore, this used to be so users wouldn't be confused in buildbot world but this is just overhead for LUCI tasks or reentrant task.

In particular this can become a real issue when the total output is larger than 16Mb, which affects the task.

AIs:
- Add a new flag in swarming.py collect to not fetch stdout[1].
- Update recipe_module/swarming/api.py to use this new flag.

[1]  While at it, we may want to not return the task process exit code either.
 

Comment 1 by rmis...@google.com, Jun 8 2017

"In particular this can become a real issue when the total output is larger than 16Mb, which affects the task."

Example affected task: https://chromium-swarm.appspot.com/task?id=36921085af83a110&refresh=0&request_detail=true&wide_logs=1
Project Member

Comment 2 by bugdroid1@chromium.org, Sep 7 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-py.git/+/adaf48426621c897d1bd094cd73906d7bfab3d50

commit adaf48426621c897d1bd094cd73906d7bfab3d50
Author: Tim 'mithro' Ansell <tansell@chromium.org>
Date: Thu Sep 07 23:41:56 2017

swarming.py: Add --task-output-stdout option

Enables the ability to control where the stdout of a task ends up going.
The options are;

 * Don't download / get at all.
 * Only output to the json summary file.
 * Only output to the console.
 * Continue to output in all of the above (default).

At the moment when using swarming on the build bots we end up with;
 - stdout in swarming
 - stdout in the collect_task output
 - stdout inside the summary.json file

For Layout Tests where the stdout is *long* this makes recipes take
significant time in just parsing the summary.json output.

BUG= 731095 

Change-Id: I0fa7228a5be21820b3089306f6e9c8646983159c
Reviewed-on: https://chromium-review.googlesource.com/654020
Reviewed-by: Tim 'mithro' Ansell <tansell@chromium.org>
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
Commit-Queue: Tim 'mithro' Ansell <tansell@chromium.org>

[modify] https://crrev.com/adaf48426621c897d1bd094cd73906d7bfab3d50/client/swarming.py
[modify] https://crrev.com/adaf48426621c897d1bd094cd73906d7bfab3d50/client/tests/swarming_test.py
[modify] https://crrev.com/adaf48426621c897d1bd094cd73906d7bfab3d50/client/tools/swarming_load_test_client.py

Owner: tansell@chromium.org
Status: Started (was: Available)
With  https://chromium-review.googlesource.com/654020 we now have a way to do this;

 swarming.py --task-output-stdout=(console|json|none|both)

I'm planning on deploying this feature for Layout Tests on swarming soon.
Project Member

Comment 4 by bugdroid1@chromium.org, Sep 8 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/519e4311ba5a076a7102b48e9efdbdcda2ed06e1

commit 519e4311ba5a076a7102b48e9efdbdcda2ed06e1
Author: Tim 'mithro' Ansell <tansell@chromium.org>
Date: Fri Sep 08 02:26:19 2017

Roll src/tools/swarming_client/ 72b6a2dc6..5e8001d9a (7 commits)

https://chromium.googlesource.com/infra/luci/client-py.git/+log/72b6a2dc6046..5e8001d9a710

$ git log 72b6a2dc6..5e8001d9a --date=short --no-merges --format='%ad %ae %s'
2017-09-08 tansell swarming.py: Add --task-output-stdout option
2017-09-07 justincohen tools/run_on_bots: Enable running a task in batches.
2017-08-31 tansell Removing arfile.
2017-08-31 tansell tools/run_on_bots: Enable giving a hash instead of script.
2017-08-30 aludwin Implement task updates in gRPC proxy
2017-08-29 justincohen Add support for no bots found in run_on_bots.py
2017-08-17 maruel More improvements to run_on_bots.py

Created with:
  roll-dep src/tools/swarming_client

R=maruel@chromium.org,dpranke@chromium.org
BUG= 731095 

Change-Id: I0d53fd7f17d3b24cc37a24f230a85c7b5b3797da
Reviewed-on: https://chromium-review.googlesource.com/656401
Commit-Queue: Tim 'mithro' Ansell <tansell@chromium.org>
Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org>
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#500489}
[modify] https://crrev.com/519e4311ba5a076a7102b48e9efdbdcda2ed06e1/DEPS

Comment 5 by kbr@chromium.org, Oct 13 2017

Blockedon: 767026

Comment 6 by mar...@chromium.org, Jun 21 2018

Status: Fixed (was: Started)

Sign in to add a comment