New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 875089 link

Starred by 6 users

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Sep 24
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug-Regression



Sign in to add a comment

led get-build stopped working after automatic cache task slices were enabled

Project Member Reported by serg...@chromium.org, Aug 16

Issue description

E.g. when running

  led get-build 8938012618933164688 | led launch

the resulting task fails immediately with NO_RESOURCE:

  https://ci.chromium.org/swarming/task/3f5db6f78f375110?server=chrome-swarming.appspot.com
 
Labels: -Restrict-View-Google
A workaround for me is to pipe in:
led edit -d caches ''
I thought I fixed this; can you confirm you're using the latest version of led?
How do I check what version is the latest?
Currently deployed version (in depot_tools) is `git_revision:9a931a5307c46b16b1c12e01e8239d4a73830b89`. If you run `led version`, you should see one of these instance ids, depending on your platform:

$ cipd resolve infra/tools/luci/led/ -version git_revision:9a931a5307c46b16b1c12e01e8239d4a73830b89
Packages:
  infra/tools/luci/led/linux-386:iIpQ5iwa7WGuykl4VFgAGmS2HnyVCUAwz26294PUAq4C
  infra/tools/luci/led/linux-amd64:ERsRSgZMwHDgzcZe2527X_hK6aAbyGBsfuNvsIKSppEC
  infra/tools/luci/led/linux-arm64:3tXD-fay5juhpK3mUH0ToIMLcW3U9ZXs-sjjv5F18_IC
  infra/tools/luci/led/linux-armv6l:twLjhS4AGVQnOjKxYSTkTfIUaVfT4l0NwWcGkW0zDnAC
  infra/tools/luci/led/linux-mips64:wbddq9_7ZFc7Vg2hu4cQqvPHd3edbdpvdpN-7ZlFIZYC
  infra/tools/luci/led/linux-mips64le:_3_uv-hjsh-29CecuNTsjkGkzVOGu7mW9eXggrog4b4C
  infra/tools/luci/led/linux-mipsle:pY10citugyciz_H6UFYJ-cAgeU1lkq41sm0yCJJJBwEC
  infra/tools/luci/led/linux-ppc64:0WsLUOPPDjJCk_j6erufPMHKhCurg1Ha-gYr7XS_E4kC
  infra/tools/luci/led/linux-ppc64le:KUve06pRLqtllRI-sPo6WzrUo5KROm6bymk6QcSuWc0C
  infra/tools/luci/led/linux-s390x:dOAtQ73ZmJKMSKc7khOs9S_6I8wooQTRoPgTtAbTuqYC
  infra/tools/luci/led/mac-amd64:CViiquK9ekBErdpJfg3yOGRF8Td8NEhADRlbK0SlANQC
  infra/tools/luci/led/windows-386:gsuQsh_HYEQlNAL5XZu76_GoOlue9LYQMo1lBD7ENJ8C
  infra/tools/luci/led/windows-amd64:NQC9vH3suHP2sFpYaluO75AJqB3UMCpPRDqRS9-cMIIC


Neither version 'latest' nor specifying that git_revision fixes the issue.
This command produced a swarming task that has caches specified and won't schedule:
led get-builder 'luci.fuchsia.ci:zircon-x64-clang-qemu_kvm' | led edit-recipe-bundle | led edit -p 'tryjob=true' | led launch

https://chromium-swarm.appspot.com/task?id=3f873efa62fa7410&refresh=10
Cc: phosek@chromium.org
Ok, phosek@google.com ran into this and I realized what the issue is :). led drops everything after slice 0. Looks like I need to rework led a bit.
I am curious what the issue was. Can you share more details please?
Cc: -iannucci@chromium.org
Labels: Type-Bug-Regression
Owner: iannucci@chromium.org
Status: Assigned (was: Untriaged)
It's a combination:
- Buildbucket started using multiple task slices (vs only one before).
- LED assumed only one task slice and used the first one (which is the warm cache path).
Workaround in #2 works great for this. One tiny caveat: for some asks it may also be necessary to edit the job manually to increase the expiration timeout - I could not find a led command to do this without manual editing. Without this change, led task expires in just 2 minutes if no bots are available atm.
setting expiration_secs from command line

led get-whatever | jq '.systemland.swarming_task.task_slices[0].expiration_secs = "600"'
Cc: garymm@google.com
Planning to fix this tomorrow
Later in the day than I hoped, but CL: https://chromium-review.googlesource.com/c/infra/infra/+/1220602
Project Member

Comment 15 by bugdroid1@chromium.org, Sep 12

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/b71c243dd3e086edf914e1bd3411c3f557505829

commit b71c243dd3e086edf914e1bd3411c3f557505829
Author: Robert Iannucci <iannucci@chromium.org>
Date: Wed Sep 12 18:02:29 2018

[led] Fix led to properly handle tasks with many slices.

This changes the JobDefinition intermediate JSON format to handle
multiple slices. Incoming tasks which didn't use slices are converted
to tasks with a single slice.

`led edit*` operations now apply to every slice in the JobDefinition.

R=hinoka@chromium.org, nodir@chromium.org, tandrii@chromium.org

Bug:  875089 
Change-Id: Iaa60bd16ec62c14e87337f5af4179f324302a0ca
Reviewed-on: https://chromium-review.googlesource.com/1220602
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Cr-Commit-Position: refs/heads/master@{#17535}
[modify] https://crrev.com/b71c243dd3e086edf914e1bd3411c3f557505829/go/src/infra/tools/led/job_userland.go
[modify] https://crrev.com/b71c243dd3e086edf914e1bd3411c3f557505829/go/src/infra/tools/led/job.go
[modify] https://crrev.com/b71c243dd3e086edf914e1bd3411c3f557505829/go/src/infra/tools/led/cmd_get_swarm.go
[modify] https://crrev.com/b71c243dd3e086edf914e1bd3411c3f557505829/go/src/infra/tools/led/job_edit.go
[modify] https://crrev.com/b71c243dd3e086edf914e1bd3411c3f557505829/go/src/infra/tools/led/job_def.go
[modify] https://crrev.com/b71c243dd3e086edf914e1bd3411c3f557505829/go/src/infra/tools/led/job_systemland.go

Project Member

Comment 16 by bugdroid1@chromium.org, Sep 13

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/dfce68bcddba7508d74016405a95e7865cbf47ba

commit dfce68bcddba7508d74016405a95e7865cbf47ba
Author: Robert Iannucci <iannucci@chromium.org>
Date: Thu Sep 13 01:23:56 2018

[led] Roll led to handle jobs with multiple swarming TaskSlices.

R=hinoka@chromium.org, nodir@chromium.org, tandrii@chromium.org

Bug:  875089 
Change-Id: I6abb7a5d0aebf53c4c3ab6be309ed43e652278c0
Reviewed-on: https://chromium-review.googlesource.com/1222883
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
Auto-Submit: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Ryan Tseng <hinoka@chromium.org>

[modify] https://crrev.com/dfce68bcddba7508d74016405a95e7865cbf47ba/cipd_manifest.txt

Project Member

Comment 17 by bugdroid1@chromium.org, Sep 13

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/de2a0a7df36d477a8a6398488c7c8d785e90373d

commit de2a0a7df36d477a8a6398488c7c8d785e90373d
Author: depot-tools-chromium-autoroll <depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Thu Sep 13 04:39:15 2018

Roll src/third_party/depot_tools 0425ebd2b395..dfce68bcddba (1 commits)

https://chromium.googlesource.com/chromium/tools/depot_tools.git/+log/0425ebd2b395..dfce68bcddba


git log 0425ebd2b395..dfce68bcddba --date=short --no-merges --format='%ad %ae %s'
2018-09-13 iannucci@chromium.org [led] Roll led to handle jobs with multiple swarming TaskSlices.


Created with:
  gclient setdep -r src/third_party/depot_tools@dfce68bcddba

The AutoRoll server is located here: https://autoroll.skia.org/r/depot-tools-chromium-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.



BUG= chromium:875089 
TBR=agable@chromium.org

Change-Id: I55fa403d9f88e95b2c70380bde69e6d1cf22fd9b
Reviewed-on: https://chromium-review.googlesource.com/1222202
Reviewed-by: depot-tools-chromium-autoroll <depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: depot-tools-chromium-autoroll <depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#590925}
[modify] https://crrev.com/de2a0a7df36d477a8a6398488c7c8d785e90373d/DEPS

Looks like recent changes broke led in a different way:

  $ led get-build 8935510276360715312 | led edit -S chrome-swarming.appspot.com | led edit-recipe-bundle | led launch
  [I 2018-09-13 11:35:25] getting build definition
  [I 2018-09-13 11:35:25] bundling recipes
  [I 2018-09-13 11:35:26] getting build definition: done
  [I 2018-09-13 11:35:26] getting task definition: "3feb0bf0dda5ba10"
  [I 2018-09-13 11:35:26] getting task definition: done
  [W 2018-09-13 11:35:26] waiting for JobDefinition on stdin...
  [I 2018-09-13 11:35:27] bundling recipes: done
  [I 2018-09-13 11:35:27] isolating recipes
  [found] [hashed/size/to hash] [looked up/to lookup] [uploaded/size/to upload/size]
  [1778] [1778/12.6Mib/1778] [1728/1778] [0/0b/0/0b] 1.6s
  [I 2018-09-13 11:35:29] isolating recipes: done
  [found] [hashed/size/to hash] [looked up/to lookup] [uploaded/size/to upload/size]
  [I 2018-09-13 11:35:29] building swarming task
  [I 2018-09-13 11:35:29] building swarming task: done
  [I 2018-09-13 11:35:29] launching swarming task
  [E 2018-09-13 11:35:29] original error: googleapi: Error 400: missing expiration_secs

I've checked the source of the generated task (after led edit-recipe-bundle) and it has two slices, both with expiration_secs. Not sure why it fails...
Same error for

$ led get-builder luci.v8-internal.try:v8_pixel2_perf_try | led edit-cr-cl https://chromium-review.googlesource.com/c/v8/v8/+/1204092 | led edit -S chrome-swarming.appspot.com | led edit-recipe-bundle | led launch
This is also affecting me. Same error as sergiyb.

"""
original error: googleapi: Error 400: missing expiration_secs
"""
Same here after update_depot_tools.

My workaround is to install an old version of led:
cipd install infra/tools/luci/led/mac-amd64 CViiquK9ekBErdpJfg3yOGRF8Td8NEhADRlbK0SlANQC
Issue 887552 has been merged into this issue.
I've tried workaround from #21, but still get "Error 400: missing expiration_secs". I've done it on linux, so I've used the following command:

  cipd install infra/tools/luci/led/linux-amd64 gT9UxxL939aCQmOX-Mwn4G3FtVUedMSntDnf6Cw2yiQC -root $HOME/depot_tools/.cipd_bin

Looks like the command above downloads wrong binary, since manually downloading the binary from [1] and then placing it into $HOME/depot_tools/.cipd_bin worked.


[1]: https://chrome-infra-packages.appspot.com/p/infra/tools/luci/led/linux-amd64/+/gT9UxxL939aCQmOX-Mwn4G3FtVUedMSntDnf6Cw2yiQC
Sorry I dropped the ball on this; CL to actually fix this is https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1241677
Project Member

Comment 25 by bugdroid1@chromium.org, Sep 24

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/64b61330ec7f3632d5eb1ea97133ce11f3bf6627

commit 64b61330ec7f3632d5eb1ea97133ce11f3bf6627
Author: Robert Iannucci <iannucci@chromium.org>
Date: Mon Sep 24 22:54:13 2018

[led] Roll led to fix missing expiration_secs issue.

TBR=tandrii@chromium.org, vadimsh@chromium.org

Bug:  875089 
Change-Id: Ia48c7c67df8bdbd9bc0573406bd9f56f6cfe5d49
Reviewed-on: https://chromium-review.googlesource.com/1241677
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/64b61330ec7f3632d5eb1ea97133ce11f3bf6627/cipd_manifest.versions
[modify] https://crrev.com/64b61330ec7f3632d5eb1ea97133ce11f3bf6627/cipd_manifest.txt

Status: Fixed (was: Assigned)
Finally fixed... so sorry for the extended breakage :(.
Project Member

Comment 27 by bugdroid1@chromium.org, Sep 25

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5e1f96f2b6b7c9e947ee8165fad27e60082929a6

commit 5e1f96f2b6b7c9e947ee8165fad27e60082929a6
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Tue Sep 25 00:36:21 2018

Roll src/third_party/depot_tools 482d615b83e1..64b61330ec7f (1 commits)

https://chromium.googlesource.com/chromium/tools/depot_tools.git/+log/482d615b83e1..64b61330ec7f


git log 482d615b83e1..64b61330ec7f --date=short --no-merges --format='%ad %ae %s'
2018-09-24 iannucci@chromium.org [led] Roll led to fix missing expiration_secs issue.


Created with:
  gclient setdep -r src/third_party/depot_tools@64b61330ec7f

The AutoRoll server is located here: https://autoroll.skia.org/r/depot-tools-chromium-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.



BUG= chromium:875089 
TBR=agable@chromium.org

Change-Id: I27e0bdd4fb470ecd7304c86c2c4563d2c0623c1e
Reviewed-on: https://chromium-review.googlesource.com/1242054
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#593776}
[modify] https://crrev.com/5e1f96f2b6b7c9e947ee8165fad27e60082929a6/DEPS

Project Member

Comment 28 by bugdroid1@chromium.org, Oct 15

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/2f727917ac2531cd26c4b4d4fc805398c0bbd132

commit 2f727917ac2531cd26c4b4d4fc805398c0bbd132
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Mon Oct 15 17:02:33 2018

git cl: add reviewers and ccs to git push command if possible.

Also fix a typo in ValidateEmail function which didn't support
email addresses with '-' in them, e.g., infra-dev@chromium.org.

R=ehmaldonado

Bug:  875089 
Change-Id: I2d73c1473527c9bf62e25e9f88250196b783fcb0
Reviewed-on: https://chromium-review.googlesource.com/c/1242849
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>

[modify] https://crrev.com/2f727917ac2531cd26c4b4d4fc805398c0bbd132/scm.py
[modify] https://crrev.com/2f727917ac2531cd26c4b4d4fc805398c0bbd132/tests/git_cl_test.py
[modify] https://crrev.com/2f727917ac2531cd26c4b4d4fc805398c0bbd132/git_cl.py

Project Member

Comment 29 by bugdroid1@chromium.org, Oct 15

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4eb8a3969e2dcf2603615ebcc933cff0f3d42c5d

commit 4eb8a3969e2dcf2603615ebcc933cff0f3d42c5d
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Mon Oct 15 21:01:02 2018

Roll src/third_party/depot_tools 9f274436bd0d..2f727917ac25 (1 commits)

https://chromium.googlesource.com/chromium/tools/depot_tools.git/+log/9f274436bd0d..2f727917ac25


git log 9f274436bd0d..2f727917ac25 --date=short --no-merges --format='%ad %ae %s'
2018-10-15 tandrii@chromium.org git cl: add reviewers and ccs to git push command if possible.


Created with:
  gclient setdep -r src/third_party/depot_tools@2f727917ac25

The AutoRoll server is located here: https://autoroll.skia.org/r/depot-tools-chromium-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.



BUG= chromium:875089 
TBR=agable@chromium.org

Change-Id: I3ecf74c74a21847cb52ffb91bd7d66687e93738d
Reviewed-on: https://chromium-review.googlesource.com/c/1281202
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#599746}
[modify] https://crrev.com/4eb8a3969e2dcf2603615ebcc933cff0f3d42c5d/DEPS

Sign in to add a comment