Per bhthompson@:

paygen stopped working in https://crosland.corp.google.com/log/11150.0.0..11151.0.0

my guess is https://chromium-review.googlesource.com/c/chromiumos/chromite/+/1261799

I think the problem is ListFiles used to not raise error when it couldn't match any string (which is totally fine). But since it is removed and replaced with gs.LS, it is now raising error if it cannot find any matched urls. I think the solution is to catch errors raised by gs.LS at points when it doesn't matter.

I'm shifting this to a P0 since it's a catastrophic issue for M71 and later...

Lets just revert the suspected change on 71, we can let ToT go ahead for now or revert there also.

https://chromium-review.googlesource.com/c/chromiumos/chromite/+/1280887

You probably need to revert at least two changes on top of that change too because they are dependent.
I can work on a fix for tot but feel free to revert the changes for m71. 
Although I think having a fix will be easier. I can try to upload a patch and test it in payloads tryjobs but that would take a while.

I'm testing crrev.com/c/1281442 on payload tryjobs. Hopefully it will fix otherwise a little more coding is needed. Will update this bug.

I think https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8932575805985398848 is passing so good news, but let it pass completely :)

Fix looks reasonable to me, +2ed it, we probably want to get this merged into 71 this afternoon so tonight's build can pass.

Requesting merge into 71 of https://chromium-review.googlesource.com/c/chromiumos/chromite/+/1281442

Approving Merge request for M71 Chrome OS per chat with bhthompson@.  Thanks!

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/25de86ba57f89fba9671b45bc7765e986db11e60

commit 25de86ba57f89fba9671b45bc7765e986db11e60
Author: Amin Hassani <ahassani@chromium.org>
Date: Mon Oct 15 21:23:39 2018

paygen: catch and ignore GSNoSuchKey exception when looking for payloads

ListFiles used to not raise an error when no match found in the payloads
directory. However, gs.LS does raise gs.GSNoSuchKey which is problematic because
when first we are looking for payloads, there are none. So having not payload is
a valid behavior.

BUG= chromium:895438 
TEST=payload tryjobs

Change-Id: Ic6e0e7329ffd174bc0496905378a4e9ed7b0a88e
Reviewed-on: https://chromium-review.googlesource.com/c/1281825
Reviewed-by: Bernie Thompson <bhthompson@chromium.org>
Commit-Queue: Bernie Thompson <bhthompson@chromium.org>
Tested-by: Bernie Thompson <bhthompson@chromium.org>

[modify] https://crrev.com/25de86ba57f89fba9671b45bc7765e986db11e60/lib/paygen/paygen_build_lib.py

Should I wait for canary tot to get green or just cherrypick this in to m71 if it is urgent?

I went ahead and merged it, so we should be good for 71. 

I'm sorry, you merged it in M71 or just chumped it in tot and I still need to merge in m71?

sorry, got it, it was merged to M71. Thanks :)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/70bac6b6f68dc16a48b52c17413ed74731b389e5

commit 70bac6b6f68dc16a48b52c17413ed74731b389e5
Author: Amin Hassani <ahassani@chromium.org>
Date: Tue Oct 16 09:57:25 2018

paygen: catch and ignore GSNoSuchKey exception when looking for payloads

ListFiles used to not raise an error when no match found in the payloads
directory. However, gs.LS does raise gs.GSNoSuchKey which is problematic because
when first we are looking for payloads, there are none. So having not payload is
a valid behavior.

BUG= chromium:895438 
TEST=payload tryjobs

Change-Id: Ic6e0e7329ffd174bc0496905378a4e9ed7b0a88e
Reviewed-on: https://chromium-review.googlesource.com/1281442
Commit-Ready: Amin Hassani <ahassani@chromium.org>
Tested-by: Amin Hassani <ahassani@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Bernie Thompson <bhthompson@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/70bac6b6f68dc16a48b52c17413ed74731b389e5/lib/paygen/paygen_build_lib.py

This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible!

If all merges have been completed, please remove any remaining Merge-Approved labels from this issue.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot