It felt like something cros_workon-able needs to be uprev'ed but I am not sure what? I have not seen this problem before when creating a new overlay, though.

Jason/Lann, just wondering if we can chomp the CL over the weekend (and revert it as soon as problem being detected)?

Thanks.

Here is the problem:

https://cs.corp.google.com/chromeos_public/chromite/scripts/cros_mark_as_stable.py?rcl=81e0d3d735ded0b537468ed6064aabc884a146b1&l=403

work_branch.CreateBranch() checks out master, which removes the new overlay CL. The CL is *supposed* to be reapplied immediately below by the rebase call, but first work_branch.Exists() is called, which ultimately calls Popen(cwd=<new overlay>), which fails.

The only mystery is how this ever worked. Maybe previously the new overlay was not being passed to cros_mark_as_stable.

Once that CL is chomped. Rest of CLs look to be fine with CQ and got merged successfully.
Thanks.

I tracked down the reason for this CL to fail where others have succeeded; this bug depends on the order of overlays passed to cros_mark_as_stable. That order appears to be derived from the iteration order of a dict, which is undefined in python. overlay-dragonegg happened to appear at the front of the list which triggers this bug. This probably also explains why the local tryjob passed (different dict iteration order).

Thanks Lann. How can we make the order deterministic?

I'm working on separate fixes for the nondeterministic ordering and the underlying bug.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/e0ef98c6338cf3ab2c4b434e1d7fc0d21e4e6a56

commit e0ef98c6338cf3ab2c4b434e1d7fc0d21e4e6a56
Author: Lann Martin <lannm@chromium.org>
Date: Thu Jun 14 00:02:09 2018

Fix cros_mark_as_stable for CLs that create overlays

If the first overlay passed to cros_mark_as_stable was created in a CL
under test, the branch management logic for overlays would fail. Fix by
using the overlay's top-level git directory as the cwd for branching
operations instead of the overlay directory itself.

BUG= chromium:851059 
TEST=cros_mark_as_stable_unittest; git_unittest
TEST=tryjobs with throwaway overlay-fake CL

Change-Id: If2dfaac9b73a163d7c6b0a06717e02ca51375953
Reviewed-on: https://chromium-review.googlesource.com/1096027
Commit-Ready: Lann Martin <lannm@chromium.org>
Tested-by: Lann Martin <lannm@chromium.org>
Reviewed-by: Jason Clinton <jclinton@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/e0ef98c6338cf3ab2c4b434e1d7fc0d21e4e6a56/lib/git.py
[modify] https://crrev.com/e0ef98c6338cf3ab2c4b434e1d7fc0d21e4e6a56/scripts/cros_mark_as_stable.py
[modify] https://crrev.com/e0ef98c6338cf3ab2c4b434e1d7fc0d21e4e6a56/lib/git_unittest.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/7fb2f1a53eabdb2a2631a433716ba608730e2773

commit 7fb2f1a53eabdb2a2631a433716ba608730e2773
Author: Lann Martin <lannm@chromium.org>
Date: Thu Jun 14 00:02:18 2018

Make FindOverlays output have deterministic ordering

This will make it easier to debug certain kinds of failures (see BUG).

BUG= chromium:851059 
TEST=portage_util_unittests

Change-Id: Ic23f3cd6b4bbaebede2e3118dd2760fb42c45e76
Reviewed-on: https://chromium-review.googlesource.com/1096119
Commit-Ready: Lann Martin <lannm@chromium.org>
Tested-by: Lann Martin <lannm@chromium.org>
Reviewed-by: Jason Clinton <jclinton@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/7fb2f1a53eabdb2a2631a433716ba608730e2773/lib/portage_util.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6fdada6c28d349e63ed16c3de4a4ea06c6823e4e

commit 6fdada6c28d349e63ed16c3de4a4ea06c6823e4e
Author: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Thu Jun 14 05:35:04 2018

Roll src/third_party/chromite 402795d084bf..17b4574e14a3 (8 commits)

https://chromium.googlesource.com/chromiumos/chromite.git/+log/402795d084bf..17b4574e14a3


git log 402795d084bf..17b4574e14a3 --date=short --no-merges --format='%ad %ae %s'
2018-06-14 chrome-bot@chromium.org Update config settings by config-updater.
2018-06-14 dgarrett@google.com master-full: Add better master/slave support for FULL_TYPE.
2018-06-14 bpastene@chromium.org cros_run_vm_test: Fix --host-cmd and --autotest cmd args.
2018-06-14 lannm@chromium.org Make FindOverlays output have deterministic ordering
2018-06-14 lannm@chromium.org Fix cros_mark_as_stable for CLs that create overlays
2018-06-13 ihf@chromium.org chromeos_config: make betty-arcnext important.
2018-06-13 bmgordon@chromium.org cros_sdk_lib: Reduce version warning to debug
2018-06-13 bmgordon@chromium.org cros_sdk: Add a --unmount action


Created with:
  gclient setdep -r src/third_party/chromite@17b4574e14a3

The AutoRoll server is located here: https://chromite-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.



BUG= chromium:851183 ,chromium:None,chromium:851059,chromium:851059,chromium:834412,chromium:None
TBR=chrome-os-gardeners@chromium.org

Change-Id: I247744af7bf25889f89c57eee10cf8568af22542
Reviewed-on: https://chromium-review.googlesource.com/1100319
Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#567155}
[modify] https://crrev.com/6fdada6c28d349e63ed16c3de4a4ea06c6823e4e/DEPS

I believe the underlying bug is fixed.

 Issue 849409  has been merged into this issue.