Issue 854634  has been merged into this issue.

Adding Mike as he helped me set up the chromeos-bsp-moblab as a CROS_WORKON build.

The failing paladin is 'moblab-generic-vm-paladin'

> The failing paladin is 'moblab-generic-vm-paladin'

guado_moblab-paladin is also failing with this symptom.

I'm not quite sure why I got assigned this bug...  I'm not
deputy, and I'm not involved with the change.

Really, this is a sheriff kind of job anyway:  Revert the
bad CL and let the owner sort out how to fix it afterward.

Initial route was because this was a bot commit that bypassed Gerrit and the CQ entirely.  I just picked an engineer who had git commit history on this portion of the source.

Marking moblab-generic-vm-paladin as EXPERIMENTAL and dialing back to P1, since this appears specific to the moblab overlays.

 Issue 854495  has been merged into this issue.

i wonder if our uprev logic has been getting by by accident, and there's an edge case that we happen to not have hit

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/1c11649c24adbe15c4318defe767c0af6d7635c9

commit 1c11649c24adbe15c4318defe767c0af6d7635c9
Author: Jonathan Brandmeyer <jbrandmeyer@chromium.org>
Date: Wed Jun 20 17:02:05 2018

[moblab] Revert "Marking set of ebuilds as stable" for project-moblab

This partially reverts commit 68237d24f46cbaa3559ee1658bcc745d58800f36.
This was a bot commit that refers to a non-existant git commit hash.

TEST=build on moblab-generic-vm
BUG=chromium:854633
Signed-off-by: Jonathan Brandmeyer <jbrandmeyer@chromium.org>

Change-Id: I97bc2f368d31886e5dceea319f624db42fbe8158
Reviewed-on: https://chromium-review.googlesource.com/1108190
Tested-by: Jonathan Brandmeyer <jbrandmeyer@chromium.org>
Trybot-Ready: Jonathan Brandmeyer <jbrandmeyer@chromium.org>
Reviewed-by: Keith Haddow <haddowk@chromium.org>
Commit-Queue: Keith Haddow <haddowk@chromium.org>
Commit-Queue: Jonathan Brandmeyer <jbrandmeyer@chromium.org>

[rename] https://crrev.com/1c11649c24adbe15c4318defe767c0af6d7635c9/project-moblab/chromeos-base/chromeos-bsp-moblab/chromeos-bsp-moblab-0.0.1-r30.ebuild

you want to take a look at this Lann since you just fixed that last cros uprev bug ?

The bot made a similar error prior to the next CQ run.

https://luci-milo.appspot.com/buildbot/chromiumos/moblab-generic-vm-paladin/1891

lannm calender shows OOO - can someone else take a look ?

This happened again to the next change that was submitted to that repo:
https://chromium-review.googlesource.com/c/chromiumos/third_party/autotest/+/1107238

moblab-generic-vm-pre-cq: The BuildPackages stage failed: Packages failed in ./build_packages: chromeos-base/chromeos-bsp-moblab in https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8943171570088917344


...
chromeos-bsp-moblab-0.0.1-r31: GIT update -->
chromeos-bsp-moblab-0.0.1-r31:    repository:               https://chromium.googlesource.com/chromiumos/overlays/board-overlays.git
chromeos-bsp-moblab-0.0.1-r31:    at the commit:            0ab2e5bd59878f72f55ed8db7dc3f7896baab29f
chromeos-bsp-moblab-0.0.1-r31:    commit:                   1171ecd46e4b44f44c4f46d6fa0d31b153eef996
chromeos-bsp-moblab-0.0.1-r31:    branch:                   master
chromeos-bsp-moblab-0.0.1-r31:    storage directory:        "/var/cache/chromeos-cache/distfiles/target/egit-src/chromiumos/overlays/board-overlays"
chromeos-bsp-moblab-0.0.1-r31:    checkout type:            bare repository
chromeos-bsp-moblab-0.0.1-r31: Cloning into '/build/moblab-generic-vm/tmp/portage/chromeos-base/chromeos-bsp-moblab-0.0.1-r31/work/chromeos-bsp-moblab-0.0.1'...
chromeos-bsp-moblab-0.0.1-r31: done.
chromeos-bsp-moblab-0.0.1-r31: fatal: reference is not a tree: 1171ecd46e4b44f44c4f46d6fa0d31b153eef996
chromeos-bsp-moblab-0.0.1-r31:  * ERROR: chromeos-base/chromeos-bsp-moblab-0.0.1-r31::moblab failed (unpack phase):
chromeos-bsp-moblab-0.0.1-r31:  *   git-2_branch: changing the branch failed
...


I don't see any point in trying to resubmit the change until this is root caused/fixed. 
 I'm pretty sure the change is just the victim and not the cause of the failure.

This also causes this CL to fail
https://chromium-review.googlesource.com/c/chromiumos/third_party/autotest/+/1097475

Same error message as in comment #13
...
chromeos-bsp-moblab-0.0.1-r31: fatal: reference is not a tree: 1171ecd46e4b44f44c4f46d6fa0d31b153eef996
chromeos-bsp-moblab-0.0.1-r31:  * ERROR: chromeos-base/chromeos-bsp-moblab-0.0.1-r31::moblab failed (unpack phase):
chromeos-bsp-moblab-0.0.1-r31:  *   git-2_branch: changing the branch failed
...

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8943124486514238064

Any thoughts about a solution for this ?  Even if we can come up with a workaround until the root cause it found.

If we keep moblab out of the CQ for a long time history suggests that it takes a lot of work to identify and fix all the issues that would have been correctly rejected by the CQ.

How about a root cause first?
Where is 1171ecd46e4b44f44c4f46d6fa0d31b153eef996 SHA1 coming from?

Looking at the build from #14:


https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8943124486514238064

From the uprev stage:

01:28:15: INFO: Creating new stable ebuild /b/swarming/w/ir/cache/cbuild/repository/src/overlays/project-moblab/chromeos-base/chromeos-bsp-moblab/chromeos-bsp-moblab-0.0.1-r32.ebuild
01:28:15: INFO: Old and new ebuild /b/swarming/w/ir/cache/cbuild/repository/src/overlays/project-moblab/chromeos-base/chromeos-bsp-moblab/chromeos-bsp-moblab-0.0.1-r32.ebuild are exactly identical; skipping uprev
01:28:16: INFO: Creating new stable ebuild /b/swarming/w/ir/cache/cbuild/repository/src/overlays/project-moblab/sys-apps/moblab-site-utils/moblab-site-utils-0.0.1-r24.ebuild
01:28:16: INFO: Old and new ebuild /b/swarming/w/ir/cache/cbuild/repository/src/overlays/project-moblab/sys-apps/moblab-site-utils/moblab-site-utils-0.0.1-r24.ebuild are exactly identical; skipping uprev
01:28:16: INFO: Creating new stable ebuild /b/swarming/w/ir/cache/cbuild/repository/src/overlays/project-moblab/sys-apps/mobmonitor/mobmonitor-0.0.1-r24.ebuild
01:28:16: INFO: Old and new ebuild /b/swarming/w/ir/cache/cbuild/repository/src/overlays/project-moblab/sys-apps/mobmonitor/mobmonitor-0.0.1-r24.ebuild are exactly identical; skipping uprev
01:28:16: INFO: Creating new stable ebuild /b/swarming/w/ir/cache/cbuild/repository/src/overlays/project-moblab/sys-apps/mobmonitor-ui/mobmonitor-ui-0.0.1-r14.ebuild
01:28:16: INFO: Old and new ebuild /b/swarming/w/ir/cache/cbuild/repository/src/overlays/project-moblab/sys-apps/mobmonitor-ui/mobmonitor-ui-0.0.1-r14.ebuild are exactly identical; skipping uprev
01:


From build_packages:

== Complete: job chromeos-bsp-moblab-0.0.1-r31 (0m1.8s) ===
Failed chromeos-base/chromeos-bsp-moblab-0.0.1-r31 (in 0m1.8s). Your build has failed.
Pending 7/818, [Time 02:10:42 | Elapsed 28m36.1s | Load 4.41 13.87 22.42]
Packages failed:
	chromeos-base/chromeos-bsp-moblab-0.0.1-r31
ERROR   : Thu Jun 21 02:10:43 PDT 2018
ERROR   :  PGID  PPID   PID     ELAPSED     TIME %CPU COMMAND
ERROR   : Arguments of 10: ./build_packages '--board=moblab-generic-vm' '--accept_licenses=@CHROMEOS' '--withdebugsymbols' '--skip_chroot_upgrade' '--withevents' '--eventfile=/mnt/host/source/trybot_archive/moblab-generic-vm-pre-cq/R69-10803.0.0-b2683403/build-events.json' '--save_install_plan=/tmp/moblab-generic-vm_install_plan.2683403' 'virtual/target-os' 'virtual/target-os-dev' 'virtual/target-os-test' 'virtual/target-os-factory' 'virtual/target-os-factory-shim' 'chromeos-base/autotest-all'
ERROR   : Backtrace:  (most recent call is last)
ERROR   :  build_packages:368:main(), called: die_err_trap  
ERROR   : 
ERROR   : Command failed:
ERROR   :   Command '( if [[ "${FLAGS_run_goma}" -eq "${FLAGS_TRUE}" ]]; then
ERROR   :     info "Starting goma compiler_proxy."; goma_ctl="${GOMA_DIR:-${HOME}/goma}/goma_ctl.py"; "${goma_ctl}" restart; trap "'${goma_ctl}' stop" EXIT;
ERROR   : fi; set -o pipefail; sudo -E "${EMERGE_CMD[@]}" "${EMERGE_FLAGS[@]}" "${PACKAGES[@]}" | tee "${tmpfile}" )' exited with nonzero code: 1
02:10:43: ERROR: 

So, that suggests that the TOT ebuild is R31, and it is a valid hash at uprev time, but not build time.

The R31 ebuild revision was committed here, which looks like an automated build generated it:


commit b5e11768b62284c3f358aec10558cfaeb8078d09
Author: chrome-bot <chrome-bot@chromium.org>
Date:   Wed Jun 20 10:32:29 2018 -0700

    Marking set of ebuilds as stable
    
    Marking 9999 ebuild for chromeos-base/chromeos-bsp-moblab as stable.
    
    Change-Id: I9153dab466716408b77c7fd1dfbedfed64759d2f
    
    Marking 9999 ebuild for chromeos-base/temp-metrics as stable.
    
    Change-Id: I936b1c51fdeee8b9119466e9461b4f61807e3f8d


And this was the previous change, which seem to have hand fixed it:

commit 1c11649c24adbe15c4318defe767c0af6d7635c9
Author: Jonathan Brandmeyer <jbrandmeyer@chromium.org>
Date:   Wed Jun 20 10:04:16 2018 -0600

    [moblab] Revert "Marking set of ebuilds as stable" for project-moblab
    
    This partially reverts commit 68237d24f46cbaa3559ee1658bcc745d58800f36.
    This was a bot commit that refers to a non-existant git commit hash.
    
    TEST=build on moblab-generic-vm
    BUG=chromium:854633
    Signed-off-by: Jonathan Brandmeyer <jbrandmeyer@chromium.org>
    
    Change-Id: I97bc2f368d31886e5dceea319f624db42fbe8158
    Reviewed-on: https://chromium-review.googlesource.com/1108190
    Tested-by: Jonathan Brandmeyer <jbrandmeyer@chromium.org>
    Trybot-Ready: Jonathan Brandmeyer <jbrandmeyer@chromium.org>
    Reviewed-by: Keith Haddow <haddowk@chromium.org>
    Commit-Queue: Keith Haddow <haddowk@chromium.org>
    Commit-Queue: Jonathan Brandmeyer <jbrandmeyer@chromium.org>



It's not safe to simply revert an ebuild uprev change like that, and it can leave chroots in a corrupted state. 

The first attempt to produce R31 was the earlier chrome-bot change https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/68237d24f46cbaa3559ee1658bcc745d58800f36

This one also referred to an incorrect git hash.

Oh... sorry, I was just dumping updates as I looked, and got distracted right when things went silent. ;>

I can force a new uprev, but would like to understand why things went wrong in the first place before I do.

This is the build that uprevved to R31 causing problems in the first place.

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8943254511256245072

What stands out to me is that the master builder failed, but still submitted this uprev despite not submitting a variety of other CLs. If one of those other CLs was in the public overlays repository, it could have caused this problem.

I thought we would refuse to submit uprevs if we didn't submit all CLs in the CQ run. However, if that's not the case, I'm now surprised we don't hit this issue more often.

I'm also not clear why chroot resets on the paladins after failed builds didn't reset everything well enough for new uprevs to work. In other words, I really don't understand this issue yet.

Hi, Is there an ETA on when this will be resolved? My CL's been rejected multiple times because of this in the past day. Was wondering if there's anything else I can do to possibly bypass this?

I'm testing a possible fix here:
  https://crrev.com/c/1110883

I'm testing it with two tryjobs:

Expected to pass:
  http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8943074240874396784

Expected to reproduce the failure:
  http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8943073504019735024

My "expected to fail" tryjob has reproduced the problem, and the "expected to pass" tryjob has gotten past build_packages.

I think the fix works.

I plan to chump as soon as it passes the PreCQ (if it does).

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/4bf701c633648e4f022d18354c31973114c3e672

commit 4bf701c633648e4f022d18354c31973114c3e672
Author: Don Garrett <dgarrett@google.com>
Date: Thu Jun 21 23:37:56 2018

Manual reset of chromeos-bsp-moblab to older hash.

BUG=chromium:854633
TEST=cros tryjob -g 1110883 guado_moblab-paladin-tryjob

Change-Id: I73214303512521b682692cefcde32f0726d8c797
Reviewed-on: https://chromium-review.googlesource.com/1110883
Tested-by: Don Garrett <dgarrett@chromium.org>
Trybot-Ready: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Keith Haddow <haddowk@chromium.org>

[rename] https://crrev.com/4bf701c633648e4f022d18354c31973114c3e672/project-moblab/chromeos-base/chromeos-bsp-moblab/chromeos-bsp-moblab-0.0.1-r32.ebuild

Believed fixed. Please reopen if it's not.

Looks like it is back
https://luci-milo.appspot.com/buildbot/chromeos/guado_moblab-paladin/9734

chromeos-bsp-moblab-0.0.1-r33: >>> Unpacking source...
chromeos-bsp-moblab-0.0.1-r33: Cloning into '/build/guado_moblab/tmp/portage/chromeos-base/chromeos-bsp-moblab-0.0.1-r33/work/chromeos-bsp-moblab-0.0.1'...
chromeos-bsp-moblab-0.0.1-r33: done.
chromeos-bsp-moblab-0.0.1-r33: fatal: reference is not a tree: 5e7a13de9216a8e378e92d141743acfde9c6e359
chromeos-bsp-moblab-0.0.1-r33:  * Cannot run git checkout 5e7a13de9216a8e378e92d141743acfde9c6e359 in /build/guado_moblab/tmp/portage/chromeos-base/chromeos-bsp-moblab-0.0.1-r33/work/chromeos-bsp-moblab-0.0.1.
chromeos-bsp-moblab-0.0.1-r33:  * Is /mnt/host/source/src/platform/../overlays/ up to date? Try running repo sync.
chromeos-bsp-moblab-0.0.1-r33:  * Falling back to git.eclass...
chromeos-bsp-moblab-0.0.1-r33: remote: Counting objects: 1   

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/b14425ab29148897ba3c5c92a714b4f67fbfd47e

commit b14425ab29148897ba3c5c92a714b4f67fbfd47e
Author: Keith Haddow <haddowk@chromium.org>
Date: Fri Jun 22 04:51:30 2018

[moblab] Trivial change to ebuild comment.

To try and fix the uprev issues https://crbug.com/854633 make a
trivial change to the ebuild

BUG=chromium:854633
TEST=None

Change-Id: I4cbd7c02d10f760c989437de30e6c50a078cfa11
Reviewed-on: https://chromium-review.googlesource.com/1111677
Reviewed-by: Keith Haddow <haddowk@chromium.org>
Commit-Queue: Keith Haddow <haddowk@chromium.org>
Tested-by: Keith Haddow <haddowk@chromium.org>
Trybot-Ready: Keith Haddow <haddowk@chromium.org>

[modify] https://crrev.com/b14425ab29148897ba3c5c92a714b4f67fbfd47e/project-moblab/chromeos-base/chromeos-bsp-moblab/chromeos-bsp-moblab-9999.ebuild

Mike, as the new oncall, can you take a look at this?

Has it happened again, or is this now fixed?

It has not happened again, it would be good to know the root cause though.

I suspect, but have not clearly demonstrated, that it's because of how we handle the mix of partial CL submission in the CQ and autogenerated ebuild uprev CLs.

We plan to make the CQ much more complicated over time, which will make this problem harder. I have proposal I started a while back that would resolve it, at the cost of more latency for ebuild uprevs.

This doesn't appear to be happening again.  Root cause is noted and will be addressed in future infrastructure proposals.  

it's happening again, the tree is broken (see https://bugs.chromium.org/p/chromium/issues/detail?id=857031)

As noted in the other bug, I suspect this happens when another ebuild is uprev'd in the same CQ run.

Presumably to unlock the tree we need to modify chromeos-bsp-moblab-9999 and hope that no other CL touches the same repo during the CQ run

I believe that is the work around.

Alec made recent changes to the uprev script, which are probably related to this surfacing more frequently, and so may have some context.

 Issue 857031  has been merged into this issue.

Does this really need to be a cros_workon ebuild?

Looking at the ebuild, I see that the ebuild is upreved depending on the overlay where it is located, I can totally imagine how this can be racy:

CROS_WORKON_COMMIT="ac9ffc23c8fa6c657541c44993998c2ff7fce49a"
CROS_WORKON_TREE="e25d41e7046bd0c581073ebaa9e8c65942b00ade"
CROS_WORKON_PROJECT="chromiumos/overlays/board-overlays"
CROS_WORKON_LOCALNAME="../overlays/"
CROS_WORKON_SUBTREE="project-moblab/chromeos-base/chromeos-bsp-moblab/files"

I see 2 easy options:
 1. Remove the cros_workon and manually uprev the ebuild when files are modified: https://chromium-review.googlesource.com/919005
 2. Move files/ to another repo, say src/platform?

(#43: for point 1, it seems like https://chromium-review.googlesource.com/919005 was avoid manual uprev...)

Suggested temporary(?) fix here: https://chromium-review.googlesource.com/c/chromiumos/overlays/board-overlays/+/1118085 .

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/f02af870bed03e25ff765e95613052ceb5e252b7

commit f02af870bed03e25ff765e95613052ceb5e252b7
Author: Nicolas Boichat <drinkcat@chromium.org>
Date: Thu Jun 28 08:53:21 2018

chromeos-bsp-moblab: Blacklist cros_workon ebuild

Until we can figure out what's wrong with the upreving logic.

BUG=chromium:854633
TEST=moblab-generic-vm-paladin passes

Change-Id: Ifc0bf4fc88689181489f6256e2031bda96d6eb16
Reviewed-on: https://chromium-review.googlesource.com/1118085
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Nicolas Boichat <drinkcat@chromium.org>
Tested-by: Nicolas Boichat <drinkcat@chromium.org>

[rename] https://crrev.com/f02af870bed03e25ff765e95613052ceb5e252b7/project-moblab/chromeos-base/chromeos-bsp-moblab/chromeos-bsp-moblab-0.0.1-r41.ebuild
[modify] https://crrev.com/f02af870bed03e25ff765e95613052ceb5e252b7/project-moblab/chromeos-base/chromeos-bsp-moblab/chromeos-bsp-moblab-9999.ebuild

#43:  That seems to be the conclusion that we've been coming to as well.  The issue seems to be occurring for this ebuild specifically and is largely based on the location.  I'm adding some further logging to portage_util to try to pinpoint exactly where things are going sideways with the uprev.  

-- Mike

I believe there are other builds that also refer to the repo in which they are defined, probably most of the BSPs.

https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/f02af870bed03e25ff765e95613052ceb5e252b7/project-moblab/chromeos-base/chromeos-bsp-moblab/chromeos-bsp-moblab-0.0.1-r41.ebuild#11 is the source of the problem: we already uprev ebuilds when the contents of files/ is modified. This is just doubly trying to do the same thing that already exists globally. This ebuild is fundamentally broken because of this.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/cb1b12779c550f0c8fe27420fdeb94132d4ebe12

commit cb1b12779c550f0c8fe27420fdeb94132d4ebe12
Author: Mike Nichols <mikenichols@chromium.org>
Date: Fri Jun 29 08:09:01 2018

[portage_util] Additional logging for portage_util to capture commit id.

Clean up some logging to make it a bit more obvious as to the actual
creation of the ebuilds due to uprev.  Additionally capture the commit
id when the new ebuild is created to help track the breakage through the
entire build process.

BUG=chromium:854633
TEST=None

Change-Id: I58646f7997c98a0903d59dd8d36f38d5c2a27da3
Reviewed-on: https://chromium-review.googlesource.com/1117890
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Mike Nichols <mikenichols@chromium.org>
Reviewed-by: Jason Clinton <jclinton@chromium.org>
Reviewed-by: Mike Nichols <mikenichols@chromium.org>

[modify] https://crrev.com/cb1b12779c550f0c8fe27420fdeb94132d4ebe12/lib/portage_util.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3f7d4aa958223a9efc9cc0a1d05d0b24078823d7

commit 3f7d4aa958223a9efc9cc0a1d05d0b24078823d7
Author: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Fri Jun 29 16:00:18 2018

Roll src/third_party/chromite 525fc9fdb588..7869e579e31e (4 commits)

https://chromium.googlesource.com/chromiumos/chromite.git/+log/525fc9fdb588..7869e579e31e


git log 525fc9fdb588..7869e579e31e --date=short --no-merges --format='%ad %ae %s'
2018-06-29 vapier@chromium.org lint: handle indentation of section subtext
2018-06-29 vapier@chromium.org lint: clean up bad docstring indentation
2018-06-29 mikenichols@chromium.org [portage_util] Additional logging for portage_util to capture commit id.
2018-06-29 dgarrett@google.com cros tryjob: Add --staging to run against test builder.


Created with:
  gclient setdep -r src/third_party/chromite@7869e579e31e

The AutoRoll server is located here: https://chromite-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.



BUG=chromium:485005,chromium:None,chromium:854633,chromium:765749
TBR=chrome-os-gardeners@chromium.org

Change-Id: I7bbfe2739cd1c72b7d07d50b7677578412b17ed8
Reviewed-on: https://chromium-review.googlesource.com/1119925
Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#571496}
[modify] https://crrev.com/3f7d4aa958223a9efc9cc0a1d05d0b24078823d7/DEPS

Pri-0 bugs are critical regressions or serious emergencies, and this bug has not been updated in three days. Could you please provide an update, or adjust the priority to a more appropriate level if applicable?

If a fix is in active development, please set the status to Started.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

I think we have a root cause now (the use of CROS_WORKON_SUBTREE=files) so lowering this to P1.

As root cause has been identified, going to lower this to a P2.  As this is the first time we've seen this behavior, and is behavior that could be introduced in the future, a great next step will to add some checking to ensure that we do not reproduce the behavior in the future.  

Setting as a good starter bug to add some futureproofing.

-- Mike