I believe the failure is happening when this line runs:

https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/chromite/api.py?rcl=fdb98accd057135662c36845e5e9197fffc60999&l=249

In the command

```git rebase --verbose --onto master: refs/remotes/origin/master```


The trailing full colon after "master" should not be there.

All PreCQ jobs appear to have started failing around 11:30 AM PDT: https://chrome-swarming.appspot.com/tasklist?c=name&c=state&c=created_ts&c=duration&c=pending_time&c=pool&c=bot&et=1529695200000&f=pool%3AChromeOS&l=50&n=true&q=pool%3AChromeOS&s=created_ts%3Adesc&st=1529608800000

Trooper we didn't change anything in our Recipe lately, any idea what's going on? Can you help?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/565fa6a4078091814163e8449609619953545947

commit 565fa6a4078091814163e8449609619953545947
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Fri Jun 22 19:50:46 2018

Revert "Wipe out shallow checkout support from bot_update."

This reverts commit 568b7a8998b127cb5e34149cd63e9b16fe823a40.

Reason for revert: temporary revert to revert previous CL.

Original change's description:
> Wipe out shallow checkout support from bot_update.
> 
> Bug:  855137 
> Change-Id: I60b8d69f9ab353c12120d4216e3a12518f99732c
> Reviewed-on: https://chromium-review.googlesource.com/1111107
> Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
> Reviewed-by: Ryan Tseng <hinoka@chromium.org>

TBR=iannucci@chromium.org,hinoka@chromium.org,tandrii@chromium.org

Change-Id: Iede67717be448be9bdd7c5dd4895fbab9fd65c66
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  855137 ,  855700 
Reviewed-on: https://chromium-review.googlesource.com/1111924
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>

[add] https://crrev.com/565fa6a4078091814163e8449609619953545947/recipes/recipe_modules/bot_update/examples/full.expected/shallow.json
[modify] https://crrev.com/565fa6a4078091814163e8449609619953545947/tests/bot_update_coverage_test.py
[modify] https://crrev.com/565fa6a4078091814163e8449609619953545947/recipes/recipe_modules/bot_update/examples/full.py
[modify] https://crrev.com/565fa6a4078091814163e8449609619953545947/recipes/README.recipes.md
[modify] https://crrev.com/565fa6a4078091814163e8449609619953545947/recipes/recipe_modules/bot_update/api.py
[modify] https://crrev.com/565fa6a4078091814163e8449609619953545947/recipes/recipe_modules/bot_update/resources/bot_update.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/612d62d0cacf15e89fb9bd14f10ab91c9adde442

commit 612d62d0cacf15e89fb9bd14f10ab91c9adde442
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Fri Jun 22 19:50:56 2018

Revert "bot_update: Don't force solutions to be unmanaged."

This reverts commit c48318a35b173ee25ca65de6910b88e918b84dec.

Reason for revert: likely broke ChromeOS

Original change's description:
> bot_update: Don't force solutions to be unmanaged.
> 
> gclient should be the one syncing the solutions to the appropriate
> revisions.
> 
> Bug: 850812, 853032
> Change-Id: Ieefc5661627d4864deb0d4e7053168a99da29d29
> Reviewed-on: https://chromium-review.googlesource.com/1102833
> Reviewed-by: Aaron Gable <agable@chromium.org>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>

TBR=hinoka@chromium.org,agable@chromium.org,ehmaldonado@chromium.org

Change-Id: I7e110108a9687d703ea8be39cc83e0888f7813c6
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 850812, 853032,  855700 
Reviewed-on: https://chromium-review.googlesource.com/1111925
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/612d62d0cacf15e89fb9bd14f10ab91c9adde442/recipes/recipe_modules/bot_update/resources/bot_update.py
[modify] https://crrev.com/612d62d0cacf15e89fb9bd14f10ab91c9adde442/tests/bot_update_coverage_test.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/b9ab0377c481451a8ecb6004c80f3750ea59ee12

commit b9ab0377c481451a8ecb6004c80f3750ea59ee12
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Fri Jun 22 19:53:25 2018

Revert "Roll recipe dependencies (trivial)."

This reverts commit f5bdafdb068d5dafa6701be2ac5a0466b53c2f9d.

Reason for revert: undoing for 855700

Original change's description:
> Roll recipe dependencies (trivial).
> 
> This is an automated CL created by the recipe roller. This CL rolls recipe
> changes from upstream projects (e.g. depot_tools) into downstream projects
> (e.g. tools/build).
> 
> 
> More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug.
> depot_tools:
>   https://crrev.com/568b7a8998b127cb5e34149cd63e9b16fe823a40 Wipe out shallow checkout support from bot_update. (tandrii@chromium.org)
> 
> 
> TBR=iannucci@chromium.org
> 
> Recipe-Tryjob-Bypass-Reason: Autoroller
> Bugdroid-Send-Email: False
> Change-Id: I2f5c63383e7ea08ab41198184e81f90d411e3a03
> Reviewed-on: https://chromium-review.googlesource.com/1112470
> Commit-Queue: Recipe Roller <recipe-roller@chromium.org>
> Reviewed-by: Recipe Roller <recipe-roller@chromium.org>

TBR=iannucci@chromium.org,recipe-roller@chromium.org

Change-Id: Ib0fb31956296ddc25c4d096ae19f428b3a2a07eb
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  855700 
Reviewed-on: https://chromium-review.googlesource.com/1111927
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/b9ab0377c481451a8ecb6004c80f3750ea59ee12/scripts/slave/README.recipes.md
[modify] https://crrev.com/b9ab0377c481451a8ecb6004c80f3750ea59ee12/infra/config/recipes.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/eeaaf008ce602e409641e9c17b557d131da8a92c

commit eeaaf008ce602e409641e9c17b557d131da8a92c
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Fri Jun 22 19:53:52 2018

Revert "Roll recipe dependencies (trivial)."

This reverts commit fdb98accd057135662c36845e5e9197fffc60999.

Reason for revert: likely broke ChromeOS

Original change's description:
> Roll recipe dependencies (trivial).
> 
> This is an automated CL created by the recipe roller. This CL rolls recipe
> changes from upstream projects (e.g. depot_tools) into downstream projects
> (e.g. tools/build).
> 
> 
> More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug.
> depot_tools:
>   https://crrev.com/c48318a35b173ee25ca65de6910b88e918b84dec bot_update: Don't force solutions to be unmanaged. (ehmaldonado@chromium.org)
> 
> 
> TBR=iannucci@chromium.org
> 
> Recipe-Tryjob-Bypass-Reason: Autoroller
> Bugdroid-Send-Email: False
> Change-Id: I489ddfdafd5cc6c3f720929bc69e23b8a43b48fa
> Reviewed-on: https://chromium-review.googlesource.com/1112461
> Reviewed-by: Recipe Roller <recipe-roller@chromium.org>
> Commit-Queue: Recipe Roller <recipe-roller@chromium.org>

TBR=iannucci@chromium.org,recipe-roller@chromium.org

Change-Id: I2a95695f3b8ef487ccf59c11ed4dca6a410561e6
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  855700  
Reviewed-on: https://chromium-review.googlesource.com/1111926
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/eeaaf008ce602e409641e9c17b557d131da8a92c/scripts/slave/README.recipes.md
[modify] https://crrev.com/eeaaf008ce602e409641e9c17b557d131da8a92c/infra/config/recipes.cfg

I think it should recover now.

Our outage started about 1.5 hours ago and at least two of those changes that were reverted were landed after that.

I don't actually see 1.5 hours ago builds, can you point me out to them?

Confirmed that builds are starting correctly now.

ehmaldonado@ your CL broke our recipe and thus broke most ChromeOS builds. The question is.... what can/should we do to avoid being broken in future?

This is still happening: https://ci.chromium.org/p/chromeos/builders/luci.chromeos.general/PreCQ/b8942992570648120400

Wait, no, those started before the attempted fix; ignore.

Notice the start time on that build. It was 12:11.

https://chrome-swarming.appspot.com/task?id=3e41b185b3829610&refresh=10 is the first build failure that we saw with this symptom. So, more like an hour. Sorry for the confusion.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5bee76eb1685337262ac8f43ad401dc7b72d6200

commit 5bee76eb1685337262ac8f43ad401dc7b72d6200
Author: depot-tools-chromium-autoroll <depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Fri Jun 22 22:59:09 2018

Roll src/third_party/depot_tools c48318a35b17..08d753e88484 (4 commits)

https://chromium.googlesource.com/chromium/tools/depot_tools.git/+log/c48318a35b17..08d753e88484


git log c48318a35b17..08d753e88484 --date=short --no-merges --format='%ad %ae %s'
2018-06-22 recipe-roller@chromium.org Roll recipe dependencies (trivial).
2018-06-22 tandrii@chromium.org Revert "bot_update: Don't force solutions to be unmanaged."
2018-06-22 tandrii@chromium.org Revert "Wipe out shallow checkout support from bot_update."
2018-06-22 tandrii@chromium.org Wipe out shallow checkout support from bot_update.


Created with:
  gclient setdep -r src/third_party/depot_tools@08d753e88484

The AutoRoll server is located here: https://depot-tools-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.



BUG=chromium:850812,chromium:853032, chromium:855700 , chromium:855137 , chromium:855700 , chromium:855137 
TBR=agable@chromium.org

Change-Id: I9f3c95031934c4be4cbdb53960a09bd707f61bc6
Reviewed-on: https://chromium-review.googlesource.com/1112227
Reviewed-by: depot-tools-chromium-autoroll <depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: depot-tools-chromium-autoroll <depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#569820}
[modify] https://crrev.com/5bee76eb1685337262ac8f43ad401dc7b72d6200/DEPS

Other than writing more tests for bot_update, I don't know.

Filed crbug.com/855786 to track efforts to mitigate breakages caused by depot_tools changes.
A doc is linked in there as well.