Clobbered build directories. The bot should be ok on next build.

I suspect the root cause is similar to https://bugs.chromium.org/p/chromium/issues/detail?id=646980#c5 and it should be fixed by https://chrome-internal.googlesource.com/infra/puppet/+/c23d342e0e10df2d084c857e9490f03ec2d3a84d

Please reopen if it happens again.

vadimsh: could you provide details of how the build directories were broken? It'd help me debug cleanup_disk. Thanks!

It attempts to delete build directory and fails midway:

[I2016-09-13T00:30:00.223960-07:00 1000 0 cleanup.go:245] Processing C:\b\c\b\*
[E2016-09-13T00:30:10.630428-07:00 1000 0 cleanup.go:166] Failed to read directory 'C:\b\c\b\winx64_high_dpi_perf_bisect'
[E2016-09-13T00:30:10.630428-07:00 1000 0 cleanup.go:166] Failed to read directory 'C:\b\c\b\winx64_high_dpi_perf_bisect'
[E2016-09-13T00:30:10.708500-07:00 1000 0 cleanup.go:166] Failed to read directory 'C:\b\c\b\winx64_high_dpi_perf_bisect'
[E2016-09-13T00:30:10.724106-07:00 1000 0 cleanup.go:166] Failed to read directory 'C:\b\c\b\winx64_high_dpi_perf_bisect'
[I2016-09-13T00:30:15.255420-07:00 1000 0 cleanup.go:107] Deleting C:\b\c\b\winx64_high_dpi_perf_bisect
[E2016-09-13T00:32:14.538341-07:00 1000 0 cleanup.go:223] Deleting/truncating path failed: C:\b\c\b\winx64_high_dpi_perf_bisect remove C:\b\c\b\winx64_high_dpi_perf_bisect\src\.git\objects\3f\86b069143db2eae640a3c85bb11aa27447f682: Access is denied.

It then leaves it following state:
$ cd /cygdrive/c/b/c/b/winx64_high_dpi_perf_bisect
$ find .
.
./src
./src/.git
./src/.git/objects
./src/.git/objects/3f
./src/.git/objects/3f/86b069143db2eae640a3c85bb11aa27447f682
./src/.git/objects/6f
./src/.git/objects/6f/5a96a07b4810812a19592b627142512a5ad567
./src/.git/objects/8f
./src/.git/objects/8f/5e660e68b92643f03fc070f94e0b25434b32ad
./src/.git/objects/90
./src/.git/objects/90/3cc029e9dbf1e6ee38fed81fe5b4a8c0bc44f7
./src/.git/objects/b5
./src/.git/objects/b5/fba195e2eb9e7f512e9705c4c20b046edf7779
./src/.git/objects/c7
./src/.git/objects/c7/b76324ec82b693b97025af8723dc80d341f66b
./src/.git/objects/ce
./src/.git/objects/ce/c60e434f05eb8ae2921f5b0a8e5217572f5ff2
./src/.git/objects/d8
./src/.git/objects/d8/fa230a774aca7e879366b42ef09dfc75c2da24
./src/.git/objects/d9
./src/.git/objects/d9/9321af04aafe665ee5d14f817cc1c04e9e569c
./src/.git/objects/fa
./src/.git/objects/fa/a36d2aa1ee05ecc43dadfcd63cba13b497df7f

And git and bot_update.py don't know how to handle it:
$ git.bat remote show origin
fatal: Not a git repository (or any of the parent directories): .git

 Issue 647297  has been merged into this issue.

 Issue 647139  has been merged into this issue.

Also happens on WebRTC bots.

Yep, looks like same thing:

chrome-bot@VM207-M4 /cygdrive/e/b/c/b/win_compile_dbg/src/.git
$ cat /cygdrive/e/chrome-infra-logs/cleanup_disk.VM207-M4.NT\ AUTHORITY_SYSTEM.log.DEBUG.20160914-003001.3268 | grep objects
[E2016-09-14T00:42:30.203069-07:00 3268 0 cleanup.go:223] Deleting/truncating path failed: E:\b\c\b\win_compile_dbg remove E:\b\c\b\win_compile_dbg\src\.git\objects\09\6cbc775e9f39c924001ac188f5837e19862c82: Access is denied.

I'll try to teach bot update to detect broken .git directories and remove them, since I suspect there's unknown number of other bots that were affected.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools.git/+/18ca30ca804679ee624a52e73017d234a8c0008f

commit 18ca30ca804679ee624a52e73017d234a8c0008f
Author: vadimsh <vadimsh@chromium.org>
Date: Thu Sep 15 22:00:04 2016

Teach bot_update to remove partially deleted git repos.

Also add logging to 'remove', helps to debug issues with 'build.dead' filling
up with crap.

R=iannucci@chromium.org, hinoka@chromium.org
BUG= 647046 

Review-Url: https://codereview.chromium.org/2342973002

[modify] https://crrev.com/18ca30ca804679ee624a52e73017d234a8c0008f/recipe_modules/bot_update/resources/bot_update.py

Looks like there are still some problems with this: https://build.chromium.org/p/tryserver.chromium.perf/builders/winx64_high_dpi_perf_bisect/builds/26

STDIO: https://build.chromium.org/p/tryserver.chromium.perf/builders/winx64_high_dpi_perf_bisect/builds/26/steps/bot_update/logs/stdio

Any idea what might be happening?

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/7c3a4ec47d9e67fb62eb2c8d6454d8a80b106b22

commit 7c3a4ec47d9e67fb62eb2c8d6454d8a80b106b22
Author: recipe-roller <recipe-roller@chromium.org>
Date: Thu Sep 15 22:28:50 2016

The fix hasn't been rolled out yet. It crawls through recipe roller rolls.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/996cd0e1cdb02ee146e29e3d57c265b08fa86a26

commit 996cd0e1cdb02ee146e29e3d57c265b08fa86a26
Author: recipe-roller <recipe-roller@chromium.org>
Date: Thu Sep 15 22:33:09 2016

Roll recipe dependencies (trivial).

This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug
(or complain)

build:
  https://crrev.com/d2845f7e87aa167cf7df78003b5768f948803d07 Roll recipe dependencies (trivial). (recipe-roller@chromium.org)
depot_tools:
  https://crrev.com/18ca30ca804679ee624a52e73017d234a8c0008f Teach bot_update to remove partially deleted git repos. (vadimsh@chromium.org)

TBR=martiniss@chromium.org,phajdan.jr@chromium.org
BUG= 647046 

Recipe-Tryjob-Bypass-Reason: Autoroller
Bugdroid-Send-Email: False
Review-Url: https://codereview.chromium.org/2346993002
Cr-Commit-Position: refs/heads/master@{#419016}

[modify] https://crrev.com/996cd0e1cdb02ee146e29e3d57c265b08fa86a26/infra/config/recipes.cfg

And now it should work. I hope.

The fix appears to be working: https://build.chromium.org/p/tryserver.webrtc/builders/win_compile_dbg/builds/7328

Git repo E:\b\c\b\win_compile_dbg\src appears to be broken, removing it
Marking for removal E:\b\c\b\win_compile_dbg\src => E:\b\c\b\build.dead\1218d826bbdc44a5973aa6e992f889f3
===Running git.bat clone --no-checkout --local --shared E:\\b\\c\\git_cache\chromium.googlesource.com-external-webrtc E:\b\c\b\win_compile_dbg\src (attempt #1)===
In directory: E:\b\c\b\win_compile_dbg
Cloning into 'E:\b\c\b\win_compile_dbg\src'...
done.
===Succeeded in 0.0 mins===
....

Yay, thanks a bunch Vadim!

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra.git/+/7d6811c4c58a9e280bca7bf65a8dfde001bef42a

commit 7d6811c4c58a9e280bca7bf65a8dfde001bef42a
Author: recipe-roller <recipe-roller@chromium.org>
Date: Fri Sep 16 23:37:07 2016

Roll recipe dependencies (trivial).

This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug
(or complain)

build:
  https://crrev.com/ba9de887ec268453298caa8641de3cc91dbfb818 Roll recipe dependencies (trivial). (recipe-roller@chromium.org)
  https://crrev.com/e1d15cd7795cc1c5465983af8a49669242857107 Add rtc_stats_unittests to the test suite (ehmaldonado@chromium.org)
  https://crrev.com/37294244d9d54d639a083d382da68c50f8fe14e5 Add linux_chromium_headless_dbg trybot (perezju@chromium.org)
  https://crrev.com/f0f4c4a27746dee8a53234f03e452dba556deb55 Reland of Enable the ninja up-to-date check for Android builders (agrieve@chromium.org)
  https://crrev.com/2744734cf3aa1d8a02d0f0b7abf99635e9f2aa48 Revert of Enable the ninja up-to-date check for Android builders (patchset #1 id:1 of https://codereview.chromium.org/2343563003/ ) (agrieve@chromium.org)
  https://crrev.com/1575589a1d8f61fee45ec53ccbf77f073fd38847 recipe_modules/chromite: Use "build_type". (dnj@chromium.org)
  https://crrev.com/5799bab975aa3eda6785f36d9f7879549074b28b Reland of Enable the ninja up-to-date check for Android builders (agrieve@chromium.org)
  https://crrev.com/ecdf065c9d2a6e28d86c8ad333432b497158ee7a chromium.android: Enable swarming on Android arm64 builder (bpastene@chromium.org)
  https://crrev.com/7a71133c89aa2a770fc2326188306cf6a3a6f1e1 Revert of Enable the ninja up-to-date check for Android builders (patchset #1 id:1 of https://codereview.chromium.org/2343953002/ ) (agrieve@chromium.org)
  https://crrev.com/c65424c419105e0720cc0de09fb9e7a36c3d4601 Pass ninja -n in the ninja -d explain step to avoid more work (agrieve@chromium.org)
  https://crrev.com/8091fad9d075ecec116e1740360ff955a7654c4a Remove use_isolate from Marshmallow 64 bit Tester (bpastene@chromium.org)
  https://crrev.com/2f0a7c064259dc0e0bf375c2d1771544f5647a1c Changed sequence of package_build step to be before package_build_for_bisect (miimnk@google.com)
  https://crrev.com/194bd0247efb60371dbcae87731f13dce4fa7934 Roll recipe dependencies (trivial). (recipe-roller@chromium.org)
  https://crrev.com/d2845f7e87aa167cf7df78003b5768f948803d07 Roll recipe dependencies (trivial). (recipe-roller@chromium.org)
  https://crrev.com/eb686c79229ef3cf3b39e6a0807f4c98e3e93cf9 Add BuildBucket manifest scheduling support. (dnj@chromium.org)
  https://crrev.com/537cdb699a70d5d55f3eefb923eae1dc5104a8eb Add goma to wasm waterfall (sbc@chromium.org)
  https://crrev.com/b6d326d6f6f9608bba307109cc4a1c5efc5946bb Run the ninja "up-to-date" check for all compiles, but just as an fyi (agrieve@chromium.org)
  https://crrev.com/07a33586eff8451bbc84366055c3cfa36e7b6fea Roll recipe dependencies (nontrivial). (recipe-roller@chromium.org)
  https://crrev.com/79d8d758281609aa49b9b9af7e18cc187c2ee814 Disable CompilerInfoCache for a while (shinyak@chromium.org)
  https://crrev.com/a8f07009101d46553ea5d4bce1353ecb365dd3a5 Revert "Roll recipe dependencies (nontrivial)." (tandrii@chromium.org)
  https://crrev.com/cbca79967afa4f28df6b0ede337db6138eeb167b Add asan=1 to GYP_DEFINES for Dart asan builds (whesse@google.com)
  https://crrev.com/25b922d8edc1927c3af4bb4a8f7db6434096dca0 WebRTC: Disable iOS API Framework Builder. (ehmaldonado@chromium.org)
  https://crrev.com/769dbabae3b7ff11023dd1b393fcd2a12e165c65 Make sure goma_ctl is running during wasm_llvm build (sbc@chromium.org)
  https://crrev.com/2e89429885f7c2503565cb12d564d0f6f58a42f1 Fix Webkit capitalization for N CTS tests. (mikecase@chromium.org)
  https://crrev.com/e549894cc4f86f3f2107e3686d561ff83331a267 crashpad/continuous: Use bot_update. (dnj@chromium.org)
  https://crrev.com/f74386f7e19ff3f32693d222262d1a3ee79799ef Roll recipe dependencies (trivial). (recipe-roller@chromium.org)
  https://crrev.com/1a9a50a459c9e3c081719d3e835cac213c319136 Roll recipe dependencies (trivial). (recipe-roller@chromium.org)
depot_tools:
  https://crrev.com/f46c20fcee6e6a0a7d75788847632cd4ac18e2e9 codereview.settings: add GIT_NUMBER_FOOTER setting. (tandrii@chromium.org)
  https://crrev.com/5d0a0421ce27046c94177511a05699316ec8097a git_cl: update outdated TODOs. (tandrii@chromium.org)
  https://crrev.com/73449b0bd49eab1e152f419102123d734896da98 Gerrit git cl land: abort if not uploaded. (tandrii@chromium.org)
  https://crrev.com/bf42940536f6c0c123a9e6278c20bc38729b3717 git cl land to refs/pending: remove unused arg. (tandrii@chromium.org)
  https://crrev.com/7475196d4c32d66e1c199bf24945b7ae28255e13 repo: update to v1.12.17-cr1 (vapier@chromium.org)
  https://crrev.com/adcd4b78d5f35da535ccf82a221afb9fa389f631 presubmit_support: Remove a noisy logging.debug() (thakis@chromium.org)
  https://crrev.com/18ca30ca804679ee624a52e73017d234a8c0008f Teach bot_update to remove partially deleted git repos. (vadimsh@chromium.org)
  https://crrev.com/972ac5040176acd90c8a1ce412f75d19f77cc4e8 bot_update: ensure correct depot_tools checkout is used. (tandrii@chromium.org)
  https://crrev.com/15a248123d9032061486cd2d4b3f64369c93a9a5 Revert of bot_update: ensure correct depot_tools checkout is used. (patchset #2 id:20001 of https://codereview.chromium.org/2346973003/ ) (tandrii@chromium.org)
  https://crrev.com/7f245d07b2282f9847072fccddf7162a7e632a2d Bump git-on-windows bleeding edge version to 2.10.0. (vadimsh@chromium.org)
  https://crrev.com/6ac12ffd596e338c43e25dc3889e8ac552c2e885 Make bot_update.py print git version it uses. (vadimsh@chromium.org)
  https://crrev.com/7e16cf303221bbcf81d632924e19ddc888da9c3b owners.py: partial fix for owners-check perf regression (nick@chromium.org)

TBR=martiniss@chromium.org,phajdan.jr@chromium.org
BUG=646165,none,646838,632008,webrtc:6372,chromium:647812,642493,647446,chromium:627996,642793,635641,645662,chromium:632203,647046,609225,642759

Recipe-Tryjob-Bypass-Reason: Autoroller
Bugdroid-Send-Email: False
Review-Url: https://codereview.chromium.org/2345413002

[modify] https://crrev.com/7d6811c4c58a9e280bca7bf65a8dfde001bef42a/infra/config/recipes.cfg

Thanks for taking care of this!