bot_update fails due to `git.bat checkout --force origin/master failed` on windows builders |
||||
Issue descriptionOS: Windows We're seeing bot_update failures (~350 this week, ~950 this month) on windows builders. Most guilty builders: win_chromium_x64_rel_ng, win_chromium_rel_ng, win_chromium_compile_dbg_ng. (Queries: https://plx.corp.google.com/script/#a=qo%7Ci=google%253A%253Ascript_cd._bd07a5_237d_40b0_bbe8_8777713c348f) 244 out of most recent 375 are failing in the following way: ===Running git.bat checkout --force origin/master (attempt #5)=== In directory: E:\b\c\b\win\src fatal: Unable to create 'E:/b/c/b/win/src/.git/index.lock': File exists. Another git process seems to be running in this repository, e.g. an editor opened by 'git commit'. Please make sure all processes Traceback (most recent call last): File "E:\b\rr\tmpihwfyp\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 1099, in <module> sys.exit(main()) File "E:\b\rr\tmpihwfyp\rw\checkout\infra\.recipe_deps\depot_tools\recipe_modules\bot_update\resources\bot_update.py", line 1083, in main are terminated then try again. If it still fails, a git process may have crashed in this repository earlier: remove the file manually to continue. ===Failed in 0.0 mins=== Something failed: git.bat checkout --force origin/master failed with code 128 in E:\b\c\b\win\src after 5 attempts. Next steps: * Look at bot_update.py, is it doing something sane? * Modify bot_update to try killing the guilty process if this happens, and then retrying. It seems like this can't hurt: the step was going to fail anyway, and this is happening on tryserver masters, so the bots are rebooted after the run anyway. * Add more logging (what exactly? TBD) to bot_update to help identify the problem * Having access to the machine while this is happening or shortly after would be helpful. To do this, I'd need to know when this is happening: Add monitoring to bot_update to notify myself if this happens.
,
Sep 30 2016
,
Sep 30 2016
,
Oct 1 2016
,
Oct 11 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/depot_tools.git/+/df66a34f68f4b2f2c08039f178d267c7dafd9d59 commit df66a34f68f4b2f2c08039f178d267c7dafd9d59 Author: katthomas <katthomas@google.com> Date: Tue Oct 11 23:55:15 2016 Remove git lockfile flakiness on win (bot_update) Hypothesis: Sometimes bot update fails because windows fails to delete a lockfile associated with a git process. Test: If this happens, let's delete that lockfile and try again. BUG= 651602 Review-Url: https://codereview.chromium.org/2382653005 [modify] https://crrev.com/df66a34f68f4b2f2c08039f178d267c7dafd9d59/recipe_modules/bot_update/resources/bot_update.py [modify] https://crrev.com/df66a34f68f4b2f2c08039f178d267c7dafd9d59/tests/bot_update_coverage_test.py
,
Oct 12 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build.git/+/9b5fda749ed57fdf79c5127fa437a6aaa504ea8d commit 9b5fda749ed57fdf79c5127fa437a6aaa504ea8d Author: recipe-roller <recipe-roller@chromium.org> Date: Wed Oct 12 00:04:35 2016 Roll recipe dependencies (trivial). This is an automated CL created by the recipe roller. This CL rolls recipe changes from upstream projects (e.g. depot_tools) into downstream projects (e.g. tools/build). More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug (or complain) depot_tools: https://crrev.com/df66a34f68f4b2f2c08039f178d267c7dafd9d59 Remove git lockfile flakiness on win (bot_update) (katthomas@google.com) TBR=martiniss@chromium.org,phajdan.jr@chromium.org BUG= 651602 Recipe-Tryjob-Bypass-Reason: Autoroller Bugdroid-Send-Email: False Review-Url: https://codereview.chromium.org/2416453003 [modify] https://crrev.com/9b5fda749ed57fdf79c5127fa437a6aaa504ea8d/infra/config/recipes.cfg
,
Oct 12 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/6126ee5fb59ae670442e4a602e41a6fad1cee1f9 commit 6126ee5fb59ae670442e4a602e41a6fad1cee1f9 Author: recipe-roller <recipe-roller@chromium.org> Date: Wed Oct 12 00:35:09 2016 Roll recipe dependencies (trivial). This is an automated CL created by the recipe roller. This CL rolls recipe changes from upstream projects (e.g. depot_tools) into downstream projects (e.g. tools/build). More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug (or complain) build: https://crrev.com/9b5fda749ed57fdf79c5127fa437a6aaa504ea8d Roll recipe dependencies (trivial). (recipe-roller@chromium.org) depot_tools: https://crrev.com/df66a34f68f4b2f2c08039f178d267c7dafd9d59 Remove git lockfile flakiness on win (bot_update) (katthomas@google.com) TBR=martiniss@chromium.org,phajdan.jr@chromium.org BUG= 651602 Recipe-Tryjob-Bypass-Reason: Autoroller Bugdroid-Send-Email: False Review-Url: https://codereview.chromium.org/2405143005 Cr-Commit-Position: refs/heads/master@{#424614} [modify] https://crrev.com/6126ee5fb59ae670442e4a602e41a6fad1cee1f9/infra/config/recipes.cfg
,
Oct 12 2016
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra.git/+/d88b106fb33dd22dcbfe755118a4140b812cc64a commit d88b106fb33dd22dcbfe755118a4140b812cc64a Author: recipe-roller <recipe-roller@chromium.org> Date: Wed Oct 12 00:57:26 2016 Roll recipe dependencies (trivial). This is an automated CL created by the recipe roller. This CL rolls recipe changes from upstream projects (e.g. depot_tools) into downstream projects (e.g. tools/build). More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug (or complain) build: https://crrev.com/9b5fda749ed57fdf79c5127fa437a6aaa504ea8d Roll recipe dependencies (trivial). (recipe-roller@chromium.org) depot_tools: https://crrev.com/df66a34f68f4b2f2c08039f178d267c7dafd9d59 Remove git lockfile flakiness on win (bot_update) (katthomas@google.com) TBR=martiniss@chromium.org,phajdan.jr@chromium.org BUG= 651602 Recipe-Tryjob-Bypass-Reason: Autoroller Bugdroid-Send-Email: False Review-Url: https://codereview.chromium.org/2412803002 [modify] https://crrev.com/d88b106fb33dd22dcbfe755118a4140b812cc64a/infra/config/recipes.cfg
,
Oct 26 2016
|
||||
►
Sign in to add a comment |
||||
Comment 1 by katthomas@chromium.org
, Sep 30 2016