chromite.lib.purge_lib_unittest testListRemoteBranches flake in CQ |
||||
Issue descriptionThis failed once on tricky-paladin: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8926688645099306576 chromite-0.0.2-r4351: chromite-0.0.2-r4351: ====================================================================== chromite-0.0.2-r4351: FAIL: [chromite.lib.purge_lib_unittest] TestHelperMethods.testListRemoteBranches chromite-0.0.2-r4351: ---------------------------------------------------------------------- chromite-0.0.2-r4351: Traceback (most recent call last): chromite-0.0.2-r4351: File "/mnt/host/source/chromite/lib/timeout_util.py", line 191, in TimeoutWrapper chromite-0.0.2-r4351: return func(*args, **kwargs) chromite-0.0.2-r4351: File "/mnt/host/source/chromite/lib/purge_lib_unittest.py", line 41, in testListRemoteBranches chromite-0.0.2-r4351: self.assertGreater(len(branches), 300) chromite-0.0.2-r4351: AssertionError: 6 not greater than 300 chromite-0.0.2-r4351: chromite-0.0.2-r4351: ====================================================================== chromite-0.0.2-r4351: FAIL: [chromite.lib.purge_lib_unittest] TestHelperMethods.testProtectedBranchVersionsLive chromite-0.0.2-r4351: ---------------------------------------------------------------------- chromite-0.0.2-r4351: Traceback (most recent call last): chromite-0.0.2-r4351: File "/mnt/host/source/chromite/lib/timeout_util.py", line 191, in TimeoutWrapper chromite-0.0.2-r4351: return func(*args, **kwargs) chromite-0.0.2-r4351: File "/mnt/host/source/chromite/lib/purge_lib_unittest.py", line 57, in testProtectedBranchVersionsLive chromite-0.0.2-r4351: self.assertGreater(len(branch_versions), 100) chromite-0.0.2-r4351: AssertionError: 0 not greater than 100 chromite-0.0.2-r4351: chromite-0.0.2-r4351: ----------------------------------------------------------------------
,
Dec 19
This seems to have happened again on tricky-paladin in the next build: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8926675748461527984 Any chance this particular machine has some leftover git cruft that can be cleaned up somehow?
,
Dec 19
It... shouldn't?? I'm wiping the builder, just in case. It will slow the start of the next build on it.
,
Dec 19
The text in question hasn't substantially changed in several years, and is NOT board specific in any way, so some kind of builder corruption makes the most sense.
,
Jan 3
Failed again on tricky-paladin: https://ci.chromium.org/p/chromeos/builders/luci.chromeos.general/CQ/b8925325699855301952
,
Jan 4
2 more failures on tricky-paladin, so changing to P1. https://ci.chromium.org/p/chromeos/builds/b8925301817712469024 https://ci.chromium.org/p/chromeos/builds/b8925289015649446192
,
Jan 4
dgarret is out; who's working on this?
,
Jan 4
Also failing on reef-paladin: https://ci.chromium.org/p/chromeos/builds/b8925281425482389856 https://ci.chromium.org/p/chromeos/builds/b8925267545060775184
,
Jan 4
The failures in #6 and #8 (and another new one: https://ci.chromium.org/p/chromeos/builds/b8925253832303887616) are all happening on the same Bot: swarm-cros-357
,
Jan 4
All that this test does is locate the chromite directory and run "git branch -lr" counting the number of lines. If it's greater than 300, the test passes. It seems that the git repo is corrupted. Investigating.
,
Jan 4
I SSH'd to the bot and entered the directory and ran the same command. It came back with 587 remotes. So, it doesn't seem to be corrupted. Continuing to investigate.
,
Jan 4
Found another copy of the repo on the disk that is indeed corrupted. And it returns 4 instead of 587 just like the failing test log. Nuking it.
,
Jan 4
I think it's fixed now by running git fetch and git fsck on the directory; will monitor the bot.
,
Jan 4
New CQ build on this bot got past the breakage point. Fixed. |
||||
►
Sign in to add a comment |
||||
Comment 1 by dgarr...@chromium.org
, Dec 19