chromeos-base/chromite unit-test stage failing due to gclient ordering |
||||||||||||
Issue descriptionFiled by sheriff-o-matic@appspot.gserviceaccount.com on behalf of coconutruben@google.com chromeos-base/chromite unit-test stage failing Builders failed on: - betty-arc64-paladin: http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8947669457083680768 - betty-arc64-release: http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8947677461115928496 - betty-paladin: http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8947669458985327552 - betty-release: http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8947677470599794320 - newbie-release: http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8947677476352247872 - novato-arc64-release: http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8947677496409305600 - novato-release: http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8947677476730931376 - moblab-generic-vm-paladin: http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8947669436305652064 Multiple release and paladin builders have been failing with the same error pattern of the chromite unittest stage failing. I didn't understand the conversation in cros-oncall about this, but maybe crrev.com/c/1037920 is related.
,
May 2 2018
,
May 2 2018
It seems that the failure might be caused here: chromite-0.0.2-r3862: AssertionError: Expected to find call(('gclient', 'config', '--spec', "solutions = [{'custom_deps': {},\n 'custom_vars': {},\n 'deps_file': '.DEPS.git',\n 'managed': True,\n 'name': 'src',\n 'url': 'https://chromium.googlesource.com/chromium/src.git@refs/tags/41.0.2270.0'}]\ntarget_os = ['chromeos']\ncache_dir = '/tmp/b/git-cache'\n"), cwd='/work/chrome') in any of: chromite-0.0.2-r3862: call(['gclient', 'config', '--spec', "solutions = [{'custom_deps': {},\n 'custom_vars': {},\n 'deps_file': '.DEPS.git',\n 'managed': True,\n 'name': 'src',\n 'url': 'https://chromium.googlesource.com/chromium/src.git@refs/tags/41.0.2270.0'}]\ncache_dir = '/tmp/b/git-cache'\ntarget_os = ['chromeos']\n"], cwd='/work/chrome') Note how on the 2nd version target_os comes after the cache, while in the expectation it comes before cache entry. So maybe this is causing our failures
,
May 2 2018
,
May 2 2018
chromite-0.0.2-r3862: [chromite.lib.gclient_unittest] GclientWrappersTest.testRevert ... ok chromite-0.0.2-r3862: [chromite.lib.gclient_unittest] GclientWrappersTest.testSync chromite-0.0.2-r3862: Test gclient.Sync() without optional arguments. ... ok chromite-0.0.2-r3862: [chromite.lib.gclient_unittest] GclientWrappersTest.testSyncWithOptions chromite-0.0.2-r3862: Test gclient.Sync() with optional arguments. ... ok chromite-0.0.2-r3862: [chromite.lib.gclient_unittest] GclientWrappersTest.testSyncWithRunArgs chromite-0.0.2-r3862: Test gclient.Sync() with run_args. ... ok chromite-0.0.2-r3862: [chromite.lib.gclient_unittest] TestGclientWriteConfigFile.testChromeSpec chromite-0.0.2-r3862: Test WriteConfigFile with chrome checkout and no revision. ... FAIL chromite-0.0.2-r3862: [chromite.lib.gclient_unittest] TestGclientWriteConfigFile.testChromeSpecDepsResolution chromite-0.0.2-r3862: Test BuildspecUsesDepsGit at release thresholds. ... ok chromite-0.0.2-r3862: [chromite.lib.gclient_unittest] TestGclientWriteConfigFile.testChromeSpecWithGclientTemplate chromite-0.0.2-r3862: Test WriteConfigFile with chrome checkout with a gclient template. ... FAIL chromite-0.0.2-r3862: [chromite.lib.gclient_unittest] TestGclientWriteConfigFile.testChromeSpecWithGitHash chromite-0.0.2-r3862: Test WriteConfigFile with chrome checkout at a given git revision. ... FAIL chromite-0.0.2-r3862: [chromite.lib.gclient_unittest] TestGclientWriteConfigFile.testChromeSpecWithGitHashNoManaged chromite-0.0.2-r3862: Like testChromeSpecWithGitHash() but with "managed" sets to False. ... FAIL chromite-0.0.2-r3862: [chromite.lib.gclient_unittest] TestGclientWriteConfigFile.testChromeSpecWithGitHead chromite-0.0.2-r3862: Test WriteConfigFile with chrome checkout at a given git revision. ... FAIL chromite-0.0.2-r3862: [chromite.lib.gclient_unittest] TestGclientWriteConfigFile.testChromeSpecWithReleaseTag chromite-0.0.2-r3862: Test WriteConfigFile with chrome checkout at a given release tag. ... FAIL chromite-0.0.2-r3862: [chromite.lib.gclient_unittest] TestGclientWriteConfigFile.testChromeSpecWithReleaseTagDepsGit chromite-0.0.2-r3862: Test WriteConfigFile with chrome checkout at a given release tag. ... FAIL chromite-0.0.2-r3862: [chromite.lib.gclient_unittest] TestGclientWriteConfigFile.testChromiumSpec chromite-0.0.2-r3862: Test WriteConfigFile with chromium checkout and no revision. ... FAIL chromite-0.0.2-r3862: [chromite.lib.gclient_unittest] TestGclientWriteConfigFile.testChromiumSpecNotUseCache chromite-0.0.2-r3862: Test WriteConfigFile with chromium checkout and no revision. ... ok chromite-0.0.2-r3862: [chromite.lib.gclient_unittest] TestGclientWriteConfigFile.testChromiumSpecWithGitHash chromite-0.0.2-r3862: Test WriteConfigFile with chromium checkout at a given git revision. ... FAIL chromite-0.0.2-r3862: [chromite.lib.gclient_unittest] TestGclientWriteConfigFile.testChromiumSpecWithGitHead chromite-0.0.2-r3862: Test WriteConfigFile with chromium checkout at a given git revision. ... FAIL chromite-0.0.2-r3862: [chromite.lib.gclient_unittest] TestGclientWriteConfigFile.testChromiumSpecWithReleaseTag chromite-0.0.2-r3862: Test WriteConfigFile with chromium checkout at a given release tag. ... FAIL Also interesting that the test not using the cache is among the ones passing.
,
May 2 2018
Could we modify gclient.py to append the target_os before the cache_dir (rather than after as done by https://chromium-review.googlesource.com/c/chromiumos/chromite/+/1037920/7/lib/gclient.py) so the expectations match? - says the ARC++ gardener who is out of his depth here...
,
May 2 2018
#6 is worth trying...
,
May 2 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/5295eaac6d224a44bac89b9b919bff2532ffcda4 commit 5295eaac6d224a44bac89b9b919bff2532ffcda4 Author: Josh Horwich <jhorwich@chromium.org> Date: Wed May 02 20:30:46 2018 Reorder target_os / cache_dir in gclient spec Unit tests on builders were failing due to mis-ordered spec vs. expectatoin. This moves target_os before cache_dir to align with expectations. BUG=chromium:838847 TEST=chromite/lib/gclient_unittest Change-Id: I7a4cc57a3446873cd993ef29ea895d657503f4b3 Reviewed-on: https://chromium-review.googlesource.com/1040305 Trybot-Ready: Josh Horwich <jhorwich@chromium.org> Reviewed-by: Don Garrett <dgarrett@chromium.org> Reviewed-by: Agi Csaki <agicsaki@chromium.org> Commit-Queue: Josh Horwich <jhorwich@chromium.org> Tested-by: Josh Horwich <jhorwich@chromium.org> [modify] https://crrev.com/5295eaac6d224a44bac89b9b919bff2532ffcda4/lib/gclient.py
,
May 2 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/039b813669fae464174b6ba21533779ef9c0c640 commit 039b813669fae464174b6ba21533779ef9c0c640 Author: chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Date: Wed May 02 22:24:19 2018 Roll src/third_party/chromite/ 9dea5038c..5295eaac6 (1 commit) https://chromium.googlesource.com/chromiumos/chromite.git/+log/9dea5038cdf1..5295eaac6d22 $ git log 9dea5038c..5295eaac6 --date=short --no-merges --format='%ad %ae %s' 2018-05-02 jhorwich Reorder target_os / cache_dir in gclient spec Created with: roll-dep src/third_party/chromite BUG=chromium:838847 The AutoRoll server is located here: https://chromite-chromium-roll.skia.org Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, please contact the current sheriff, who should be CC'd on the roll, and stop the roller if necessary. TBR=chrome-os-gardeners@chromium.org Change-Id: I48c42f1dab561e0fc5f8c18287dbd0b7acb6954f Reviewed-on: https://chromium-review.googlesource.com/1040770 Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Cr-Commit-Position: refs/heads/master@{#555578} [modify] https://crrev.com/039b813669fae464174b6ba21533779ef9c0c640/DEPS
,
May 3 2018
I believe the above (comment #8 and comment #9) changes should clear this error.
,
May 3 2018
going by the comments on the CL, we didn't actually fix the bad tests, just worked around them. can we fix the actual tests ?
,
May 3 2018
Agreed, now that we have a workaround, can we make it robust?
,
May 4 2018
Just as a note, that fix seems to have worked.
,
May 8 2018
,
May 8 2018
,
May 8 2018
,
Aug 2
,
Oct 17
|
||||||||||||
►
Sign in to add a comment |
||||||||||||
Comment 1 by coconutruben@chromium.org
, May 2 2018Components: Infra>Client>ChromeOS
Labels: Hotlist-CrOS-Sheriffing