Mike, why the builder_run.config is None? Add Don

no idea.  looks weird though.

Find unittest failure on another builder too.
https://uberchromegw.corp.google.com/i/chromeos/builders/x86-alex-paladin/builds/24441/steps/UnitTest/logs/stdio

There IS a build config in general, as can be seen in this earlier build step.

https://uberchromegw.corp.google.com/i/chromeos/builders/guado_moblab-paladin/builds/2597/steps/ConfigDump/logs/stdio


My first thought is that a bad CL was being tested.... but there aren't any CLs being tested.

The only recent CL that seems at all relevant is here, but I don't see how it could be the cause.

https://chromium-review.googlesource.com/#/c/343564/4

That CL was landed yesterday afternoon, and this failure first happened on yesterday night. It is very likely it is because of that CL. 

The changes in the https://chromium-review.googlesource.com/#/c/343564/4 touched DistributedBuilder, which is a subclass of SimpleBuilder. so would say the changes shouldn't affect testRunStagesChrootBuilder, which only calls SimpleBuilder.

No ideas why builder_run.config became None there. And the followed builds passed the UnitTestStage. Maybe adding more logs can help to catch this error?

I assume that you ran the simple_builder_unittest locally and passed, right? If this error happens again, we should add some loggings help debug.

I run it locally before and after the merging, they passed. 

There is an other build that failed with the same error . https://uberchromegw.corp.google.com/i/chromeos/builders/monroe-release/builds/637/steps/UnitTest/logs/stdio

Looks like the UnitTest https://uberchromegw.corp.google.com/i/chromeos/builders/monroe-release/builds/637/steps/UnitTest/logs/stdio is unstable, it first failed at [chromite.lib.patch_unittest] TestGerritPatch.testAddFooters, and failed at SimpleBuilderTest.testRunStagesChrootBuilder when retried.

chromite-0.0.2-r2380: 
chromite-0.0.2-r2380: ======================================================================
chromite-0.0.2-r2380: ERROR: [chromite.lib.patch_unittest] TestGerritPatch.testAddFooters
chromite-0.0.2-r2380: ----------------------------------------------------------------------
chromite-0.0.2-r2380: Traceback (most recent call last):
chromite-0.0.2-r2380:   File "/mnt/host/source/chromite/lib/cros_test_lib.py", line 262, in _stacked_setUp
chromite-0.0.2-r2380:     target(obj)
chromite-0.0.2-r2380:   File "/mnt/host/source/chromite/lib/patch_unittest.py", line 105, in setUp
chromite-0.0.2-r2380:     self._CreateSourceRepo(self.source)
chromite-0.0.2-r2380:   File "/mnt/host/source/chromite/lib/patch_unittest.py", line 99, in _CreateSourceRepo
chromite-0.0.2-r2380:     cwd=tmp_path)
chromite-0.0.2-r2380:   File "/mnt/host/source/chromite/lib/patch_unittest.py", line 132, in _run
chromite-0.0.2-r2380:     cmd, cwd=cwd, print_cmd=False, capture_output=True).output.strip()
chromite-0.0.2-r2380:   File "/mnt/host/source/chromite/lib/cros_build_lib.py", line 619, in RunCommand
chromite-0.0.2-r2380:     raise RunCommandError(msg, cmd_result)
chromite-0.0.2-r2380: RunCommandError: return code: 128; command: git commit --allow-empty -m 'initial commit'
chromite-0.0.2-r2380: 
chromite-0.0.2-r2380: *** Please tell me who you are.
chromite-0.0.2-r2380: 
chromite-0.0.2-r2380: Run
chromite-0.0.2-r2380: 
chromite-0.0.2-r2380:   git config --global user.email "you@example.com"
chromite-0.0.2-r2380:   git config --global user.name "Your Name"
chromite-0.0.2-r2380: 
chromite-0.0.2-r2380: to set your account's default identity.
chromite-0.0.2-r2380: Omit --global to set the identity only in this repository.
chromite-0.0.2-r2380: 
chromite-0.0.2-r2380: fatal: unable to auto-detect email address (got 'chrome-bot@cros-beefy47-c2.(none)')
chromite-0.0.2-r2380: 
chromite-0.0.2-r2380: cwd=/tmp/chromite.run_tests.XjtY4p/chromite.testpORzoq/source.git-tmp
chromite-0.0.2-r2380: 
chromite-0.0.2-r2380: ----------------------------------------------------------------------
chromite-0.0.2-r2380: Ran 50 tests in 114.005s
chromite-0.0.2-r2380: 
chromite-0.0.2-r2380: FAILED (errors=1)

Interesting.

We no longer pre-configure things in ".gitconfig" when setting up a new GCE builder, since we use service account based authentication. It may be possible to fix this by running the tests inside the chroot. Though... I'm surprised that they would be flaky, instead of failing every single time.

Also, that failure looks very different from the initial error, even if it's in the same file.

To run the test in question inside the chroot, add it to the following "SPECIAL" list.

https://cs.corp.google.com/chromeos_public/chromite/cbuildbot/run_tests.py?rcl=eb62769e34bd3fbee8d6c70712347c67757739c5&l=65

hit this again: https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_minnie-cheets-paladin/builds/1093

https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_minnie-cheets-paladin/builds/1093/steps/UnitTest/logs/stdio

This builder run simple_builders_unittest twice, the first run failed at:

chromite-0.0.2-r2406: ERROR: [chromite.cbuildbot.builders.simple_builders_unittest] SimpleBuilderTest.testRunStagesChrootBuilder
chromite-0.0.2-r2406: Verify RunStages for CHROOT_BUILDER_TYPE builders
chromite-0.0.2-r2406: ----------------------------------------------------------------------
chromite-0.0.2-r2406: Traceback (most recent call last):
chromite-0.0.2-r2406:   File "/mnt/host/source/chromite/lib/timeout_util.py", line 170, in TimeoutWrapper
chromite-0.0.2-r2406:     func(*args, **kwargs)
chromite-0.0.2-r2406:   File "/mnt/host/source/chromite/cbuildbot/builders/simple_builders_unittest.py", line 105, in testRunStagesChrootBuilder
chromite-0.0.2-r2406:     simple_builders.SimpleBuilder(builder_run).RunStages()
chromite-0.0.2-r2406:   File "/mnt/host/source/chromite/cbuildbot/builders/simple_builders.py", line 333, in RunStages
chromite-0.0.2-r2406:     self._RunDefaultTypeBuild()
chromite-0.0.2-r2406:   File "/mnt/host/source/chromite/cbuildbot/builders/simple_builders.py", line 320, in _RunDefaultTypeBuild
chromite-0.0.2-r2406:     self.RunBuildStages()
chromite-0.0.2-r2406:   File "/mnt/host/source/chromite/cbuildbot/builders/simple_builders.py", line 287, in RunBuildStages
chromite-0.0.2-r2406:     if builder_run.config.afdo_generate_min:
chromite-0.0.2-r2406: AttributeError: 'NoneType' object has no attribute 'afdo_generate_min'



The second run failed at:

chromite-0.0.2-r2406: ERROR: [chromite.cbuildbot.builders.simple_builders_unittest] SimpleBuilderTest.testRunStagesChrootBuilder
chromite-0.0.2-r2406: Verify RunStages for CHROOT_BUILDER_TYPE builders
chromite-0.0.2-r2406: ----------------------------------------------------------------------
chromite-0.0.2-r2406: Traceback (most recent call last):
chromite-0.0.2-r2406:   File "/mnt/host/source/chromite/lib/timeout_util.py", line 170, in TimeoutWrapper
chromite-0.0.2-r2406:     func(*args, **kwargs)
chromite-0.0.2-r2406:   File "/mnt/host/source/chromite/cbuildbot/builders/simple_builders_unittest.py", line 105, in testRunStagesChrootBuilder
chromite-0.0.2-r2406:     simple_builders.SimpleBuilder(builder_run).RunStages()
chromite-0.0.2-r2406:   File "/mnt/host/source/chromite/cbuildbot/builders/simple_builders.py", line 333, in RunStages
chromite-0.0.2-r2406:     self._RunDefaultTypeBuild()
chromite-0.0.2-r2406:   File "/mnt/host/source/chromite/cbuildbot/builders/simple_builders.py", line 320, in _RunDefaultTypeBuild
chromite-0.0.2-r2406:     self.RunBuildStages()
chromite-0.0.2-r2406:   File "/mnt/host/source/chromite/cbuildbot/builders/simple_builders.py", line 283, in RunBuildStages
chromite-0.0.2-r2406:     if not builder_run.config.build_packages_in_background:
chromite-0.0.2-r2406: AttributeError: 'NoneType' object has no attribute 'build_packages_in_background'
chromite-0.0.2-r2406: 


Is testRunStagesChrootBuilder an important test case, is that possible to turn it off?

In both cases, that looks like a build config had a value of None.

That sounds like the test cases aren't properly setting up the config.

But if it's flaky, that sounds like a race condition. Are the stages in question multiprocess or something?

This is still happening today and blocking CQ randomly.
https://uberchromegw.corp.google.com/i/chromeos/builders/beaglebone-paladin/builds/11015

observed before in  crbug.com/470907 

+akeshet who closed the old bug.

Should we reopen the old one and dup them together? 

Sure go for it.