Assigning to ahassani@. Please re-assign as necessary.

I think this is the same symptom of  crbug.com/905704  (I verified).
Assigning this to chowes@ to mark it as fixed once their changes landed.

@chowes: you can check this by running:
sudo FEATURES=test USE=asan emerge update_engine

and makeing sure it passes with no error.

The errors:

update_engine-0.0.3-r3298: [       OK ] UpdateAttempterTest.ResetRollbackHappenedEnterprise (2 ms)
update_engine-0.0.3-r3298: [ RUN      ] UpdateAttempterTest.SetRollbackHappenedRollback
update_engine-0.0.3-r3298: [INFO:update_attempter.cc(996)] Processing Done.
update_engine-0.0.3-r3298: [ERROR:delta_performer.cc(1784)] prefs->SetInt64(kPrefsUpdateStateNextOperation, kUpdateStateOperationInvalid) failed.
update_engine-0.0.3-r3298: [INFO:update_attempter.cc(1053)] DLC successfully installed, no reboot needed.
update_engine-0.0.3-r3298: ../../../../../../../tmp/portage/chromeos-base/update_engine-0.0.3-r3298/work/update_engine-0.0.3/aosp/system/update_engine/update_attempter_unittest.cc:1416: Failure
update_engine-0.0.3-r3298: Actual function call count doesn't match EXPECT_CALL(*fake_system_state_.mock_payload_state(), SetRollbackHappened(true))...
update_engine-0.0.3-r3298:          Expected: to be called once
update_engine-0.0.3-r3298:            Actual: never called - unsatisfied and active
update_engine-0.0.3-r3298: [  FAILED  ] UpdateAttempterTest.SetRollbackHappenedRollback (1 ms)

This should have been fixed by https://chromium-review.googlesource.com/c/aosp/platform/system/update_engine/+/1359109/.

ah, there might be something more about it:

TEST_F(UpdateAttempterTest, SetRollbackHappenedRollback) {
  attempter_.install_plan_.reset(new InstallPlan);
  attempter_.install_plan_->is_rollback = true;

  EXPECT_CALL(*fake_system_state_.mock_payload_state(),
              SetRollbackHappened(true))
      .Times(1);
  attempter_.ProcessingDone(nullptr, ErrorCode::kSuccess);
}

This test is not calling Rollback function so is_install is not set to be false. So the unit test needs to be fixed.

Let me take this over.

But don't we set it to false in the ctor?

also this is not producible on a build without asan (otherwise CQ will fail), this is probably a bug in gtest (testing object not being reconstructed each time a test fixture is running) or llvm compiler itself. 

I can reproduce it with:
sudo FEATURES=test USE=asan emerge update_engine

yes but update_attempter is reused (ctor is called once at the beginning) across fixtures (is_install is set to true at one point) due to an unknown bug...

sure, amd64-generic-asan can also reproduce this reliably.

re #8 that that is not true, THe update attempter is initialized for each test fixture.

Actually I think this bug is already resolved with Colin's fixing the ctor initialization.

re #6. This is WAI. The asan builders are separated builders that are different with normal builders and are not run in the CQ because they are slow. That is why once in a while asan builders fail and we have to fix the root cause :)

Colin's fix is irrelevant to this bug. 

My point: there is a bug outside of update_engine that causes update_engine_unittest to behave differently with asan or not.

then probably we should not fix update_engine since it's not its problem in the first place. :)

assigning back to toolchain team for further triaging cause I suspect this may cause other test regression as well. so let's fix them all by rooting cause this.

well, collin's fix, fixed the issue on my build :) so I'm not really sure what else could be going on here ;-)

When I ran it locally the test didn't seem to respect my local changes (I
cherry picked an old commit and the problem persisted, with logging that
didn't exist). xiaochu suggested it might be an issue with specifying a
different flag after running setup_board? Anyway, can we run these tests on
a trybot to be sure?

On Tue, Dec 4, 2018, 17:27 ahassani via monorail <
monorail+v2.3083992306@chromium.org wrote:

check it here: https://cros-goldeneye.corp.google.com/chromeos/legoland/builderHistory?buildConfig=amd64-generic-asan&buildBranch=master

it is still RED!

It probably hasn't run since the fix went in though right?

On Tue, Dec 4, 2018, 17:34 xiaochu via monorail <
monorail+v2.3576894634@chromium.org wrote:

the last RED run is at 2018-12-04 15:36

ah, that builder is running release branch it looks like:R73-11345.0.0-b3191852 instead of ToT. So let's wait to see.

still it doesn't explain why the unittest only fails on asan builders.

It is red, but there could be other packages failing too? if you would've looked at the logs you'd see that mostly shill unittests are failing (and some other packages). there is no sign the update_engine is failing. We would know that for sure other packages have successfully fixed.

But if the local unittests with asan are passing, we can close this bug. it can be reopened if it persists.

Thanks, looks like last runs of asan and ubsan builders do not have update_engine failing anymore.

Note that R73-11345.0.0-b3191852 is a build on the master branch (I don't know how the numbers are generated but R72 is the current beta branch).

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8928025921888873008

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8928028330239177232