New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 787050 link

Starred by 2 users

Issue metadata

Status: Archived
Owner:
Closed: Dec 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

Flaky PreferencesTest.MultiProfiles

Project Member Reported by yzshen@chromium.org, Nov 20 2017

Issue description

It crashes fairly frequent in mash_browser_tests of Mojo ChromiumOS FYI bot: 
Example link: https://chromium-swarm.appspot.com/task?id=39e9a48ba0796810&refresh=10&show_raw=1&wide_logs=true
There are two different crash stacks on this page:
https://paste.googleplex.com/4800970454204416
https://paste.googleplex.com/5660411529527296

I looked at the flakiness dashboard. This test appears to be quite stable, but it also failed without mash sometimes. E.g., there was one crash on linux-chromeos-rel recently:

https://logs.chromium.org/v/?s=chromium%2Fbb%2Fchromium.chromiumos%2Flinux-chromeos-rel%2F2756%2F%2B%2Frecipes%2Fsteps%2Fbrowser_tests%2F0%2Fstdout

The crash stack is different again:
https://paste.googleplex.com/4854165100560384


I suspect that there is memory corruption (e.g., use-after-free). Maybe running an Asan build (with mash, which seems to make it easier to reproduce) will help us to identify the issue.
 

Comment 1 by sky@chromium.org, Nov 21 2017

Cc: jamescook@chromium.org
 Issue 787121  has been merged into this issue.
Project Member

Comment 3 by bugdroid1@chromium.org, Nov 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/84c9ecb20735c69ed258ac33b8208b8c99365866

commit 84c9ecb20735c69ed258ac33b8208b8c99365866
Author: Jonathan <jonross@chromium.org>
Date: Tue Nov 21 17:12:40 2017

Disable flaking PreferencesTest.MultiProfiles Everywhere

PreferencesTest.MultiProfiles is flaking both on ChromeOS as well as with mash.
This change disables the test everywhere until the flake can be resolved.

This also reverts having the test filtered out for mash.

TBR=jamescook@chromium.org
TEST=PreferencesTest,MultiProfiles

Bug:  787050 
Change-Id: Iaa651e1dd23b836f6d8354fbe9e9d2834f855e0b
Reviewed-on: https://chromium-review.googlesource.com/782041
Reviewed-by: Jonathan Ross <jonross@chromium.org>
Commit-Queue: Jonathan Ross <jonross@chromium.org>
Cr-Commit-Position: refs/heads/master@{#518285}
[modify] https://crrev.com/84c9ecb20735c69ed258ac33b8208b8c99365866/chrome/browser/chromeos/preferences_chromeos_browsertest.cc
[modify] https://crrev.com/84c9ecb20735c69ed258ac33b8208b8c99365866/testing/buildbot/filters/mojo.fyi.mash.browser_tests.filter

Components: Internals>Preferences
Labels: Proj-Mustash-Mash OS-Chrome
Owner: jamescook@chromium.org
Status: Assigned (was: Untriaged)
This may be sunk cost bias, but I spent a while making this test work at all under mash, so I'll try to get it working again. Potential UAF seems bad.

Cc: roc...@chromium.org
Status: Started (was: Assigned)
rockot says the call stacks look like issue 788713 ("Crash Report - mojo::edk::WatcherDispatcher::NotifyHandleState"), which is probably memory corruption somewhere that happens to hit mojo.

I'm going to try to repro locally with asan, but given similar stacks on Android it's possible this issue is unrelated to this test or mash.

Given that there are multiple such stacks from this test I'm hoping it's
simply more prone to triggering whatever the offending codepath is. There
are also similar stacks on Windows FWIW, though I haven't seen any on bots.
I found the mash problem -- it's a heap buffer overwrite, caused by incorrectly static_cast-ing a stub object to a real impl and writing into it. That's consistent with other "mojo crashes" that are caused by unrelated memory corruption.

I've run the fixed test hundreds of times, both with --mash and without. I suspect the crash yzshen@ saw is unrelated and may have been fixed separately.

I'm going to fix the test for --mash and re-enable everywhere.

Project Member

Comment 8 by bugdroid1@chromium.org, Dec 2 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0b38d85981910c08d453b4823fdf467c26b429f5

commit 0b38d85981910c08d453b4823fdf467c26b429f5
Author: James Cook <jamescook@chromium.org>
Date: Sat Dec 02 00:04:03 2017

cros: Fix PreferencesTest.MultiProfiles and re-enable everywhere

The crashes under --mash are due to heap corruption from code trying
to disable multi-user switching animations. That code does not seem
necessary any more. Per crrev.com/223823004 when it was added this
test was reliably failing. The multi-user switch animations have
been recently refactored and a shelf animation removed. The test
now passes even without the animation disabling code.

I cannot reproduce failures or crashes without --mash. I've done
ASAN runs as well. I suspect the non-mash problem in the bug is
unrelated to this test and may have been fixed separately.

Bug:  787050 
Test: browser_tests --gtest_filter=PreferencesTest.* --gtest_repeat=100
Change-Id: I08dc3897361440fdbd49df1a11429e06fb9bac51
Reviewed-on: https://chromium-review.googlesource.com/804401
Reviewed-by: Michael Wasserman <msw@chromium.org>
Commit-Queue: James Cook <jamescook@chromium.org>
Cr-Commit-Position: refs/heads/master@{#521134}
[modify] https://crrev.com/0b38d85981910c08d453b4823fdf467c26b429f5/chrome/browser/chromeos/preferences_chromeos_browsertest.cc

Cc: -jamescook@chromium.org
Status: Fixed (was: Started)
Please re-disable the test and reopen this bug if you see flake.

I haven't seen any flake in the last 3 days, so I think this is fixed.

https://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=PreferencesTest.MultiProfiles
Status: Archived (was: Fixed)

Sign in to add a comment