New issue
Advanced search Search tips

Issue 757243 link

Starred by 1 user

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Mac
Pri: 3
Type: Bug-Regression



Sign in to add a comment

Some FeedbackTest tests are flaky on Mac ASan 64 Tests (1)

Project Member Reported by kjellander@chromium.org, Aug 20 2017

Issue description

These tests are flaky on Mac ASan 64 Tests (1):
* FeedbackTest.ShowLoginFeedback
* FeedbackTest.AnonymousUser

The first observed failure is https://build.chromium.org/p/chromium.memory/builders/Mac%20ASan%2064%20Tests%20%281%29/builds/33281
but it's hard to find any culprit in that CL's blamelist.
Assigning to xiyuan@chromium.org and CCing afakhry@chromium.org according to git blame for https://cs.chromium.org/chromium/src/chrome/browser/extensions/api/feedback_private/feedback_browsertest.cc
 
Project Member

Comment 1 by bugdroid1@chromium.org, Aug 20 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/60d9f6888c6911f4d64c7bff1d2bd5e3a8c36b1b

commit 60d9f6888c6911f4d64c7bff1d2bd5e3a8c36b1b
Author: Henrik Kjellander <kjellander@chromium.org>
Date: Sun Aug 20 17:43:47 2017

Disable FeedbackTest.{AnonymousUser,ShowLoginFeedback} for Mac ASan

These tests are flaky on Mac ASan 64 Tests (1)
https://build.chromium.org/p/chromium.memory/builders/Mac%20ASan%2064%20Tests%20%281%29/builds/33281

BUG=757243
TBR=xiyuan@chromium.org

Change-Id: Ida871b11e2c195b07b3ccf68392f25530c911e36
Reviewed-on: https://chromium-review.googlesource.com/622847
Reviewed-by: Henrik Kjellander <kjellander@chromium.org>
Commit-Queue: Henrik Kjellander <kjellander@chromium.org>
Cr-Commit-Position: refs/heads/master@{#495843}
[modify] https://crrev.com/60d9f6888c6911f4d64c7bff1d2bd5e3a8c36b1b/chrome/browser/extensions/api/feedback_private/feedback_browsertest.cc

Comment 2 by xiy...@chromium.org, Aug 21 2017

Think this is a dup of  issue 756959 . A fix is landed (r495686). The failed build seems does not pick it up.

We can probably just re-enable the tests and see things would run fine.
FeedbackTest.ShowFeedback is also failing flakily.
Summary: Some FeedbackTest tests are flaky on Mac ASan 64 Tests (1) (was: FeedbackTest.{AnonymousUser,ShowLoginFeedback} tests flaky on Mac ASan 64 Tests (1))
Project Member

Comment 5 by bugdroid1@chromium.org, Sep 12 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/702112b13b5d4dae10b6ace5d6c8e2708c868221

commit 702112b13b5d4dae10b6ace5d6c8e2708c868221
Author: Ben Wells <benwells@chromium.org>
Date: Tue Sep 12 07:27:23 2017

Disable flaky FeedbackTest.ShowFeedback

Bug: 757243
Tbr: xiyuan@chromium.org
Change-Id: Icc3358ecd4970fdfba9ee7783c3ef130506e3723
Reviewed-on: https://chromium-review.googlesource.com/662083
Reviewed-by: Ben Wells <benwells@chromium.org>
Commit-Queue: Ben Wells <benwells@chromium.org>
Cr-Commit-Position: refs/heads/master@{#501208}
[modify] https://crrev.com/702112b13b5d4dae10b6ace5d6c8e2708c868221/chrome/browser/extensions/api/feedback_private/feedback_browsertest.cc

BTW, the ASAN failure is gone, at least according to the CLs for a tryjob to re-enable them. The next FeedbackTest failure is  bug 778869 .
Project Member

Comment 7 by bugdroid1@chromium.org, Oct 28 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0ee60de7722a0f5003a0ee85791c07abd2dc6588

commit 0ee60de7722a0f5003a0ee85791c07abd2dc6588
Author: Lei Zhang <thestig@chromium.org>
Date: Sat Oct 28 16:35:33 2017

Re-enable FeedbackTest on ASAN.

The ASAN failure has been fixed.

BUG=757243

Change-Id: I64867f85503aab620d51b6bd2b7a911c31c6bd22
Reviewed-on: https://chromium-review.googlesource.com/741543
Reviewed-by: Xiyuan Xia <xiyuan@chromium.org>
Commit-Queue: Lei Zhang <thestig@chromium.org>
Cr-Commit-Position: refs/heads/master@{#512402}
[modify] https://crrev.com/0ee60de7722a0f5003a0ee85791c07abd2dc6588/chrome/browser/extensions/api/feedback_private/feedback_browsertest.cc

Status: Fixed (was: Assigned)
Issue 764080 has been merged into this issue.
Status: Assigned (was: Fixed)
This test failed on 6 of 13 runs of Mac ASan 64 Tests (1) this morning. I'm going to disable it again.
Project Member

Comment 11 by bugdroid1@chromium.org, Oct 31 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/45a78cec3e3dbfb23fbb9db59101c1eba4805856

commit 45a78cec3e3dbfb23fbb9db59101c1eba4805856
Author: Justin Donnelly <jdonnelly@chromium.org>
Date: Tue Oct 31 19:20:31 2017

Disable flaky FeedbackTest.ShowFeedback on ASan

Bug: 757243
Change-Id: Iba827221e50dea80eeeabb341d1af6067cdb249a
Reviewed-on: https://chromium-review.googlesource.com/747046
Reviewed-by: Ahmed Fakhry <afakhry@chromium.org>
Commit-Queue: Justin Donnelly <jdonnelly@chromium.org>
Cr-Commit-Position: refs/heads/master@{#512903}
[modify] https://crrev.com/45a78cec3e3dbfb23fbb9db59101c1eba4805856/chrome/browser/extensions/api/feedback_private/feedback_browsertest.cc

Cc: guidou@chromium.org
All FeedbackTest cases have started flaking heavily since https://uberchromegw.corp.google.com/i/chromium.memory/builders/Mac%20ASan%2064%20Tests%20%281%29/builds/36626. Suspecting https://chromium-review.googlesource.com/809005 as the culprit. I couldn't do a clean revert (broke compile) so I am going to disable the rest of the tests.
Project Member

Comment 13 by bugdroid1@chromium.org, Dec 5 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/beda52671ea5c38b6e1f42ea4e40993b22e202a9

commit beda52671ea5c38b6e1f42ea4e40993b22e202a9
Author: Derek Cheng <imcheng@chromium.org>
Date: Tue Dec 05 23:37:03 2017

[Sheriff] Disable the rest of FeedbackTest on Asan.



Bug: 757243
Change-Id: I4e82b842301cc824f6113229bf99aab5add15b74
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/809939
Reviewed-by: Derek Cheng <imcheng@chromium.org>
Commit-Queue: Derek Cheng <imcheng@chromium.org>
Cr-Commit-Position: refs/heads/master@{#521896}
[modify] https://crrev.com/beda52671ea5c38b6e1f42ea4e40993b22e202a9/chrome/browser/extensions/api/feedback_private/feedback_browsertest.cc

Cc: chfremer@chromium.org
The CL that made the test fail more frequently just changes a function on the renderer process to run on the main thread instead of a worker thread. Thus, it probably exposed some existing timing issue on the browser side.

chfremer@: Could this be related to the new video-capture service?

This is a stack trace from the failed bot:

2974:46083:1205/093225.870512:FATAL:video_capture_manager.cc(112)] Check failed: controllers_.empty().
0   browser_tests                       0x000000010c1f7eac base::debug::StackTrace::StackTrace(unsigned long) + 28
1   browser_tests                       0x000000010c2543cb logging::LogMessage::~LogMessage() + 699
2   browser_tests                       0x00000001085a8303 content::VideoCaptureManager::~VideoCaptureManager() + 387
3   browser_tests                       0x00000001085a8b6e content::VideoCaptureManager::~VideoCaptureManager() + 14
4   browser_tests                       0x0000000108527ac7 content::MediaDevicesManager::~MediaDevicesManager() + 1271
5   browser_tests                       0x0000000108527c1e content::MediaDevicesManager::~MediaDevicesManager() + 14
6   browser_tests                       0x000000010855b5d1 content::MediaStreamManager::WillDestroyCurrentMessageLoop() + 977
7   browser_tests                       0x000000010c28dff2 base::MessageLoop::~MessageLoop() + 1410
8   browser_tests                       0x000000010c28ed0e base::MessageLoop::~MessageLoop() + 14
9   browser_tests                       0x000000010c3e5abc base::Thread::ThreadMain() + 1980
10  browser_tests                       0x000000010c3ccb7e base::(anonymous namespace)::ThreadFunc(void*) + 206
11  libsystem_pthread.dylib             0x00007fff86c7e899 _pthread_body + 138
12  libsystem_pthread.dylib             0x00007fff86c7e72a _pthread_struct_init + 0
13  libsystem_pthread.dylib             0x00007fff86c82fc9 thread_start + 13
It is possible that this was introduced with the new video capture service, but it is also possible that a timing issue existed regardless of whether the new service or the legacy code path is used.

In either case, the DCHECK in the destructor of VideoCaptureManager should never be hit, so this needs to be analyzed and fixed.

Was anyone already able to repro this on a local build?
Does this only happen on Mac ASAN builds or also on other platforms?

guidou@: When you say "made the test fail more frequently" does it mean the test already failed before with the same stack trace? If yes, do we know when this first started?

I created https://bugs.chromium.org/p/chromium/issues/detail?id=792543 to track the work for investigating and fixing the DCHECKed condition happening.



chfremer@: The part about (some of) the tests flaking before is based on several messages from September and October in this same bug, which disabled tests. See #3 and #10, for example.
The part about failing more frequently is based on message #12, which says that tests "started flaking heavily" since https://chromium-review.googlesource.com/809005

That CL only introduces slightly different timing on the renderer process.

Before the CL
*renderer starts processing getUserMedia() on the main thread
*posts a task to process constraints on the worker thread
*worker thread runs algorithm and posts result back to main thread
*reply arrives to the main thread and message to generate stream is sent to the browser.

After the CL
*renderer starts processing getUserMedia()
*constraints are processed on the main thread
*message to generate stream is sent to the browser

Sign in to add a comment