Chrome <-> Firefox interop test disabled because of Firefox sandboxing bug |
||||||||||
Issue descriptionhttps://build.chromium.org/p/chromium.webrtc/builders/Linux%20Tester/builds/30912 The test has been failing for the last week with the errors below: [21717:21717:0725/120057.995346:INFO:CONSOLE(38)] "5.100: WSS->C: {"msg":"{\"type\":\"candidate\",\"label\":0,\"id\":\"audio\",\"candidate\":\"candidate:0 1 UDP 2122252543 192.168.195.56 46677 typ host\"}","error":""}", source: http://localhost:9999/js/apprtc.debug.js (38) [21717:21717:0725/120057.996450:INFO:CONSOLE(38)] "5.101: Remote candidate added successfully.", source: http://localhost:9999/js/apprtc.debug.js (38) (/b/c/b/Linux_Tester/firefox-nightly/firefox/firefox:22135): GLib-GObject-CRITICAL **: g_object_ref: assertion 'object->ref_count > 0' failed (/b/c/b/Linux_Tester/firefox-nightly/firefox/firefox:22135): GLib-GObject-CRITICAL **: g_object_unref: assertion 'object->ref_count > 0' failed (/b/c/b/Linux_Tester/firefox-nightly/firefox/firefox:22135): GLib-GObject-CRITICAL **: g_object_ref: assertion 'object->ref_count > 0' failed (/b/c/b/Linux_Tester/firefox-nightly/firefox/firefox:22135): GLib-GObject-CRITICAL **: g_object_unref: assertion 'object->ref_count > 0' failed I will disable the test for now. jansson@ can you PTAL if it is an actual interop issue? ehmaldonado@ can you see if something is wrong in testing environment?
,
Aug 7 2017
I do think there is anything wrong with the test, the previous test is fine. I see the following error which I think is the main issue since all Browser tests stack traces are printed shortly after: Home directory not accessible: Permission denied ehmaldonado@ do you know if some infra change has been done?
,
Aug 7 2017
,
Aug 11 2017
This test appears to still be running (and failing) on the FYI waterfall: https://build.chromium.org/p/chromium.webrtc.fyi/builders/Linux%20Tester. Not sur e what's up with that.
,
Sep 1 2017
I tried running with a mid-july Firefox, and the test passes, so this is caused by a Firefox change. I'll see if I can dig out debug info and file a bug on them.
,
Sep 4 2017
All right, I've confirmed this a real bug. Firefox fails to talk to Chrome in the latest nightly but succeeds in the nightly a couple weeks back. That's a very serious bug. I'll file one right away on Firefox.
,
Sep 4 2017
Firefox bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1396542 Guido, maybe you and hbos@ should be aware that interop is about to get broken firefox<->chrome unless chrome/firefox/apprtc is updated.
,
Sep 4 2017
Task after fix, whether in Chrome, firefox or apprtc: re-enable test
,
Sep 4 2017
To clarify, the breakage on Aug 1 must either 1) be unrelated, or 2) it broke, got fixed and then regressed again in Firefox.
,
Sep 5 2017
FYI, Chrome won't load the appspot address. on appr.tc, it works for us. The assertions (if they're related to this failure, which isn't certain) would point to some sort of GDK issue, which would be very odd. Perhaps you can try disabling multi-process in settings and see if it works (since the regression range pointed to is linux-only sandbox changes). This also would be odd, though. What OS is this tested on? Has the OS been updated? Can you make a run with MOZ_LOG=MediaManager:4,jsep:5,mtransport:4,GetUserMedia:4,webrtc_trace:5 WEBRTC_TRACE=moz_log MOZ_LOG_FILE=some temp file, then attach the log to the bugzilla bug? Thanks
,
Sep 12 2017
It turns out this isn't an interop problem, but a problem with sandboxing for Firefox on Linux. https://bugzilla.mozilla.org/show_bug.cgi?id=1396542
,
Sep 25 2017
The bug is fixed now and the fix should be in nightly, let's try re-enabling...
,
Sep 26 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/6569f043431d49051467dd9cc52df361034e3771 commit 6569f043431d49051467dd9cc52df361034e3771 Author: Patrik Höglund <phoglund@chromium.org> Date: Tue Sep 26 10:50:02 2017 Re-enabling Firefox interop test. The fix is now in the Firefox nightly; verified the test passes locally. Depends on https://webrtc-review.googlesource.com/c/webrtc.DEPS/+/3401. R=ehmaldonado@chromium.org Bug: 751211 Change-Id: I191db39747711b09a0c36e367e8e08fac0c53ea5 Reviewed-on: https://chromium-review.googlesource.com/681701 Reviewed-by: Patrik Höglund <phoglund@chromium.org> Reviewed-by: Henrik Kjellander <kjellander@chromium.org> Commit-Queue: Patrik Höglund <phoglund@chromium.org> Cr-Commit-Position: refs/heads/master@{#504331} [modify] https://crrev.com/6569f043431d49051467dd9cc52df361034e3771/chrome/browser/media/webrtc/webrtc_apprtc_browsertest.cc
,
Sep 26 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/b34fe684c6a5cd0bc0efacafadf8e934dc392a0f commit b34fe684c6a5cd0bc0efacafadf8e934dc392a0f Author: Patrik Höglund <phoglund@chromium.org> Date: Tue Sep 26 13:25:00 2017 Revert "Re-enabling Firefox interop test." This reverts commit 6569f043431d49051467dd9cc52df361034e3771. Reason for revert: Still fails on the bot for some reason, but passes on my local machine. Original change's description: > Re-enabling Firefox interop test. > > The fix is now in the Firefox nightly; verified the test passes > locally. > > Depends on https://webrtc-review.googlesource.com/c/webrtc.DEPS/+/3401. > > R=ehmaldonado@chromium.org > > Bug: 751211 > Change-Id: I191db39747711b09a0c36e367e8e08fac0c53ea5 > Reviewed-on: https://chromium-review.googlesource.com/681701 > Reviewed-by: Patrik Höglund <phoglund@chromium.org> > Reviewed-by: Henrik Kjellander <kjellander@chromium.org> > Commit-Queue: Patrik Höglund <phoglund@chromium.org> > Cr-Commit-Position: refs/heads/master@{#504331} TBR=kjellander@chromium.org,phoglund@chromium.org,ehmaldonado@chromium.org Change-Id: I75ad86884d347f7853d6c616dbf284fbe0729c11 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: 751211 Reviewed-on: https://chromium-review.googlesource.com/684313 Reviewed-by: Patrik Höglund <phoglund@chromium.org> Commit-Queue: Patrik Höglund <phoglund@chromium.org> Cr-Commit-Position: refs/heads/master@{#504350} [modify] https://crrev.com/b34fe684c6a5cd0bc0efacafadf8e934dc392a0f/chrome/browser/media/webrtc/webrtc_apprtc_browsertest.cc
,
Oct 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/4b214c7d22d1f352a6a4dc585344549868216090 commit 4b214c7d22d1f352a6a4dc585344549868216090 Author: Patrik Höglund <phoglund@chromium.org> Date: Mon Oct 09 13:43:04 2017 Re-enabling Firefox interop test. The fix is now in the Firefox nightly; verified the test passes locally. I will land https://webrtc-review.googlesource.com/c/webrtc.DEPS/+/6820 in case the bot still has a problem with ff sandboxing, and then try to turn that off. If that fails, the test is at least running again and we can follow up more with mozilla why the bot fails to play audio but not my workstation. Bug: 751211 Change-Id: I34f7293a123424c9077f12559713597bba294c01 Reviewed-on: https://chromium-review.googlesource.com/702240 Reviewed-by: Henrik Kjellander <kjellander@chromium.org> Commit-Queue: Patrik Höglund <phoglund@chromium.org> Cr-Commit-Position: refs/heads/master@{#507367} [modify] https://crrev.com/4b214c7d22d1f352a6a4dc585344549868216090/chrome/browser/media/webrtc/webrtc_apprtc_browsertest.cc
,
Oct 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/0e06ba216dbbf85b4a8f634b566d912606d87d3d commit 0e06ba216dbbf85b4a8f634b566d912606d87d3d Author: Guido Urdaneta <guidou@chromium.org> Date: Mon Oct 09 16:44:59 2017 Revert "Re-enabling Firefox interop test." This reverts commit 4b214c7d22d1f352a6a4dc585344549868216090. Reason for revert: Breaks WebRTC Chromium Linux Tester bot See https://build.chromium.org/p/chromium.webrtc/builders/Linux%20Tester/builds/32384 Sample crash logs: BrowserTestBase received signal: Terminated. Backtrace: #0 0x00000460fa17 base::debug::StackTrace::StackTrace() #1 0x000004ab4b81 content::(anonymous namespace)::DumpStackTraceSignalHandler() #2 0x7f4401164d40 <unknown> #3 0x7f440121b12d __poll #4 0x7f44066e8fe4 <unknown> #5 0x7f44066e90ec g_main_context_iteration #6 0x00000462c603 base::MessagePumpGlib::Run() #7 0x00000464c285 base::RunLoop::Run() #8 0x000004ae51d1 content::MessageLoopRunner::Run() #9 0x000004ab8a28 content::DOMMessageQueue::WaitForMessage() #10 0x000004ab78e9 content::(anonymous namespace)::ExecuteScriptHelper() #11 0x000004ab751e content::ExecuteScriptAndExtractString() #12 0x000002375eb5 test::SleepInJavascript() #13 0x000002376300 test::PollingWaitUntil() #14 0x00000236242f WebRtcApprtcBrowserTest_MANUAL_FirefoxApprtcInteropTest_Test::RunTestOnMainThread() #15 0x000004ab48ef content::BrowserTestBase::ProxyRunTestOnMainThreadLoop() #16 0x000004702d7c ChromeBrowserMainParts::PreMainMessageLoopRunImpl() #17 0x000004701bba ChromeBrowserMainParts::PreMainMessageLoopRun() #18 0x0000036b9d9d content::BrowserMainLoop::PreMainMessageLoopRun() #19 0x0000039d72e7 content::StartupTaskRunner::RunAllTasksNow() #20 0x0000036b816d content::BrowserMainLoop::CreateStartupTasks() #21 0x0000036bc5d1 content::BrowserMainRunnerImpl::Initialize() #22 0x0000036b57c2 content::BrowserMain() #23 0x0000045f9f11 content::ContentMainRunnerImpl::Run() #24 0x000005f8797d service_manager::Main() #25 0x0000045f8a52 content::ContentMain() #26 0x000004ab45c5 content::BrowserTestBase::SetUp() #27 0x0000046b2fdb InProcessBrowserTest::SetUp() #28 0x00000310c34e testing::Test::Run() #29 0x00000310ce80 testing::TestInfo::Run() #30 0x00000310d367 testing::TestCase::Run() #31 0x000003113857 testing::internal::UnitTestImpl::RunAllTests() #32 0x0000031134e3 testing::UnitTest::Run() #33 0x0000046c5558 base::TestSuite::Run() #34 0x00000460515a ChromeTestSuiteRunner::RunTestSuite() #35 0x000004adf183 content::LaunchTests() #36 0x00000460551e LaunchChromeTests() #37 0x0000046050c7 main #38 0x7f440114fec5 __libc_start_main Original change's description: > Re-enabling Firefox interop test. > > The fix is now in the Firefox nightly; verified the test passes > locally. > > I will land https://webrtc-review.googlesource.com/c/webrtc.DEPS/+/6820 > in case the bot still has a problem with ff sandboxing, and then > try to turn that off. If that fails, the test is at least running > again and we can follow up more with mozilla why the bot fails to > play audio but not my workstation. > > Bug: 751211 > Change-Id: I34f7293a123424c9077f12559713597bba294c01 > Reviewed-on: https://chromium-review.googlesource.com/702240 > Reviewed-by: Henrik Kjellander <kjellander@chromium.org> > Commit-Queue: Patrik Höglund <phoglund@chromium.org> > Cr-Commit-Position: refs/heads/master@{#507367} TBR=kjellander@chromium.org,phoglund@chromium.org Change-Id: I1df253a052610b89fc27e8045005252165fb5be3 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: 751211 Reviewed-on: https://chromium-review.googlesource.com/707274 Reviewed-by: Guido Urdaneta <guidou@chromium.org> Commit-Queue: Guido Urdaneta <guidou@chromium.org> Cr-Commit-Position: refs/heads/master@{#507391} [modify] https://crrev.com/0e06ba216dbbf85b4a8f634b566d912606d87d3d/chrome/browser/media/webrtc/webrtc_apprtc_browsertest.cc
,
Oct 10 2017
I logged into the bot, and the test runs fine there (!?!?!) My only theory is that the added thread_restrictions things makes the test hang, and that I saw the IO error in my local checkout because I built debug? I'll try to make it work in release and land it without thread restrictions. I'll also revert the e10s ff patch since it turns out that's not the problem. Of course, the test should work in debug as well but that will be a later problem...
,
Oct 10 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/c31e3896b8fe7f6e0a73c70633b37f7ab06f0236 commit c31e3896b8fe7f6e0a73c70633b37f7ab06f0236 Author: Patrik Höglund <phoglund@chromium.org> Date: Tue Oct 10 13:05:16 2017 Re-enable ff interop test, without adding thread restrictions. I suspect the thread restrictions I added in my last patch made the test hang, so let's try just enabling it. The error prevented by the thread restrictions is only thrown in debug mode anyway. Bug: 751211 Change-Id: I8be06ce061e4111ea844b54ff52d8c9e79b43915 Reviewed-on: https://chromium-review.googlesource.com/708795 Reviewed-by: Henrik Kjellander <kjellander@chromium.org> Commit-Queue: Patrik Höglund <phoglund@chromium.org> Cr-Commit-Position: refs/heads/master@{#507651} [modify] https://crrev.com/c31e3896b8fe7f6e0a73c70633b37f7ab06f0236/chrome/browser/media/webrtc/webrtc_apprtc_browsertest.cc
,
Oct 10 2017
Ok, the test both passes and fails on the bot, I'm going nuts.
,
Oct 10 2017
It seems to be failing consistently on Linux tester: https://build.chromium.org/p/chromium.webrtc/builders/Linux%20Tester/builds/32405 https://build.chromium.org/p/chromium.webrtc/builders/Linux%20Tester/builds/32406
,
Oct 10 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/f4598a83d452309101f79cb72df0f170f97bb7b7 commit f4598a83d452309101f79cb72df0f170f97bb7b7 Author: Guido Urdaneta <guidou@chromium.org> Date: Tue Oct 10 17:16:39 2017 Revert "Re-enable ff interop test, without adding thread restrictions." This reverts commit c31e3896b8fe7f6e0a73c70633b37f7ab06f0236. Reason for revert: Breaks WebRTC Linux Tester bot. See https://build.chromium.org/p/chromium.webrtc/builders/Linux%20Tester/builds/32405 Sample error logs: [32472:32472:1010/094403.887663:INFO:CONSOLE(38)] "5.152: Remote candidate added successfully.", source: http://localhost:9999/js/apprtc.debug.js (38) BrowserTestBase received signal: Terminated. Backtrace: #0 0x0000046c1737 base::debug::StackTrace::StackTrace() #1 0x000004b58761 content::(anonymous namespace)::DumpStackTraceSignalHandler() #2 0x7f9988abdd40 <unknown> #3 0x7f9988b7412d __poll #4 0x7f998e041fe4 <unknown> #5 0x7f998e0420ec g_main_context_iteration #6 0x0000046de263 base::MessagePumpGlib::Run() #7 0x0000046fdee5 base::RunLoop::Run() #8 0x000004b88da1 content::MessageLoopRunner::Run() #9 0x000004b5c5f8 content::DOMMessageQueue::WaitForMessage() #10 0x000004b5b4c9 content::(anonymous namespace)::ExecuteScriptHelper() #11 0x000004b5b0fe content::ExecuteScriptAndExtractString() #12 0x00000237c1a5 test::SleepInJavascript() #13 0x00000237c5f0 test::PollingWaitUntil() #14 0x00000236871f WebRtcApprtcBrowserTest_MANUAL_FirefoxApprtcInteropTest_Test::RunTestOnMainThread() #15 0x000004b584cf content::BrowserTestBase::ProxyRunTestOnMainThreadLoop() #16 0x0000047ac9dc ChromeBrowserMainParts::PreMainMessageLoopRunImpl() #17 0x0000047ab81a ChromeBrowserMainParts::PreMainMessageLoopRun() #18 0x00000376838d content::BrowserMainLoop::PreMainMessageLoopRun() #19 0x000003a856f7 content::StartupTaskRunner::RunAllTasksNow() #20 0x00000376675d content::BrowserMainLoop::CreateStartupTasks() #21 0x00000376abc1 content::BrowserMainRunnerImpl::Initialize() #22 0x000003763db2 content::BrowserMain() #23 0x0000046abbb1 content::ContentMainRunnerImpl::Run() #24 0x000005f9d89d service_manager::Main() #25 0x0000046aa6f2 content::ContentMain() #26 0x000004b581a5 content::BrowserTestBase::SetUp() #27 0x00000475c6db InProcessBrowserTest::SetUp() #28 0x0000031143ee testing::Test::Run() #29 0x000003114f20 testing::TestInfo::Run() #30 0x000003115407 testing::TestCase::Run() #31 0x00000311b8f7 testing::internal::UnitTestImpl::RunAllTests() #32 0x00000311b583 testing::UnitTest::Run() #33 0x00000476ec58 base::TestSuite::Run() #34 0x0000046b6e7a ChromeTestSuiteRunner::RunTestSuite() #35 0x000004b82d53 content::LaunchTests() #36 0x0000046b723e LaunchChromeTests() #37 0x0000046b6de7 main #38 0x7f9988aa8ec5 __libc_start_main #39 0x0000020dc029 <unknown> 2017/10/10 09:49:21 Deregistered client 43795133 from room some_room [76/76] WebRtcApprtcBrowserTest.MANUAL_FirefoxApprtcInteropTest (TIMED OUT) Original change's description: > Re-enable ff interop test, without adding thread restrictions. > > I suspect the thread restrictions I added in my last patch made the > test hang, so let's try just enabling it. The error prevented by > the thread restrictions is only thrown in debug mode anyway. > > Bug: 751211 > Change-Id: I8be06ce061e4111ea844b54ff52d8c9e79b43915 > Reviewed-on: https://chromium-review.googlesource.com/708795 > Reviewed-by: Henrik Kjellander <kjellander@chromium.org> > Commit-Queue: Patrik Höglund <phoglund@chromium.org> > Cr-Commit-Position: refs/heads/master@{#507651} TBR=kjellander@chromium.org,phoglund@chromium.org Change-Id: Ib74f9efd3274384197b23c0a820260c406995e89 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: 751211 Reviewed-on: https://chromium-review.googlesource.com/709794 Reviewed-by: Guido Urdaneta <guidou@chromium.org> Commit-Queue: Guido Urdaneta <guidou@chromium.org> Cr-Commit-Position: refs/heads/master@{#507701} [modify] https://crrev.com/f4598a83d452309101f79cb72df0f170f97bb7b7/chrome/browser/media/webrtc/webrtc_apprtc_browsertest.cc
,
Oct 11 2017
Ok, I have exhausted all the ideas I had. I'm thinking about deleting this test now, to be honest. I think FF interoperability has been solid for a long time, and the complexity of this test makes it hard to extend and maintain. Also, it's running ancient libs from mozilla for downloading and running firefox. I'm in favor for deleting it. Guido, Henrik B, WDYT? Would you find it scary to lose FF interop testing? Perhaps we can look into using KITE for interop testing in Q1.
,
Jan 24 2018
,
Jan 24 2018
,
Feb 12 2018
Ok, I have decided to delete the test, CLs out...
,
Feb 12 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/3c3b58d6a87f039c0733619cc177d1ff8e7af1e1 commit 3c3b58d6a87f039c0733619cc177d1ff8e7af1e1 Author: Patrik Höglund <phoglund@chromium.org> Date: Mon Feb 12 09:59:11 2018 Remove Firefox - Chromium WebRTC interop test. This test has been broken for a couple months, we have no idea how to fix it, and its value is questionable compared to its complexity anyway. It finds ~1 bug per year, tops, and even when it does it's hard to troubleshoot. Also, the code to interact with Firefox has aged badly and is hard to extend. Bug: chromium:751211 Change-Id: I0b4f84d80a3f8c05fd40fe3d58b82ccabbb4e9d2 Reviewed-on: https://chromium-review.googlesource.com/913248 Reviewed-by: Guido Urdaneta <guidou@chromium.org> Commit-Queue: Patrik Höglund <phoglund@chromium.org> Cr-Commit-Position: refs/heads/master@{#536046} [modify] https://crrev.com/3c3b58d6a87f039c0733619cc177d1ff8e7af1e1/chrome/browser/media/webrtc/webrtc_apprtc_browsertest.cc
,
Feb 12 2018
|
||||||||||
►
Sign in to add a comment |
||||||||||
Comment 1 by bugdroid1@chromium.org
, Aug 1 2017