Assigning to phoglund@chromium.org@ who owns webrtc browser tests. Also ccing sadrul@ based on blame for GLAndSoftwareCompositing/CompositingRenderWidgetHostViewBrowserTestTabCapture.CopyFromCompositingSurface_Cropped_Scaled test.

I'll hand this over to ehmaldonado who is owner-in-training :) 

This issue is marked as Beta blocker, M53 is scheduled to be promoted to Beta next week (07/27) please resolve asap.

ehmaldonado@webrtc.org can't be an OWNER as they are not a project member. phoglund, can you make sure that this is fixed before M-53 is launched to stable?

I don't even know which bot you're referring to. I can't find any 10.12 bots in the chromium.mac or chromium.fyi waterfalls.

Also define 'fix', do you want the tests to stop failing or the error to be fixed? The above looks like it can be a real problem. Disabling the tests is easily done if that's what you want.

Is there any way to get a preview version of 10.12 so we can repro locally?

cc:ing in pthatcher and deadbeef: could be we have problems with ip handling in the upcoming mac os 10.12.

Aha, Edward now has a chromium account.

There are no 10.12 bots, as we can't put pre-release software on public waterfalls. This test was run manually on a 10.12  device. If you're in MTV, you can stop by my desk to borrow mine - otherwise, you'll need to find a non-corp device to install 10.12.

I would be nice if someone who knows more about WebRTC can look into the error and determine whether it should be a release blocker. If it's a real problem, then it needs to be fixed before M-53 goes stable.

I see. If all the other tests pass, we know at least that most of WebRTC is working. Can you get us the full output from the test as well? It should say where it is failing. WebRtcBrowserIPPermissionDeniedTest.GatherLocalCandidates can fail either if there loopback candidates in there or if we get no candidates at all, for instance.

I've attached more detail failure logs.

Deleted: content_browsertest_sierra_failures.txt 10.2 KB

content_browsertest_sierra_failures.txt 10.2 KB View Download

[ RUN      ] WebRtcBrowserIPPermissionDeniedTest.GatherLocalCandidates
objc[97082]: Class MockCrApp is implemented in both /Users/dev/projects/chromium/src/out/gn/libtest_runner.dylib (0x11a8695d8) and /Users/dev/projects/chromium/src/out/gn/Content Shell.app/Contents/Frameworks/Content Shell Framework.framework/Content Shell Framework (0x10661e340). One of the two will be used. Which one is undefined.
objc[97082]: Class CocoaTestHelperWindow is implemented in both /Users/dev/projects/chromium/src/out/gn/libtest_runner.dylib (0x11a869560) and /Users/dev/projects/chromium/src/out/gn/Content Shell.app/Contents/Frameworks/Content Shell Framework.framework/Content Shell Framework (0x10661e368). One of the two will be used. Which one is undefined.
objc[97083]: Class MockCrApp is implemented in both /Users/dev/projects/chromium/src/out/gn/libtest_runner.dylib (0x119ab05d8) and /Users/dev/projects/chromium/src/out/gn/Content Shell.app/Contents/Frameworks/Content Shell Framework.framework/Content Shell Framework (0x1064c4340). One of the two will be used. Which one is undefined.
objc[97083]: Class CocoaTestHelperWindow is implemented in both /Users/dev/projects/chromium/src/out/gn/libtest_runner.dylib (0x119ab0560) and /Users/dev/projects/chromium/src/out/gn/Content Shell.app/Contents/Frameworks/Content Shell Framework.framework/Content Shell Framework (0x1064c4368). One of the two will be used. Which one is undefined.
[97083:32007:0721/093915:217077851306886:ERROR:webrtcsession.cc(1241)] ConnectDataChannel called when data_channel_ is NULL.
[97083:32007:0721/093915:217077852974572:WARNING:mediasession.cc(353)] Duplicate id found. Reassigning from 101 to 127
[97083:31751:0721/093915:217077967385644:WARNING:p2ptransportchannel.cc(462)] Jingle:Port[0x7fc5c230de40:data:1:0:local:Net[any:0.0.0.0/0:Unknown]]: SetOption(5, 0) failed: 0
[97083:31751:0721/093915:217077967809260:WARNING:p2ptransportchannel.cc(462)] Jingle:Port[0x7fc5c0c54030:data:1:0:local:Net[any:::/0:Unknown]]: SetOption(5, 0) failed: 0
From javascript: Error: expected 'true', got 'false'.
    at failTest (http://127.0.0.1:59180/media/webrtc_test_utilities.js:40:15)
    at assertEquals (http://127.0.0.1:59180/media/webrtc_test_utilities.js:240:5)
    at http://127.0.0.1:59180/media/peerconnection-call.html:880:9
    at RTCPeerConnection.pc.onicecandidate (http://127.0.0.1:59180/media/peerconnection-call.html:836:9)
When executing 'callAndExpectNonLoopbackCandidates();'
../../content/test/webrtc_content_browsertest_base.cc:89: Failure

For reference, normal successful output is
[7164:7512:0725/005111:1167593:ERROR:dxva_video_decode_accelerator_win.cc(306)] EGL_EXT_device_query missing
[7164:7512:0725/005111:1167593:ERROR:mf_helpers.cc(12)] Error in dxva_video_decode_accelerator_win.cc on line 306
[8088:7724:0725/005111:1167625:ERROR:singleton_hwnd.cc(34)] Cannot create windows on non-UI thread!
[8088:8092:0725/005111:1167843:ERROR:webrtcsession.cc(1241)] ConnectDataChannel called when data_channel_ is NULL.
[8088:8092:0725/005111:1167843:WARNING:mediasession.cc(353)] Duplicate id found. Reassigning from 101 to 127
[8088:7996:0725/005111:1167953:WARNING:p2ptransportchannel.cc(462)] Jingle:Port[078B8AF8:data:1:0:local:Net[loopback_ipv4:127.0.0.x/32:Unknown]]: SetOption(5, 0) failed: 0
[8088:7996:0725/005111:1167953:WARNING:p2ptransportchannel.cc(462)] Jingle:Port[078B96E0:data:1:0:local:Net[{33C023D1-11FD-405B-B860-11CDE05888D0}:192.168.195.x/24:Ethernet]]: SetOption(5, 0) failed: 0
[7908:2232:0725/005111:1168000:INFO:CONSOLE(29)] "Test Success", source: http://127.0.0.1:54198/media/webrtc_test_utilities.js (29)
[7164:2916:0725/005111:1168000:ERROR:node_controller.cc(1099)] Could not be introduced to peer 8A64E3BDD582A562.C7211225EAECB1F3

pthatcher, please help with an owner. Someone the networking team needs to get a non-corp device and install MacOS Sierra on it and try the above tests.

erikchen, feel free to disable the above WebRTC tests if you want to get any bots green.

Relevant changes:


Working:

[...p2ptransportchannel.cc(462)] Jingle:Port[...Net[loopback_ipv4:127.0.0.x/32:Unknown]]: SetOption(5, 0) failed: 0


Not working:

[...p2ptransportchannel.cc(462)] Jingle:Port[...Net[any:0.0.0.0/0:Unknown]]: SetOption(5, 0) failed: 0
[...p2ptransportchannel.cc(462)] Jingle:Port[...Net[any:::/0:Unknown]]: SetOption(5, 0) failed: 0


Option 5 is DSCP, by the way.  But SetOption() failing isn't the bug.  It's just convenient that it's logging so we see a different in the ports.


"loopback_ipv4" is added as long as base::CommandLine::ForCurrentProcess()->HasSwitch("allow-loopback-in-peer-connection").  See https://cs.chromium.org/chromium/src/content/renderer/p2p/ipc_network_manager.cc?cl=GROK&l=137


"any" is added as long as multiple routes is disabled or camera access has not been granted.  See https://cs.chromium.org/chromium/src/third_party/webrtc/p2p/client/basicportallocator.cc?q=GetAnyAddressNetworks&sq=package:chromium&dr=C&l=519.


So it's possible this is some kind of configuration difference.  There is a slight chance it's a race condition.  

Taylor, can you take a look at this?  Please give it the right P1 treatment.

Over to Vlad :)

These tests (and a few others) are also failing for me on a laptop running the current stable OS X version (10.11, El Capitan). These tests pass on the bots (https://build.chromium.org/p/chromium.mac/builders/Mac10.11%20Tests/builds/2244/steps/content_browsertests/logs/stdio) - so there might indeed be some configuration difference.

This is a bug in the test function that tests whether an IP address is loopback. It's looking for "::1" as a substring - and it's possible to have a non-loopback IPv6 address with that substring, e.g. "2345::1234:1234".

The fix is in code review; I'd suggest reducing the priority of the bug as it's been there for a long time and is merely a test issue. 

We should open a separate bug for the compositing test, as it's unrelated to WebRTC.

Oh, so the interesting variable isn't macos versions but rather laptop vs bot/vm? Ok, reducing priority.

Do you think your fix in #17 will solve the problem? I'm thinking maybe these tests make too many assumptions about the network environment on the host machine? In that case we should remove them, or maybe skip if we detect an unexpected network environment.

The fix in #17 (https://codereview.chromium.org/2185533007/) should make the WebRTC tests work in this particular case - when the IPv6 address has a "::1" substring. The tests still require at least one non-loopback IP address - so they will still fail if a machine is offline.

Right. For desktop we assume that the machine is connected when we run tests, so that's fine (we expect phones to be in flight mode though and we have a special flag to deal with that).

Ok, ship it and then close this bug! Spawning a new bug for compositing test SGTM.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/60b186ed1accde051b06bf74b966a88ec9b54fba

commit 60b186ed1accde051b06bf74b966a88ec9b54fba
Author: skvlad <skvlad@chromium.org>
Date: Fri Jul 29 19:32:38 2016

Fix spurious WebRTC test failure on certain IPv6 networks

A helper function that determines whether an IP address is loopback was
checking for a "::1" substring in the IP address. Non-loopback IPv6
addresses can have that substring - e.g. "1234::1234:1234". If the test
runs on a machine with one of these addresses, it would fail
erroneously.

The fix simply adds spaces on both sides of the pattern, so that it only
matches complete IP addresses (always surrounded by spaces in an ICE
candidate line).

BUG= 629705 

Review-Url: https://codereview.chromium.org/2185533007
Cr-Commit-Position: refs/heads/master@{#408726}

[modify] https://crrev.com/60b186ed1accde051b06bf74b966a88ec9b54fba/content/test/data/media/peerconnection-call.html

I've opened https://bugs.chromium.org/p/chromium/issues/detail?id=632799 for the compositing test failure.