No obvious errors, I'll try to bisect on the ff version since it seems most likely to come from there.

Doesn't repro on my workstation, that's quite annoying.

Aha, we're getting "No TURN server; unlikely that media will traverse networks. If this persists please" on the bot. I'm not getting it on my workstation though. What's going on?

Or wait, no, I am seeing it flash by briefly on my workstation but it works anyway. Hmmm.

Firefox js console on the bot. The TURN errors below could be the explanation, I'll see if I get them on my workstation.

This appears to be Firefoxapprtc.debug.js:38:3
0.595: Initializing; server= undefined.apprtc.debug.js:38:3
0.595: Initializing; room=some_room.apprtc.debug.js:38:3
navigator.mozGetUserMedia has been replaced by navigator.mediaDevices.getUserMediaapprtc.debug.js:166:13
0.615: Opening signaling channel.apprtc.debug.js:38:3
0.642: Signaling channel opened.apprtc.debug.js:38:3
0.688: Joined the room.apprtc.debug.js:38:3
0.689: Registering signaling channel.apprtc.debug.js:38:3
0.690: Signaling channel registered.apprtc.debug.js:38:3
0.718: WSS->C: {"msg":"","error":"Max room capacity reached"}apprtc.debug.js:38:3
0.719: Signaling server error message: Max room capacity reachedapprtc.debug.js:38:3
0.720: Channel closed with code:1000 reason:apprtc.debug.js:38:3
0.743: Got access to local media with mediaConstraints:
  '{"fake":true,"audio":true,"video":true}'apprtc.debug.js:38:3
0.743: User has granted access to local media.apprtc.debug.js:38:3
0.743: Attaching local stream.apprtc.debug.js:38:3
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://computeengineondemand.appspot.com/turn?username=101585814&key=4080218913. (Reason: CORS header \u2018Access-Control-Allow-Origin\u2019 missing).(unknown)
1.013: TURN server request error: Status=0, response=apprtc.debug.js:38:3
1.014: Starting signaling.apprtc.debug.js:38:3
1.014: Creating RTCPeerConnnection with:
  config: '{"bundlePolicy":"max-bundle","iceServers":[],"rtcpMuxPolicy":"require"}';
  constraints: '{"optional":[]}'.apprtc.debug.js:38:3
WebRTC interfaces with the \u201cmoz\u201d prefix (mozRTCPeerConnection, mozRTCSessionDescription, mozRTCIceCandidate) have been deprecated.apprtc.debug.js:106:13
onaddstream is deprecated! Use peerConnection.ontrack instead.apprtc.debug.js:1394
1.025: Created PeerConnectionClientapprtc.debug.js:38:3
1.025: Adding local stream.apprtc.debug.js:38:3
1.027: No preference on audio send codec.apprtc.debug.js:38:3
1.027: No preference on video send codec.apprtc.debug.js:38:3
1.028: Sending answer to peer.apprtc.debug.js:38:3
1.104: Remote stream added.apprtc.debug.js:38:3
1.109: Signaling state changed to: have-remote-offerapprtc.debug.js:38:3
1.111: Set remote session description success.apprtc.debug.js:38:3
RTCPeerConnection.getLocalStreams/getRemoteStreams are deprecated. Use RTCPeerConnection.getSenders/getReceivers instead.apprtc.debug.js:1517:22
1.115: Waiting for remote video.apprtc.debug.js:38:3
1.180: No preference on audio receive codec.apprtc.debug.js:38:3
1.180: Prefer video receive codec: VP9apprtc.debug.js:38:3
1.181: C->WSS: {"sdp":"v=0\r\no=mozilla...THIS_IS_SDPARTA-50.0a1 1500599930454347615 0 IN IP4 0.0.0.0\r\ns=-\r\nt=0 0\r\na=fingerprint:sha-256 98:7E:0E:C5:18:66:1E:6A:26:99:2B:EC:2A:39:9B:41:D4:37:2E:C9:78:B2:46:CF:85:1D:AC:41:37:01:AE:6C\r\na=group:BUNDLE audio video\r\na=ice-options:trickle\r\na=msid-semantic:WMS *\r\nm=audio 9 UDP/TLS/RTP/SAVPF 111\r\nc=IN IP4 0.0.0.0\r\na=sendrecv\r\na=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level\r\na=fmtp:111 maxplaybackrate=48000;stereo=1;useinbandfec=1\r\na=ice-pwd:12299db2ccd9f571e8df2a322013890d\r\na=ice-ufrag:d6cf615f\r\na=mid:audio\r\na=msid:{a11ddbfb-5300-455d-b0cd-89c0607a8fa5} {d29a8ae3-9455-430e-8bbc-2b1a61ee0eb0}\r\na=rtcp-mux\r\na=rtpmap:111 opus/48000/2\r\na=setup:active\r\na=ssrc:923468631 cname:{956716eb-82f5-4d03-8e48-ccd7627f4364}\r\nm=video 9 UDP/TLS/RTP/SAVPF 100\r\nc=IN IP4 0.0.0.0\r\na=sendrecv\r\na=fmtp:100 max-fs=12288;max-fr=60\r\na=ice-pwd:12299db2ccd9f571e8df2a322013890d\r\na=ice-ufrag:d6cf615f\r\na=mid:video\r\na=msid:{a11ddbfb-5300-455d-b0cd-89c0607a8fa5} {b20dd776-01e4-4a21-bc66-cd1f231e3267}\r\na=rtcp-fb:100 nack\r\na=rtcp-fb:100 nack pli\r\na=rtcp-fb:100 ccm fir\r\na=rtcp-fb:100 goog-remb\r\na=rtcp-mux\r\na=rtpmap:100 VP8/90000\r\na=setup:active\r\na=ssrc:2001868507 cname:{956716eb-82f5-4d03-8e48-ccd7627f4364}\r\n","type":"answer"}apprtc.debug.js:38:3
1.182: Remote candidate added successfully.apprtc.debug.js:38:3
1.742: Signaling state changed to: stableapprtc.debug.js:38:3
1.743: Set session description success.apprtc.debug.js:38:3
1.778: ICE connection state changed to: checkingapprtc.debug.js:38:3
1.780: C->WSS: {"type":"candidate","label":0,"id":"audio","candidate":"candidate:0 1 UDP 2122252543 192.168.195.11 53636 typ host"}apprtc.debug.js:38:3
1.781: End of candidates.apprtc.debug.js:38:3

This appears to be Firefoxapprtc.debug.js:38:3
0.609: Initializing; server= undefined.apprtc.debug.js:38:3
0.610: Initializing; room=some_room.apprtc.debug.js:38:3
navigator.mozGetUserMedia has been replaced by navigator.mediaDevices.getUserMediaapprtc.debug.js:166:13
0.623: Opening signaling channel.apprtc.debug.js:38:3
0.635: Signaling channel opened.apprtc.debug.js:38:3
0.635: Got access to local media with mediaConstraints:
  '{"video":true,"fake":true,"audio":true}'apprtc.debug.js:38:3
0.636: User has granted access to local media.apprtc.debug.js:38:3
0.637: Attaching local stream.apprtc.debug.js:38:3
0.655: Joined the room.apprtc.debug.js:38:3
0.656: Registering signaling channel.apprtc.debug.js:38:3
0.657: Signaling channel registered.apprtc.debug.js:38:3
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://computeengineondemand.appspot.com/turn?username=569698775&key=4080218913. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing).(unknown)
1.149: TURN server request error: Status=0, response=apprtc.debug.js:38:3
1.149: Starting signaling.apprtc.debug.js:38:3
1.150: Creating RTCPeerConnnection with:
  config: '{"iceServers":[],"rtcpMuxPolicy":"require","bundlePolicy":"max-bundle"}';
  constraints: '{"optional":[]}'.apprtc.debug.js:38:3
WebRTC interfaces with the “moz” prefix (mozRTCPeerConnection, mozRTCSessionDescription, mozRTCIceCandidate) have been deprecated.apprtc.debug.js:106:13
onaddstream is deprecated! Use peerConnection.ontrack instead.apprtc.debug.js:1394
1.166: Created PeerConnectionClientapprtc.debug.js:38:3
1.167: Adding local stream.apprtc.debug.js:38:3
1.169: No preference on audio send codec.apprtc.debug.js:38:3
1.169: No preference on video send codec.apprtc.debug.js:38:3
1.171: Sending answer to peer.apprtc.debug.js:38:3
1.202: Remote stream added.apprtc.debug.js:38:3
1.203: Signaling state changed to: have-remote-offerapprtc.debug.js:38:3
RTCPeerConnection.getLocalStreams/getRemoteStreams are deprecated. Use RTCPeerConnection.getSenders/getReceivers instead.apprtc.debug.js:1517:22
1.206: Set remote session description success.apprtc.debug.js:38:3
1.206: Waiting for remote video.apprtc.debug.js:38:3
1.217: No preference on audio receive codec.apprtc.debug.js:38:3
1.217: Prefer video receive codec: VP9apprtc.debug.js:38:3
1.218: C->WSS: {"sdp":"v=0\r\no=mozilla...THIS_IS_SDPARTA-50.0a1 8758662667891458381 0 IN IP4 0.0.0.0\r\ns=-\r\nt=0 0\r\na=fingerprint:sha-256 C3:5D:FF:E1:6F:E3:04:43:15:2F:ED:57:9D:92:DE:7B:E0:F1:95:79:31:F0:60:57:20:56:CD:F2:DC:D0:E0:9C\r\na=group:BUNDLE audio video\r\na=ice-options:trickle\r\na=msid-semantic:WMS *\r\nm=audio 9 UDP/TLS/RTP/SAVPF 111\r\nc=IN IP4 0.0.0.0\r\na=sendrecv\r\na=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level\r\na=fmtp:111 maxplaybackrate=48000;stereo=1;useinbandfec=1\r\na=ice-pwd:13e12e01e0ecccf304604954bcb20f6b\r\na=ice-ufrag:04f67b3a\r\na=mid:audio\r\na=msid:{0508bd21-5348-4ec9-8a3b-03bbbb03aaef} {2831f766-ede9-4e2d-9b26-4264cbe15a53}\r\na=rtcp-mux\r\na=rtpmap:111 opus/48000/2\r\na=setup:active\r\na=ssrc:4083276173 cname:{0b81f953-3fac-4746-af10-d8ddc14f6ad8}\r\nm=video 9 UDP/TLS/RTP/SAVPF 100\r\nc=IN IP4 0.0.0.0\r\na=sendrecv\r\na=fmtp:100 max-fs=12288;max-fr=60\r\na=ice-pwd:13e12e01e0ecccf304604954bcb20f6b\r\na=ice-ufrag:04f67b3a\r\na=mid:video\r\na=msid:{0508bd21-5348-4ec9-8a3b-03bbbb03aaef} {4feee16d-91b5-4ebd-9bb5-a32a92f5617f}\r\na=rtcp-fb:100 nack\r\na=rtcp-fb:100 nack pli\r\na=rtcp-fb:100 ccm fir\r\na=rtcp-fb:100 goog-remb\r\na=rtcp-mux\r\na=rtpmap:100 VP8/90000\r\na=setup:active\r\na=ssrc:1692932553 cname:{0b81f953-3fac-4746-af10-d8ddc14f6ad8}\r\n","type":"answer"}apprtc.debug.js:38:3
1.219: Remote candidate added successfully.apprtc.debug.js:38:3
1.248: Signaling state changed to: stableapprtc.debug.js:38:3
1.249: Set session description success.apprtc.debug.js:38:3
1.332: C->WSS: {"type":"candidate","label":0,"id":"audio","candidate":"candidate:0 1 UDP 2121728255 172.16.32.65 47874 typ host"}apprtc.debug.js:38:3
1.333: ICE connection state changed to: checkingapprtc.debug.js:38:3
1.349: ICE connection state changed to: connectedapprtc.debug.js:38:3
1.350: C->WSS: {"type":"candidate","label":0,"id":"audio","candidate":"candidate:1 1 UDP 2122187007 2620:0:1043:1:248a:225d:60da:48eb 36711 typ host"}apprtc.debug.js:38:3
1.351: C->WSS: {"type":"candidate","label":0,"id":"audio","candidate":"candidate:2 1 UDP 2121990399 2620:0:1043:1:4b4:814a:e8b:526b 44910 typ host"}apprtc.debug.js:38:3
1.352: C->WSS: {"type":"candidate","label":0,"id":"audio","candidate":"candidate:3 1 UDP 2121793791 2620:0:1043:1:a807:2fb8:8c31:104f 38611 typ host"}apprtc.debug.js:38:3
1.352: C->WSS: {"type":"candidate","label":0,"id":"audio","candidate":"candidate:4 1 UDP 2122252543 2620:0:1043:1:2130:af5d:ab42:3dd8 50881 typ host"}apprtc.debug.js:38:3
1.353: C->WSS: {"type":"candidate","label":0,"id":"audio","candidate":"candidate:5 1 UDP 2122055935 2620:0:1043:1:48c6:f039:2301:e90d 60238 typ host"}apprtc.debug.js:38:3
1.353: C->WSS: {"type":"candidate","label":0,"id":"audio","candidate":"candidate:6 1 UDP 2121924863 2620:0:1043:1:590a:8716:88e2:a3c4 34394 typ host"}apprtc.debug.js:38:3
1.354: C->WSS: {"type":"candidate","label":0,"id":"audio","candidate":"candidate:7 1 UDP 2122121471 2620:0:1043:1:3c65:a808:8526:9536 35795 typ host"}apprtc.debug.js:38:3
1.355: C->WSS: {"type":"candidate","label":0,"id":"audio","candidate":"candidate:8 1 UDP 2121859327 2620:0:1043:1:6e3b:e5ff:fe1a:ae86 39010 typ host"}apprtc.debug.js:38:3
1.356: End of candidates.apprtc.debug.js:38:3
1.467: Remote video started; currentTime: 0.22929705215419502apprtc.debug.js:38:3
1.468: Call setup time: 318ms.apprtc.debug.js:38:3
1.468: reattachMediaStream: apprtc.debug.js:38:3

The CORS and TURN errors are red herrings, and the main differences are in the candidate list. Why are so few candidates getting generated on the bot?

Theory: ipv6 has gotten disabled on the bot, since all ipv6 candidates are conspicuously missing from the bot's candidate list, and WebRTC fails to get a call up on the sole ipv4 candidate.

pthatcher, deadbeef, can you help me understand what's going on here?

Hm, no the ipv6 theory is a red herring too, we only got one ipv4 candidate on the bot before the test started failing:

WSS->C: {"msg":"{\"type\":\"candidate\",\"label\":0,\"id\":\"audio\",\"candidate\":\"candidate:0 1 UDP 2122252543 192.168.195.56 57913 typ host\"}","error":""}", source: http://localhost:9999/js/apprtc.debug.js (38)

Source: https://build.chromium.org/p/chromium.webrtc/builders/Linux%20Tester/builds/18914

Meh. I'll disable the test over the weekend and take a new look on Monday. What I can see on the bot is that I get the TURN warning overlay and no video.

Re CORS/TURN I assume it's an old build of AppRTC since we do not use CEOD since a while back (only on iOS).

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/02ae49883285d4d9ec6032ca0d4db5420b6e80d6

commit 02ae49883285d4d9ec6032ca0d4db5420b6e80d6
Author: phoglund <phoglund@chromium.org>
Date: Fri Jul 08 16:10:11 2016

Temporarily disabling WebRTC FF interop test.

BUG= 626556 
TBR=pthatcher@chromium.org

Review-Url: https://codereview.chromium.org/2131983002
Cr-Commit-Position: refs/heads/master@{#404404}

[modify] https://crrev.com/02ae49883285d4d9ec6032ca0d4db5420b6e80d6/chrome/browser/media/webrtc_apprtc_browsertest.cc

Re #10: Yeah, it is old. I can try updating the AppRTC version but it's a long shot it will fix this problem.

Re #12, yes indeed, was merely pointing out the fact that the AppRTC version was a bit old ;)

Patrik: Where did you get the Firefox JS console output from comments #5 and #6? I'd like to take a look myself.

Also, did you notice "Max room capacity reached"? Could that be the issue? The signaling server may just be dropping the answer.

Re #15: In Firefox, do ctrl+shift+k to get to the web console (or menu -> developer -> web console).

Re #16: I don't think so, looks like the call gets up, it's just that no video is playing.

Re #17: In that case, how do you access the running instance of Firefox on the Linux test bot? Also, the crucial difference I see between the working/nonworking logs (on the Chrome side) is that Firefox's ICE candidate is not signaled in the nonworking case.

Re #18: I use my magical admin powers and remote desktop into the bot :) Sorry, I thought you meant running locally in #15. When I think of it, it would have been really nice for the test to capture this output from firefox as well.

Aha, so you mean
1.780: C->WSS: {"type":"candidate","label":0,"id":"audio","candidate":"candidate:0 1 UDP 2122252543 192.168.195.11 53636 typ host"}apprtc.debug.js:38:3

is the Chrome candidate in the Firefox log in #5, but there should be another one coming from Firefox?...

Yes, that's right. There's a "C->WSS" with the candidate on the Firefox side, but no "WSS->C" on the Chrome side (at least, in https://build.chromium.org/p/chromium.webrtc/builders/Linux%20Tester/builds/18915/steps/browser_tests/logs/stdio). Chrome receives the answer but no ICE candidates. Hence it never actually starts ICE.

Going on vacation so someone else will have to look at this. The test is still offline which isn't great.

I tried setting up a regular AppRTC call on Linux using the latest Firefox nightly downloaded from https://nightly.mozilla.org/ and a Chrome 54.0.2810.2 build. The call is established fine so I'm going to try re-enabling the test now.

FTR I launched Chrome using
 ./chrome --user-data-dir=/tmp/chrome-testing --enable-logging --v=1 --use-fake-device-for-media-stream

I also tried using a local apprtc+collider instance from a Chromium checkout:

python ../google_appengine/dev_appserver.py out/apprtc/out/app_engine/ --port=9999 --admin_port=9998 --skip_sdk_update_check --clear_datastore=yes

out/go-workspace/bin/collidermain -tls=false -port=8089 -room-server=http://localhost:9999

Then connecting both browsers to http://localhost:9999/r/some_room?wshpp=localhost:8089&wstls=false&firefox_fake_device=1

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7b82ba149c5512aff538cb06a2b2047da36eca38

commit 7b82ba149c5512aff538cb06a2b2047da36eca38
Author: kjellander <kjellander@chromium.org>
Date: Tue Aug 02 13:11:59 2016

Revert of Temporarily disabling WebRTC FF interop test. (patchset #1 id:1 of https://codereview.chromium.org/2131983002/ )

Reason for revert:
A manual test with the latest Firefox nightly is working, so I'm re-enabling this. See  crbug.com/626556  for more info.

Original issue's description:
> Temporarily disabling WebRTC FF interop test.
>
> BUG= 626556 
> TBR=pthatcher@chromium.org
>
> Committed: https://crrev.com/02ae49883285d4d9ec6032ca0d4db5420b6e80d6
> Cr-Commit-Position: refs/heads/master@{#404404}

TBR=pthatcher@chromium.org,phoglund@chromium.org
NOTRY=True
BUG= 626556 

Review-Url: https://codereview.chromium.org/2207543002
Cr-Commit-Position: refs/heads/master@{#409179}

[modify] https://crrev.com/7b82ba149c5512aff538cb06a2b2047da36eca38/chrome/browser/media/webrtc_apprtc_browsertest.cc

https://build.chromium.org/p/chromium.webrtc/builders/Linux%20Tester/builds/19928/steps/browser_tests/logs/stdio shows the test is now passing, yay!

Reopening due to recent failure.

 Issue 640628  has been merged into this issue.

Not reproing on my workstation, by the way.

It is not reproing on my workstation either.

Assigning to the current WebRTC-in-Chrome sheriff.

This still works fine when running locally.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c71d7e55a9a5915f6e3417e6214138e87cd28789

commit c71d7e55a9a5915f6e3417e6214138e87cd28789
Author: perkj <perkj@chromium.org>
Date: Fri Sep 02 12:07:23 2016

Re-enable WebRTCBrowserTests.MANUAL_FirefoxApprtcInteropTest

This test pass locally and we want to run it on the webrtc build bots.

This reverts commit f472eb573a7e50134d78aac9117f37f26e205dce.

BUG= 626556 
NOTRY=true
TBR=phoglund@chromium.org

Review-Url: https://codereview.chromium.org/2302763003
Cr-Commit-Position: refs/heads/master@{#416243}

[modify] https://crrev.com/c71d7e55a9a5915f6e3417e6214138e87cd28789/chrome/browser/media/webrtc_apprtc_browsertest.cc

Re #24, #25: yeah, like I found earlier in this bug errors sometimes don't repro on local workstations. It really should, which is very unfortunate. I'd love to find out what the root cause is; according to #20 the bot sometimes fail to allocate candidates for a loopback call on the bot, probably because of the network environment on the bot. Maybe the problem is a race in AppRTC or a problem when we bring AppRTC up as a local instance.

Let's hope this was due to some reconfiguration of the network in the bot lab. The test is back up now so let's close this one for now. Thanks for everyone's work on this.

Another thought: if this was really an AppRTC signaling or persistence problem, you'd think it would repro locally. When the test failed on the bot, it failed a 100% of the time right? That's what I think the bot environment had an effect here. Or, like I wrote in #8, maybe workstations don't repro it because they have ipv6 candidates to fall back on in case the ipv4 candidate doesn't work?