New issue
Advanced search Search tips

Issue 793808 link

Starred by 2 users
Status: Assigned
Owner:
hbos@chromium.org
Cc:
guidou@chromium.org
hbos@chromium.org
hta@chromium.org
mlamouri@chromium.org
foolip@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

Video tag sourced from remote WebRTC stream does not update when receiving new frames in content_shell

Project Member Reported by hbos@chromium.org, Dec 11 2017 
I am able to do the following:

sendCanvas = document.getElementById(...) for a <canvas> tag.
pc1.addStream(sendCanvas.captureStream());
Connect pc1 and pc2.
receiveStream = pc2.getRemoteStreams()[0];
receiveVideo = document.getElementById(...) for a <video> tag.
receiveVideo.srcObject = receiveStream;

Subsequently, if I draw to sendCanvas with its getContext('2d').fillRect(...) a new frame is produced, sent and received, showing up in the receiveVideo tag within an event execution cycle.

To verify video is received without manual intervention I draw the video onto a canvas (getContext('2d').drawImage) and read its pixel data (getContext('2d').getImageData), programmatically getting the color of the received image.

In chrome or browser_tests I am able to draw red and verify red is received.

But when I run the same code in content_shell web platform tests the video tag never appears to be updated, I never get a different color.
I am able to verify with RTCPeerConnection.getStats that frames are received, but the color of the receiving side's canvas is forever its original color.

Comment 1 by hbos@chromium.org, Dec 11 2017

Description: Show this description

Comment 2 by hbos@chromium.org, Dec 11 2017

Owner: hbos@chromium.org
Making myself owner until I have provided more information, like a jsfiddle or test code.

Comment 3 by hbos@chromium.org, Dec 11 2017 

Correction - in web platform tests, the video never receives any frames, the whiteCanvas never gets a different color than white.

Here's a jsfiddle that works:
https://jsfiddle.net/c1zxryny/

Output:
  Waiting for pollNextVideoColor to receive a new color...
  pollNextVideoColor initial color: white
  Received red color!

Here's the equivalent code to the jsfiddle but as a web platform test:

<!doctype html>
<meta charset=utf-8>
<title>RTCPeerConnection.prototype.setRemoteDescription - add/remove remote tracks</title>
<script src="/resources/testharness.js"></script>
<script src="/resources/testharnessreport.js"></script>
<script src="RTCPeerConnection-helper.js"></script>
<body>

  <table border="0">
    <tr>
      <td><video id="remote-view" autoplay style="display:none"></video></td>
      <td><canvas id="redCanvas" width="10" height="10" style="background-color:#FF0000"/></td>
      <td><canvas id="whiteCanvas" width="10" height="10" style="background-color:#FFFFFF"/></td>
    </tr>
  </table>

</body>
<script>
  'use strict';

  async_test(t => {
    return runTest()
    .then(result => {
      assert_true(result);
      t.done();
    });
  }, 'replaceTrack() sends new track');

  async function runTest() {
    const redCanvas = document.getElementById('redCanvas');
    const redCanvasStream = redCanvas.captureStream();
    const remoteVideo = document.getElementById('remote-view');
    const remoteVideoCanvas = document.getElementById('whiteCanvas');

    const caller = new RTCPeerConnection();
    const callee = new RTCPeerConnection();

    // Connect and send "redCanvas" to callee.
    const sender = caller.addStream(redCanvasStream);
    const connectPromise = connect(caller, callee);
    const streamEvent = await eventAsAsyncFunction(callee, 'onaddstream');
    remoteVideo.srcObject = streamEvent.stream;
    await connectPromise;

    // Ensure a red frame is sent by redrawing it the canvas.
    fillCanvas(redCanvas, 'red');
    console.log('Waiting for pollNextVideoColor to receive a new color...');
    let receivedColor = await pollNextVideoColor(remoteVideo,
                                                 remoteVideoCanvas);
    if (receivedColor != 'red') {
      console.log('Expected red, but received: ' + receivedColor);
      return false;
    }
    console.log('Received red color!');

    return true;
  }

  /** @private */
  async function connect(caller, callee) {
    caller.onicecandidate = (e) => {
      if (e.candidate)
        callee.addIceCandidate(new RTCIceCandidate(e.candidate));
    }
    callee.onicecandidate = (e) => {
      if (e.candidate)
        caller.addIceCandidate(new RTCIceCandidate(e.candidate));
    }
    let offer = await caller.createOffer();
    await caller.setLocalDescription(offer);
    await callee.setRemoteDescription(offer);
    let answer = await callee.createAnswer();
    await callee.setLocalDescription(answer);
    return caller.setRemoteDescription(answer);
  }

  /**
   * Makes the next |object[eventname]| event resolve the returned promise with
   * the event argument and resets the event handler to null.
   */
  async function eventAsAsyncFunction(object, eventname) {
    const resolver = new Resolver();
    object[eventname] = e => {
      object[eventname] = null;
      resolver.resolve(e);
    }
    return resolver.promise;
  }

  /**
   * Updates the canvas, filling it with |color|, e.g. 'red', 'lime' or 'blue'.
   * @private
   */
  function fillCanvas(canvas, color) {
    const canvasContext = canvas.getContext('2d');
    canvasContext.fillStyle = color;
    canvasContext.fillRect(0, 0, canvas.width, canvas.height);
  }

  /**
   * Gets the dominant color of the center of the canvas, meaning the color that
   * is closest to that pixel's color amongst: 'black', 'white', 'red', 'lime'
   * and 'blue'.
   * @private
   */
  function getDominantCanvasColor(canvas) {
    const colorData = canvas.getContext('2d').getImageData(
        Math.floor(canvas.width / 2), Math.floor(canvas.height / 2), 1, 1).data;

    const dominantColors = [
      { name: 'black', colorData: [0, 0, 0] },
      { name: 'white', colorData: [255, 255, 255] },
      { name: 'red', colorData: [255, 0, 0] },
      { name: 'lime', colorData: [0, 255, 0] },
      { name: 'blue', colorData: [0, 0, 255] },
    ];
    function getColorDistanceSquared(colorData1, colorData2) {
      const colorDiff = [ colorData2[0] - colorData1[0],
                          colorData2[1] - colorData1[1],
                          colorData2[2] - colorData1[2] ];
      return colorDiff[0] * colorDiff[0] +
             colorDiff[1] * colorDiff[1] +
             colorDiff[2] * colorDiff[2];
    }
    let dominantColor = dominantColors[0];
    let dominantColorDistanceSquared =
        getColorDistanceSquared(dominantColor.colorData, colorData);
    for (let i = 1; i < dominantColors.length; ++i) {
      const colorDistanceSquared =
          getColorDistanceSquared(dominantColors[i].colorData, colorData);
      if (colorDistanceSquared < dominantColorDistanceSquared) {
        dominantColor = dominantColors[i];
        dominantColorDistanceSquared = colorDistanceSquared;
      }
    }
    return dominantColor.name;
  }

  /**
   * Polls the video's dominant color (see getDominantCanvasColor()) until a
   * color different than the initial color is retrieved, resolving the returned
   * promise with the new color name. The video color is read by drawing the
   * video onto a canvas and reading the color of the canvas.
   * @private
   */
  async function pollNextVideoColor(video, canvas) {
    canvas.getContext('2d').drawImage(video, 0, 0);
    const initialColor = getDominantCanvasColor(canvas);
    console.log('pollNextVideoColor initial color: ' + initialColor);
    const resolver = new Resolver();
    function checkColor() {
      canvas.getContext('2d').drawImage(video, 0, 0);
      const color = getDominantCanvasColor(canvas);
      if (color != initialColor) {
        resolver.resolve(color);
        return;
      }
      setTimeout(checkColor, 0);
    }
    setTimeout(checkColor, 0);
    return resolver.promise;
  }

</script>

Output:
  ...
  CONSOLE MESSAGE: line 47: Waiting for pollNextVideoColor to receive a new color...
  CONSOLE MESSAGE: line 149: pollNextVideoColor initial color: white
  ...
  (we never receive another color)

Comment 4 by hbos@chromium.org, Dec 11 2017

Description: Show this description

Comment 5 by hbos@chromium.org, Dec 11 2017

Cc: hbos@chromium.org
Owner: mlamouri@chromium.org
Status: Assigned (was: Untriaged)
mlamouri@, I don't know if this is a video element issue or webrtc issue, but there seem to be something about content_shell that makes a video tag not update on new frames. I suspect the bug is closer to Blink>Media>Video than Blink>WebRTC>PeerConnection. Can you take a look or retriage?

If the bug is in WebRTC then I should be the owner but I'd like someone who knows more about the video element to take a look.

Comment 6 by hbos@chromium.org, Dec 11 2017 

The web platform test in #3 was placed in external/wpt/webrtc/ in a new file ending with .https.html.

Comment 7 by mlamouri@chromium.org, Dec 11 2017

Cc: mlamouri@chromium.org
Owner: hbos@chromium.org
Looking at the code, it seems that HTMLMediaElement should be calling `WebMediaPlayerMS::Paint`. Did you check if this is returning the returning the right frames?

Otherwise, because it seems to work in `content_shell` but not in LayoutTests, I assume we are setting some flags when running tests.

Comment 8 by mlamouri@chromium.org, Dec 28 2017 

I had another look at this running content_shell and the test seems highly flaky: it will pass sometimes but fail more often.

Comment 9 by hbos@chromium.org, Dec 28 2017 

Hmm. It could be a matter of having to re-fill the element to ensure a frame arrives at the other end, if there is a risk it is dropped or something is racing? I had to update my browser_tests version of the test at one point to re-fill in a loop until it detects a new color.
https://chromium-review.googlesource.com/c/chromium/src/+/810765/7/chrome/test/data/webrtc/peerconnection_replacetrack.js

I could try to run the latest version of the test as WPT/content_shell and see if that did the trick or if there still is a problem.
►

Sign in to add a comment