All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=875391

(For debugging:) Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?sid=0e241481c1277b590a8430ef5abe60144ee03e29616a1f10cab18a5f9296e116


Bot(s) for this bug's original alert(s):

webrtc-linux-large-tests

Looks caused by https://webrtc-review.googlesource.com/93261.

Regression seems to mainly affect VP9 K/SVC.

Nisse, can you investigate?

I think I'll need a little help to understand this. When looking at send_bandwidth, there's a large increase for vp9ksvc_3sl_{low,medium, high}, but a large decrease for vp9ksvc_3sl_medium_netwoek_restricted.

I've been able to reproduce locally, and I have compared the logs from FullStackTest.VP9KSVC_3SL_High.

It appears results of probing are slightly different.

nisse@nisse:~/hack/webrtc/src$ grep 'Probing successful' ../webrtc_perf_tests-24253.log
(probe_bitrate_estimator.cc:149): Probing successful [cluster id: 0] [send: 30480 bytes / 34 ms = 896.471 kb/s] [receive: 31488 bytes / 34 ms = 926.118 kb/s]
(probe_bitrate_estimator.cc:149): Probing successful [cluster id: 0] [send: 41688 bytes / 46 ms = 906.261 kb/s] [receive: 42696 bytes / 47 ms = 908.426 kb/s]
(probe_bitrate_estimator.cc:149): Probing successful [cluster id: 1] [send: 33624 bytes / 20 ms = 1681.2 kb/s] [receive: 33624 bytes / 20 ms = 1681.2 kb/s]
(probe_bitrate_estimator.cc:149): Probing successful [cluster id: 1] [send: 44832 bytes / 25 ms = 1793.28 kb/s] [receive: 44832 bytes / 25 ms = 1793.28 kb/s]
(probe_bitrate_estimator.cc:149): Probing successful [cluster id: 2] [send: 39608 bytes / 11 ms = 3600.73 kb/s] [receive: 40720 bytes / 12 ms = 3393.33 kb/s]
(probe_bitrate_estimator.cc:149): Probing successful [cluster id: 2] [send: 50848 bytes / 14 ms = 3632 kb/s] [receive: 51960 bytes / 15 ms = 3464 kb/s]
(probe_bitrate_estimator.cc:149): Probing successful [cluster id: 3] [send: 50960 bytes / 10 ms = 5096 kb/s] [receive: 55896 bytes / 10 ms = 5589.6 kb/s]
(probe_bitrate_estimator.cc:149): Probing successful [cluster id: 3] [send: 61096 bytes / 12 ms = 5091.33 kb/s] [receive: 64480 bytes / 12 ms = 5373.33 kb/s]
(probe_bitrate_estimator.cc:149): Probing successful [cluster id: 3] [send: 69560 bytes / 12 ms = 5796.67 kb/s] [receive: 72944 bytes / 12 ms = 6078.67 kb/s]

nisse@nisse:~/hack/webrtc/src$ grep 'Probing successful' ../webrtc_perf_tests-24254.log
(probe_bitrate_estimator.cc:149): Probing successful [cluster id: 0] [send: 30480 bytes / 34 ms = 896.471 kb/s] [receive: 31488 bytes / 35 ms = 899.657 kb/s]
(probe_bitrate_estimator.cc:149): Probing successful [cluster id: 0] [send: 41688 bytes / 47 ms = 886.979 kb/s] [receive: 42696 bytes / 47 ms = 908.426 kb/s]
(probe_bitrate_estimator.cc:149): Probing successful [cluster id: 1] [send: 33624 bytes / 19 ms = 1769.68 kb/s] [receive: 33624 bytes / 19 ms = 1769.68 kb/s]
(probe_bitrate_estimator.cc:149): Probing successful [cluster id: 1] [send: 44832 bytes / 26 ms = 1724.31 kb/s] [receive: 44832 bytes / 25 ms = 1793.28 kb/s]
(probe_bitrate_estimator.cc:149): Probing successful [cluster id: 2] [send: 39168 bytes / 11 ms = 3560.73 kb/s] [receive: 40920 bytes / 11 ms = 3720 kb/s]
(probe_bitrate_estimator.cc:149): Probing successful [cluster id: 2] [send: 47584 bytes / 14 ms = 3398.86 kb/s] [receive: 48176 bytes / 15 ms = 3211.73 kb/s]
(probe_bitrate_estimator.cc:149): Probing successful [cluster id: 3] [send: 59304 bytes / 13 ms = 4561.85 kb/s] [receive: 63408 bytes / 12 ms = 5284 kb/s]
(probe_bitrate_estimator.cc:149): Probing successful [cluster id: 3] [send: 69400 bytes / 15 ms = 4626.67 kb/s] [receive: 71872 bytes / 15 ms = 4791.47 kb/s]
(probe_bitrate_estimator.cc:149): Probing successful [cluster id: 3] [send: 77744 bytes / 15 ms = 5182.93 kb/s] [receive: 80216 bytes / 15 ms = 5347.73 kb/s]

The value from probe 3 (5796666 bit/s before the cl, 5182933 bit/s after) is reported as available bitrate and the perf value "send_bandwidth".

What's the next step?

Philip, can you see any reason for the change in probing results? As far as I understand, probing depends on transport feedback, with methods invoked from Call, before the packets reach the code in RtpVideoStreamReceier that is change in my cl.

It also looks a bit odd that most probes have received bytes > sent bytes, but that's the case both before and after my cl.

I've focused on the *_High test, where I don't think the transport used in the test does discards any packets or rewrites sequence numbers.

We've tracked this down to some problem with the nack logic. Running the test 

webrtc_perf_tests --gtest_filter='FullStackTest.VP9KSVC_3SL_Medium_Network_Restricted'

and from debug printf, I see that packets have continuous sequence numbers when they reach RtpVideoStreamReceiver::OnRtpPacket, but a few of them don't make it to NackModule::OnReceivedPacket. 

So we get nacks and rtx traffic which we don't have before my cl.

I've prepared a cl, https://webrtc-review.googlesource.com/c/src/+/95146

It seems to make the send_bandwidth stats recover in local testing. Waiting for Erik's review.

The following revision refers to this bug:
  https://webrtc.googlesource.com/src.git/+/0b9267830ca0e85a84f7bce5ab93417c7a0ce2d1

commit 0b9267830ca0e85a84f7bce5ab93417c7a0ce2d1
Author: Niels Möller <nisse@webrtc.org>
Date: Wed Aug 22 08:15:27 2018

Fix NACK logic for empty packets

This was broken in cl
https://webrtc-review.googlesource.com/c/src/+/93261.

Bug:  chromium:875391 , webrtc:7135
Change-Id: Id2051bde8a5248dd5aeefa782f9d63513d107df4
Reviewed-on: https://webrtc-review.googlesource.com/95146
Reviewed-by: Philip Eliasson <philipel@webrtc.org>
Reviewed-by: Erik Språng <sprang@webrtc.org>
Commit-Queue: Niels Moller <nisse@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#24378}
[modify] https://crrev.com/0b9267830ca0e85a84f7bce5ab93417c7a0ce2d1/video/rtp_video_stream_receiver.cc
[modify] https://crrev.com/0b9267830ca0e85a84f7bce5ab93417c7a0ce2d1/video/rtp_video_stream_receiver.h

Graphs look recovering. Marking as fixed.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c4cd81444beef302fb5dfdaf062f8acd1394a297

commit c4cd81444beef302fb5dfdaf062f8acd1394a297
Author: webrtc-chromium-autoroll <webrtc-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Mon Aug 27 09:46:12 2018

Roll src/third_party/webrtc efbcb31cb67e..1946a3f0fe2c (62 commits)

https://webrtc.googlesource.com/src.git/+log/efbcb31cb67e..1946a3f0fe2c


git log efbcb31cb67e..1946a3f0fe2c --date=short --no-merges --format='%ad %ae %s'
2018-08-26 ssilkin@webrtc.org Add frame rate parameter to SpatialLayer struct.
2018-08-24 peah@webrtc.org AEC3: Add state-specific suppressor behaviors
2018-08-24 benwright@webrtc.org Adds support for JSON config in video_replay.
2018-08-24 aleloi@webrtc.org Optionally disable digital adaptive AGC2.
2018-08-24 magjed@webrtc.org Fix a bug in barcode_decoder.py
2018-08-24 brandtr@webrtc.org Add VideoEncoder::Settings::numberOfSimulcastStreams.
2018-08-24 magjed@webrtc.org Reland "Update video_quality_analysis to align videos instead of using barcodes"
2018-08-24 magjed@webrtc.org Revert "Update video_quality_analysis to align videos instead of using barcodes"
2018-08-24 mbonadei@webrtc.org Remove unused include of <android/log.h>.
2018-08-24 ilnik@webrtc.org Enable video adaptation for all screenshare content
2018-08-24 gustaf@webrtc.org AEC3: Removing more dead code from the suppressor
2018-08-24 jonaso@webrtc.org Add support for enabling simulcast in "Plan B" using MediaConstraints.
2018-08-24 magjed@webrtc.org Update video_quality_analysis to align videos instead of using barcodes
2018-08-24 ilnik@webrtc.org Add unlimited retransmission experiment for screenshare
2018-08-24 phoglund@webrtc.org Remove client.webrtc.branches.
2018-08-24 kthelgason@webrtc.org Remove assumption that all video codecs are known.
2018-08-24 asapersson@webrtc.org Change some pointers to std::unique_ptr in rtp_rtcp tests.
2018-08-24 buildbot@webrtc.org Roll chromium_revision c86aa801eb..ca3a5e1cbb (585622:585726)
2018-08-24 gustaf@webrtc.org AEC3: Removing old suppressor logic
2018-08-24 peah@webrtc.org Adding quiet mode for audioproc_f
2018-08-24 zstein@webrtc.org Support domain name ICE candidates
2018-08-23 buildbot@webrtc.org Roll chromium_revision 607c2083aa..c86aa801eb (585516:585622)
2018-08-23 ouj@fb.com The default logic for creating video bitrate allocator.
2018-08-23 mbonadei@webrtc.org Revert "Reland "Enable any address ports by default.""
2018-08-23 zstein@webrtc.org Ad-hoc rate limiting for UDPPort::SendTo failures.
2018-08-23 magjed@webrtc.org Android PeerConnectionFactory: Build without video codecs by default
2018-08-23 steveanton@webrtc.org Add steveanton to WATCHLISTS for api/ and pc/
2018-08-23 buildbot@webrtc.org Roll chromium_revision bd84ccec52..607c2083aa (585399:585516)
2018-08-23 magjed@webrtc.org Add tool for aliging video files
2018-08-23 ssilkin@webrtc.org Write scalability structure in flexible mode.
2018-08-23 gustaf@webrtc.org Correct audioproc_f to support the new echo canceller activation III
2018-08-23 yinwa@webrtc.org Fix flaky test TestFlexfecRtpStatePreservation.
2018-08-23 gustaf@webrtc.org AEC3: Removing the coherence computation
2018-08-23 nisse@webrtc.org Delete unneeded includes of rtp_payload_registry.h
2018-08-23 magjed@webrtc.org Remove SequencedTaskChecker from Y4mFileReader
2018-08-23 titovartem@webrtc.org Remove temporary SetConfig method from NetworkSimulatioInterface.
2018-08-23 peah@webrtc.org AEC3: Adding the option for applying a fixed delay to the capture signal
2018-08-23 magjed@webrtc.org Add Y4mFileReader
2018-08-23 titovartem@webrtc.org Remove old sigslot.h header
2018-08-23 nisse@webrtc.org Delete almost all use of MediaConstraintsInterface in the PeerConnection API
2018-08-23 mbonadei@webrtc.org Add //test:fileutils_unittests to //test:test_support_unittests deps.
2018-08-23 gustaf@webrtc.org Correct audioproc_f to support the new echo canceller activation II
2018-08-23 buildbot@webrtc.org Roll chromium_revision 6f37604c1d..bd84ccec52 (585288:585399)
2018-08-23 buildbot@webrtc.org Roll chromium_revision 07be93b641..6f37604c1d (585077:585288)
2018-08-22 tkchin@webrtc.org Add didRemoveReceiver delegate callback.
2018-08-22 buildbot@webrtc.org Roll chromium_revision ab3ab1ba02..07be93b641 (584945:585077)
2018-08-22 lizeb@chromium.org Fix VideoSendStreamImpl::OnPacketFeedbackError: operator precedence.
2018-08-22 nisse@webrtc.org Delete deprecated methods on AudioCodingModule
2018-08-22 nisse@webrtc.org Eliminate NackModule dependency on VCMPacket
2018-08-22 titovartem@webrtc.org Eliminate methods SetConfig() from DirectTransport and FakeNetworkPipe
2018-08-22 ilnik@webrtc.org Revert removal of simulcast screenshare experimental code (killswitch checks)
2018-08-22 srte@webrtc.org Adds start bitrate handling to task queue congestion controller.
2018-08-22 ilnik@webrtc.org Reland "Move FakeCodec to separate target and behave like real encoder."
2018-08-22 srte@webrtc.org Avoids posting tasks in congestion controller.
2018-08-22 mbonadei@webrtc.org Add Chromium default values for rtc_exclude_field_trial_default.
2018-08-22 nisse@webrtc.org Enable --rtc_event_log_name flag for more tests.
2018-08-22 mbonadei@webrtc.org Reland "Enable any address ports by default."
2018-08-22 nisse@webrtc.org Fix NACK logic for empty packets
2018-08-22 emircan@webrtc.org Disable flaky TestFlexfecRtpStatePreservation
2018-08-22 koloskova@webrtc.org Network & bitrate controllers are added for PCC.
2018-08-22 kwiberg@webrtc.org Audio encoder tests: Create audio encoders the new way
2018-08-22 qingsi@webrtc.org Revert "Reland "Enable any address ports by default.""


Created with:
  gclient setdep -r src/third_party/webrtc@1946a3f0fe2c

The AutoRoll server is located here: https://autoroll.skia.org/r/webrtc-chromium-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:linux_chromium_archive_rel_ng;master.tryserver.chromium.mac:mac_chromium_archive_rel_ng

BUG=chromium:None,chromium:690537,chromium:877279,chromium:none,chromium:None,chromium:None,chromium:856088,chromium:None,chromium:None,chromium:None,chromium:None,chromium:None,chromium:None,chromium:None,chromium:None,chromium:None,chromium:690537,chromium:none,chromium:None,chromium:875391
TBR=webrtc-chromium-sheriffs-robots@google.com

Change-Id: I201d39e11331d6b91c327b49d7be7589947b5a65
Reviewed-on: https://chromium-review.googlesource.com/1188496
Reviewed-by: webrtc-chromium-autoroll <webrtc-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: webrtc-chromium-autoroll <webrtc-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#586220}
[modify] https://crrev.com/c4cd81444beef302fb5dfdaf062f8acd1394a297/DEPS