All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=836163

(For debugging:) Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?sid=d65e764a89fc347e5d068b7c9026b7de2d72633a3095c67a7b44bda46a714f10


Bot(s) for this bug's original alert(s):

webrtc-linux-large-tests
webrtc-win-large-tests

Graphs also correlate with initial landing as ##22905 and revert as #22914.

One known bug related to quality scaling was just fixed, #22993, let's see if that has any effect before taking further action.

No sign of recovery. Will spend tomorrow investigating. I'm afraid it's going to be difficult to revert. Almost all the changes seem related to H.264.

I checked the bad frames manually, there are no artifacts. Maybe minor bandwidth drop caused it. I think there's no pressure to revert the CL.

I'm trying to reproduce locally on a glinux box, using

./out/Release/webrtc_perf_tests --gtest_filter=FullStackTest.ForemanCif30kbpsWithoutPacketlossH264

This is what I get:

On commit #22905 (Reland "Move creating encoder to VideoStreamEncoder.").

$ ./out/Release/webrtc_perf_tests --gtest_filter=FullStackTest.ForemanCif30kbpsWithoutPacketlossH264
Note: Google Test filter = FullStackTest.ForemanCif30kbpsWithoutPacketlossH264
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from FullStackTest
[ RUN      ] FullStackTest.ForemanCif30kbpsWithoutPacketlossH264
[OpenH264] this = 0x0x7ff494001940, Warning:Change QP Range from(0,51) to (12,42)
[OpenH264] this = 0x0x7ff494001940, Warning:[Rc] iDid = 0,iContinualSkipFrames(3) is large
[OpenH264] this = 0x0x7ff494001940, Warning:[Rc] iDid = 0,iContinualSkipFrames(6) is large
[OpenH264] this = 0x0x7ff494001940, Warning:[Rc] iDid = 0,iContinualSkipFrames(3) is large
[OpenH264] this = 0x0x7ff494001940, Warning:[Rc] iDid = 0,iContinualSkipFrames(3) is large
[OpenH264] this = 0x0x7ff494001940, Warning:[Rc] iDid = 0,iContinualSkipFrames(3) is large
[OpenH264] this = 0x0x7ff494001940, Warning:[Rc] iDid = 0,iContinualSkipFrames(3) is large
[OpenH264] this = 0x0x7ff494001940, Warning:[Rc] iDid = 0,iContinualSkipFrames(6) is large
[OpenH264] this = 0x0x7ff494001940, Warning:[Rc] iDid = 0,iContinualSkipFrames(9) is large
[OpenH264] this = 0x0x7ff494001940, Warning:[Rc] iDid = 0,iContinualSkipFrames(3) is large
[OpenH264] this = 0x0x7ff494001940, Warning:[Rc] iDid = 0,iContinualSkipFrames(6) is large
- Uh, I'm-I'm not quite dead, sir.
[OpenH264] this = 0x0x7ff494001940, Warning:[Rc] iDid = 0,iContinualSkipFrames(3) is large
RESULT psnr: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= {30.575821,1.257406}  dB
RESULT ssim: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= {0.8375881,0.03816712}  score
RESULT sender_time: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= {4.5217391,4.1172647}  ms
RESULT receiver_time: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= {5.2274247,4.1350671}  ms
RESULT network_time: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= {2.5117057,1.5110618}  ms
RESULT total_delay_incl_network: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= {12.26087,5.4606075}  ms
RESULT time_between_rendered_frames: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= {150.47987,238.03223}  ms
RESULT encode_frame_rate: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= {7.4285714,3.5263826}  fps
RESULT encode_time: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= {1.3095238,0.55583893}  ms
RESULT media_bitrate: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= {30229.881,16019.738}  bps
RESULT fec_bitrate: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= {0,0}  bps
RESULT send_bandwidth: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= {4772028.5,643768.62}  bps
RESULT time_between_freezes: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= {5221.5714,4993.5789}  ms
RESULT min_psnr: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= 27.373966 dB
RESULT decode_time: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= {1,0}  ms
RESULT dropped_frames: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= 151 frames
RESULT cpu_usage: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= 5.103642 %
- Farewell, sweet Concorde!
[       OK ] FullStackTest.ForemanCif30kbpsWithoutPacketlossH264 (45426 ms)
[----------] 1 test from FullStackTest (45426 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (45426 ms total)
[  PASSED  ] 1 test.

On preceding commit #22904:

$ ./out/Release/webrtc_perf_tests --gtest_filter=FullStackTest.ForemanCif30kbpsWithoutPacketlossH264
Note: Google Test filter = FullStackTest.ForemanCif30kbpsWithoutPacketlossH264
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from FullStackTest
[ RUN      ] FullStackTest.ForemanCif30kbpsWithoutPacketlossH264
[OpenH264] this = 0x0x7fae940016f0, Warning:Change QP Range from(0,51) to (12,42)
[OpenH264] this = 0x0x7fae940016f0, Warning:[Rc] iDid = 0,iContinualSkipFrames(3) is large
[OpenH264] this = 0x0x7fae940016f0, Warning:[Rc] iDid = 0,iContinualSkipFrames(3) is large
[OpenH264] this = 0x0x7fae940016f0, Warning:[Rc] iDid = 0,iContinualSkipFrames(3) is large
[OpenH264] this = 0x0x7fae940016f0, Warning:[Rc] iDid = 0,iContinualSkipFrames(3) is large
[OpenH264] this = 0x0x7fae940016f0, Warning:[Rc] iDid = 0,iContinualSkipFrames(6) is large
[OpenH264] this = 0x0x7fae940016f0, Warning:[Rc] iDid = 0,iContinualSkipFrames(9) is large
- Uh, I'm-I'm not quite dead, sir.
[OpenH264] this = 0x0x7fae940016f0, Warning:[Rc] iDid = 0,iContinualSkipFrames(3) is large
[OpenH264] this = 0x0x7fae940016f0, Warning:Actual input framerate 17.000000 is different from framerate in setting 10.000000, suggest to use other rate control modes
RESULT psnr: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= {30.410248,1.3302797}  dB
RESULT ssim: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= {0.83174351,0.043419405}  score
RESULT sender_time: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= {4.9076923,3.6899993}  ms
RESULT receiver_time: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= {6.0338462,3.4873961}  ms
RESULT network_time: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= {2.6307692,1.6619658}  ms
RESULT total_delay_incl_network: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= {13.572308,4.4485264}  ms
RESULT time_between_rendered_frames: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= {138.42284,167.25842}  ms
RESULT encode_frame_rate: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= {7.9545455,3.0818718}  fps
RESULT encode_time: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= {1.5348837,0.65941613}  ms
RESULT media_bitrate: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= {30517.773,13921.615}  bps
RESULT fec_bitrate: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= {0,0}  bps
RESULT send_bandwidth: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= {4970787.8,444863.06}  bps
RESULT time_between_freezes: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= {4856.625,4847.2589}  ms
RESULT min_psnr: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= 27.288564 dB
RESULT decode_time: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= {1.0714286,0.37115374}  ms
RESULT dropped_frames: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= 125 frames
RESULT cpu_usage: foreman_cif_30kbps_net_delay_0_0_plr_0_H264= 6.6777305 %
- Farewell, sweet Concorde!
[       OK ] FullStackTest.ForemanCif30kbpsWithoutPacketlossH264 (45428 ms)
[----------] 1 test from FullStackTest (45428 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (45428 ms total)
[  PASSED  ] 1 test.

So some fairly large differences, e.g., dropped frames (125 vs 151).

I and Rasmus suspected that the VideoCodec passed to H264 was different before and after the change. But I've put a breakpoint at webrtc::H264EncoderImpl::InitEncode, and codec_settings are identical,

(gdb) p *codec_settings
$1 = {codecType = webrtc::kVideoCodecH264, plType = 0 '\000', width = 352, height = 288, startBitrate = 30, maxBitrate = 30, minBitrate = 30, 
  targetBitrate = 0, maxFramerate = 10, active = true, qpMax = 56, numberOfSimulcastStreams = 1 '\001', simulcastStream = {{width = 352, 
      height = 288, numberOfTemporalLayers = 1 '\001', maxBitrate = 30, targetBitrate = 30, minBitrate = 30, qpMax = 56, active = true}, {
      width = 0, height = 0, numberOfTemporalLayers = 0 '\000', maxBitrate = 0, targetBitrate = 0, minBitrate = 0, qpMax = 0, active = false}, {
      width = 0, height = 0, numberOfTemporalLayers = 0 '\000', maxBitrate = 0, targetBitrate = 0, minBitrate = 0, qpMax = 0, active = false}, {
      width = 0, height = 0, numberOfTemporalLayers = 0 '\000', maxBitrate = 0, targetBitrate = 0, minBitrate = 0, qpMax = 0, active = false}}, 
  spatialLayers = {{width = 0, height = 0, numberOfTemporalLayers = 0 '\000', maxBitrate = 0, targetBitrate = 0, minBitrate = 0, qpMax = 0, 
      active = false}, {width = 0, height = 0, numberOfTemporalLayers = 0 '\000', maxBitrate = 0, targetBitrate = 0, minBitrate = 0, qpMax = 0, 
      active = false}, {width = 0, height = 0, numberOfTemporalLayers = 0 '\000', maxBitrate = 0, targetBitrate = 0, minBitrate = 0, qpMax = 0, 
      active = false}, {width = 0, height = 0, numberOfTemporalLayers = 0 '\000', maxBitrate = 0, targetBitrate = 0, minBitrate = 0, qpMax = 0, 
      active = false}, {width = 0, height = 0, numberOfTemporalLayers = 0 '\000', maxBitrate = 0, targetBitrate = 0, minBitrate = 0, qpMax = 0, 
      active = false}}, mode = webrtc::kRealtimeVideo, expect_encode_from_texture = false, timing_frame_thresholds = {delay_ms = 200, 
    outlier_ratio_percent = 250}, codec_specific_ = {VP8 = {complexity = webrtc::kComplexityHigh, resilience = (unknown: 3000), 
      numberOfTemporalLayers = 0 '\000', denoisingOn = false, automaticResizeOn = false, frameDroppingOn = false, keyFrameInterval = 0}, VP9 = {
      complexity = webrtc::kComplexityHigh, resilienceOn = 184, numberOfTemporalLayers = 11 '\v', denoisingOn = false, frameDroppingOn = false, 
      keyFrameInterval = 0, adaptiveQpMode = false, automaticResizeOn = false, numberOfSpatialLayers = 0 '\000', flexibleMode = false}, H264 = {
      frameDroppingOn = true, keyFrameInterval = 3000, spsData = 0x0, spsLen = 0, ppsData = 0x0, ppsLen = 0, 
      profile = webrtc::H264::kProfileConstrainedBaseline}}}