All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=645589

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg4eX7qQoM


Bot(s) for this bug's original alert(s):

chromium-rel-win7-x64-dual

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9001970626326580640

=== Auto-CCing suspected CL author perkj@chromium.org ===

Hi perkj@chromium.org, the bisect results pointed to your CL below as possibly
causing a regression. Please have a look at this info and see whether
your CL be related.


===== BISECT JOB RESULTS =====
Status: failed


===== SUSPECTED CL(s) =====
Subject : Roll WebRTC 13869:13951 (74 commits)
Author  : Per
Commit description:
  
Changes: https://chromium.googlesource.com/external/webrtc/trunk/webrtc.git/+log/446b8bf..f80f5b3

$ git log 446b8bf..f80f5b3 --date=short --no-merges --format=%ad %ae %s
2016-08-29 ehmaldonado@webrtc.org Remove Chromium Clang warnings only on Windows.
2016-08-29 sakal@webrtc.org Remove the old AndroidVideoCapturer stack code.
2016-08-29 nisse@webrtc.org Reland of Delete method cricket::VideoFrame::Copy. (patchset #1 id:1 of https://codereview.webrtc.org/2275313003/ )
2016-08-29 nisse@webrtc.org Delete deprecated and unused method VideoFrame::SetTimeStamp.
2016-08-28 kjellander@webrtc.org MB: Flip Windows bots to GN by default
2016-08-26 deadbeef@webrtc.org Combining "SetTransportChannel" and "SetRtcpTransportChannel".
2016-08-26 deadbeef@webrtc.org Add parameter to TransportController to not change ICE role on restart.
2016-08-26 kwiberg@webrtc.org Fix Chromium clang plugin warnings
2016-08-26 ehmaldonado@webrtc.org GN: Fix windows clang errors. Attempt 2.
2016-08-26 kjellander@webrtc.org Adding GYP/GN owners to stats/ aligning with all other dirs.
2016-08-26 zhihuang@webrtc.org Log how often DTLS negotiation failed because of incompatible ciphersuites.
2016-08-26 peah@webrtc.org Added logging of the level controller metrics.
2016-08-26 danilchap@webrtc.org Change RtpSender::OnReceiveNACK name and signature Name changed to follow style. list replaced with vector to decrease number of included headers.
2016-08-26 peah@webrtc.org Deactivated the intelligibility enhancement functionality by default
2016-08-26 peah@webrtc.org Revert of Added functionality for specifying the initial signal level to use for the gain estimation in the l… (patchset #8 id:160001 of https://codereview.webrtc.org/2254973003/ )
2016-08-26 kjellander@webrtc.org MB: Flip Linux bots to GN by default.
2016-08-26 danilchap@webrtc.org AbsoluteSendTime rtp header extension publish MsTo24Bit conversion Since this conversion is used in multiple place and extension seems right place to keep it in.
2016-08-26 ossu@webrtc.org Removed virtual from several methods in DecoderDatabase to minimize the number of points that need to be mocked for testing.
2016-08-26 peah@webrtc.org This CL adds functionality in the level controller to receive a signal level to use initially, instead of the default initial signal level.
2016-08-26 kwiberg@webrtc.org Replace calls to assert() with RTC_DCHECK_*() in .c code
2016-08-26 johan@webrtc.org Skip unit test if GYP_DEFINES="rtc_use_h264=1" is not set.
2016-08-25 henrik.lundin@webrtc.org NetEq: Update CNG code to accommodate 48 kHz sample rate
2016-08-25 peah@webrtc.org Adding AecDump functionality to AppRTCDemo for iOS
2016-08-25 deadbeef@webrtc.org Renaming BaseChannel methods and adding comments for added clarity.
2016-08-25 philipel@webrtc.org Revert of Delete method cricket::VideoFrame::Copy. (patchset #3 id:210001 of https://codereview.webrtc.org/2275243002/ )
2016-08-25 nisse@webrtc.org Reland of Delete method cricket::VideoFrame::Copy. (patchset #1 id:1 of https://codereview.webrtc.org/2087923004/ )
2016-08-25 henrika@webrtc.org Avoids java.lang.NullPointerException in WebRtcAudioRecord
2016-08-25 ivoc@webrtc.org Moved format_macros.h from rtc_base to rtc_base_approved.
2016-08-25 ehmaldonado@webrtc.org GN: Fix Windows Clang errors
2016-08-25 maxmorin@webrtc.org Fix error when accumulating floats in an int.
2016-08-25 hbos@webrtc.org Refactor certificate stats collection, added SSLCertificateStats.
2016-08-25 magjed@webrtc.org Implement CVO for iOS capturer
2016-08-25 ehmaldonado@webrtc.org Add missing "//build/config/sanitizers:deps" to executable targets.
2016-08-25 brandtr@webrtc.org Move InsertZeroColumns and CopyColumn to ::internal.
2016-08-25 kwiberg@webrtc.org GN build rules for four audio processing test executables
2016-08-25 henrik.lundin@webrtc.org Make neteq_rtpplay parse RTP header extensions
2016-08-25 aleloi@webrtc.org Removed inline definitions and added destructors to fix chromium-style.
2016-08-25 henrik.lundin@webrtc.org NetEq: only update current_rtp_payload_type_ when validated
2016-08-24 deadbeef@webrtc.org Fix setting the MTU for SCTP.
2016-08-24 deadbeef@webrtc.org Fixing inconsistency with behavior of `ClearGettingPorts`.
2016-08-24 deadbeef@webrtc.org Fixing segfault caused by TurnServer.
2016-08-24 deadbeef@webrtc.org Fixing off-by-one error with max SCTP id.
2016-08-24 deadbeef@webrtc.org Fixing timestamp comparison assert.
2016-08-24 glaznev@webrtc.org Increase QP threshold for H.264 encoder QP based scaling.
2016-08-24 tkchin@webrtc.org Restart capture session if needed on active.
2016-08-24 henrik.lundin@webrtc.org NetEq: Don't check sample rate and frame size upon error
2016-08-24 henrik.lundin@webrtc.org Make FakeDecodeFromFile handle codec-internal CNG
2016-08-24 kjellander@webrtc.org MB: Flip Mac bots to GN by default.
2016-08-24 ehmaldonado@webrtc.org Roll chromium_revision e3860bd297..938114be1e (412289:414059)
2016-08-24 kjellander@webrtc.org GN: Synchronize resources between Android and iOS.
2016-08-24 maxmorin@webrtc.org Make dependency of audio_device of ApplicationServices explicit. Tested in https://codereview.webrtc.org/2276903002.
2016-08-24 philipel@webrtc.org Now probe for x3 and x6 of the initial start bitrate and allow for faster receive bitrates when calculating probing estimates.
2016-08-24 ivoc@webrtc.org Added GN target for webrtc_opus_fec_test.
2016-08-24 ehmaldonado@webrtc.org Disable examples for GYP Android bots.
2016-08-24 sakal@webrtc.org Revert of GN build rules for four audio processing test executables (patchset #3 id:40001 of https://codereview.webrtc.org/2267403003/ )
2016-08-24 kwiberg@webrtc.org GN build rules for four audio processing test executables
2016-08-24 philipel@webrtc.org Only use payload size within the know send/receive interval for probing calculations.
2016-08-24 kwiberg@webrtc.org iLBC: Handle a case of bad input data
2016-08-24 philipel@webrtc.org Set send side bitrate estimate on successful probing attempt.
2016-08-24 kjellander@webrtc.org GN: Add resources for webrtc_perf_tests on Android
2016-08-24 ivoc@webrtc.org Added GN target for isac_test.
2016-08-24 aleloi@webrtc.org Removals and renamings in the new audio mixer.
2016-08-24 nisse@webrtc.org Add ThreadChecker to the TimestampAligner class.
2016-08-24 aleloi@webrtc.org Increased column width for python tool rtp_analyzer.py.
2016-08-24 aleloi@webrtc.org Updated mixer unittests and fixed a related bug in the new mixer.
2016-08-24 hbos@webrtc.org RTCStats and RTCStatsReport added (webrtc/stats).
2016-08-24 aleloi@webrtc.org Added a level indicator to new mixer.
2016-08-24 kthelgason@webrtc.org Remove outdated symlink
2016-08-24 sakal@webrtc.org Fix AppRTC Android Demo GN build when is_component_build=true.
2016-08-24 kjellander@webrtc.org MB: Flip Android bots to GN by default.
2016-08-23 terelius@webrtc.org Define a protobuf format for representing plots. Add code to convert the C-representation generated by the RtcEventLog analysis tool, to the new protobuf format.
2016-08-23 terelius@webrtc.org Adds function for computing moving average to visualization tool.
2016-08-23 honghaiz@webrtc.org Add logs and small change in BasicPortAllocator.
2016-08-23 isheriff@google.com ProbingEstimator: Erase history based on time threshold

TBR=
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_chromium_archive_rel_ng;master.tryserver.chromium.mac:mac_chromium_archive_rel_ng
BUG=

Review URL: https://codereview.chromium.org/2293433002 .

Cr-Commit-Position: refs/heads/master@{#415227}
Commit  : 55b05938527117dfcb5db481cd8bd8590c65f0a4
Date    : Tue Aug 30 06:45:22 2016


===== TESTED REVISIONS =====
Revision         Mean    Std Dev  N  Good?
chromium@415178  138657  7824.27  8  good
chromium@415205  145047  24955.8  8  good
chromium@415218  134120  4841.61  5  good
chromium@415225  141283  3226.85  5  good
chromium@415226  131550  6004.49  5  good
chromium@415227  204226  1664.36  5  bad    <--
chromium@415228  201606  2843.78  5  bad
chromium@415231  204356  5626.77  8  bad

Bisect job ran on: win_x64_perf_bisect
Bug ID: 645589

Test Command: src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --also-run-disabled-tests webrtc.datachannel
Test Metric: vm_working_set_final_size_total/vm_working_set_final_size_total
Relative Change: 44.00%
Score: 99.9

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_x64_perf_bisect/builds/1470
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9001970626326580640


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5824512879230976

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Yikes. Any way to narrow down?

Yes - you get quite a bit range when we hold up webrtc rolls around the Chrome cuts. 

deadbeaf@
Can you take a look since you have touched sctp in this range? 

Friendly ping on this.

Can someone help with the instruction to run the test manually?
I can run this: src/tools/perf/run_benchmark -v  --output-format=chartjson --upload-results --also-run-disabled-tests webrtc.datachannel
But not sure how to extract the Test Metric vm_working_set_final_size_total/vm_working_set_final_size_total. 
And if I added the --browser flag, it would complain not being able to find the chrome browser. 

The output of the test command (under Linux) is as follows. 
(INFO) 2016-09-22 09:55:17,643 wpr_server._IsStarted:152  WPR ports: {}
(INFO) 2016-09-22 09:55:17,743 wpr_server._IsStarted:152  WPR ports: {}
(INFO) 2016-09-22 09:55:17,844 wpr_server._IsStarted:152  WPR ports: {'http': 48217, 'https': 35124}
(INFO) 2016-09-22 09:55:17,863 msr_power_monitor._CheckMSRs:94  Cannot monitor power: [Errno 2] No such file or directory
(INFO) 2016-09-22 09:55:17,864 msr_power_monitor.CanMonitorPower:119  Try running tools/telemetry/build/linux_setup_msr.py.
(INFO) 2016-09-22 09:55:17,864 power.__init__:42  PowerMetric created, not running
(INFO) 2016-09-22 09:55:17,865 desktop_browser_backend.GetBrowserStartupArgs:250  Requested remote debugging port: 0
....
(INFO) 2016-09-22 09:55:50,497 browser._LogBrowserInfo:124    gl_renderer         : Quadro 600/PCIe/SSE2
(INFO) 2016-09-22 09:55:50,497 browser._LogBrowserInfo:124    gl_reset_notification_strategy: 33362
(INFO) 2016-09-22 09:55:50,497 browser._LogBrowserInfo:124    gl_vendor           : NVIDIA Corporation
(INFO) 2016-09-22 09:55:50,497 browser._LogBrowserInfo:124    gl_version          : 4.5.0 NVIDIA 361.42
(INFO) 2016-09-22 09:55:50,498 browser._LogBrowserInfo:124    gl_ws_extensions    : GLX_EXT_visual_info GLX_EXT_visual_rating GLX_SGIX_fbconfig GLX_SGIX_pbuffer GLX_SGI_video_sync GLX_SGI_swap_control GLX_EXT_swap_control GLX_EXT_swap_control_tear GLX_EXT_texture_from_pixmap GLX_EXT_buffer_age GLX_ARB_create_context GLX_ARB_create_context_profile GLX_EXT_create_context_es_profile GLX_EXT_create_context_es2_profile GLX_ARB_create_context_robustness GLX_NV_delay_before_swap GLX_EXT_stereo_tree GLX_ARB_context_flush_control GLX_ARB_multisample GLX_NV_float_buffer GLX_ARB_fbconfig_float GLX_NV_swap_group GLX_EXT_framebuffer_sRGB GLX_NV_multisample_coverage GLX_NV_copy_image GLX_NV_copy_buffer GLX_NV_video_capture 
(INFO) 2016-09-22 09:55:50,498 browser._LogBrowserInfo:124    gl_ws_vendor        : NVIDIA Corporation
(INFO) 2016-09-22 09:55:50,498 browser._LogBrowserInfo:124    gl_ws_version       : 1.4
(INFO) 2016-09-22 09:55:50,498 browser._LogBrowserInfo:124    in_process_gpu      : False
(INFO) 2016-09-22 09:55:50,498 browser._LogBrowserInfo:124    initialization_time : 4.8432
(INFO) 2016-09-22 09:55:50,498 browser._LogBrowserInfo:124    jpeg_decode_accelerator_supported: False
(INFO) 2016-09-22 09:55:50,498 browser._LogBrowserInfo:124    lenovo_dcute        : False
(INFO) 2016-09-22 09:55:50,498 browser._LogBrowserInfo:124    max_msaa_samples    : 64
(INFO) 2016-09-22 09:55:50,498 browser._LogBrowserInfo:124    optimus             : False
(INFO) 2016-09-22 09:55:50,498 browser._LogBrowserInfo:124    pixel_shader_version: 4.50
(INFO) 2016-09-22 09:55:50,499 browser._LogBrowserInfo:124    process_crash_count : 0
(INFO) 2016-09-22 09:55:50,499 browser._LogBrowserInfo:124    sandboxed           : True
(INFO) 2016-09-22 09:55:50,499 browser._LogBrowserInfo:124    software_rendering  : False
(INFO) 2016-09-22 09:55:50,499 browser._LogBrowserInfo:124    vertex_shader_version: 4.50
(INFO) 2016-09-22 09:55:50,499 browser._LogBrowserInfo:124    video_decode_accelerator_flags: 0
(INFO) 2016-09-22 09:55:50,499 browser._LogBrowserInfo:126  Feature Status:
(INFO) 2016-09-22 09:55:50,499 browser._LogBrowserInfo:128    2d_canvas           : enabled
(INFO) 2016-09-22 09:55:50,499 browser._LogBrowserInfo:128    flash_3d            : enabled
(INFO) 2016-09-22 09:55:50,499 browser._LogBrowserInfo:128    flash_stage3d       : enabled
(INFO) 2016-09-22 09:55:50,499 browser._LogBrowserInfo:128    flash_stage3d_baseline: enabled
(INFO) 2016-09-22 09:55:50,500 browser._LogBrowserInfo:128    gpu_compositing     : enabled
(INFO) 2016-09-22 09:55:50,500 browser._LogBrowserInfo:128    multiple_raster_threads: enabled_on
(INFO) 2016-09-22 09:55:50,500 browser._LogBrowserInfo:128    native_gpu_memory_buffers: disabled_software
(INFO) 2016-09-22 09:55:50,500 browser._LogBrowserInfo:128    rasterization       : disabled_software
(INFO) 2016-09-22 09:55:50,500 browser._LogBrowserInfo:128    video_decode        : unavailable_software
(INFO) 2016-09-22 09:55:50,500 browser._LogBrowserInfo:128    video_encode        : enabled
(INFO) 2016-09-22 09:55:50,500 browser._LogBrowserInfo:128    vpx_decode          : unavailable_software
(INFO) 2016-09-22 09:55:50,500 browser._LogBrowserInfo:128    webgl               : enabled
(INFO) 2016-09-22 09:55:50,500 browser._LogBrowserInfo:130  Driver Bug Workarounds:
(INFO) 2016-09-22 09:55:50,501 browser._LogBrowserInfo:132    clear_uniforms_before_first_program_use
(INFO) 2016-09-22 09:55:50,501 browser._LogBrowserInfo:132    disable_discard_framebuffer
(INFO) 2016-09-22 09:55:50,501 browser._LogBrowserInfo:132    disable_framebuffer_cmaa
(INFO) 2016-09-22 09:55:50,501 browser._LogBrowserInfo:132    disable_transparent_visuals
(INFO) 2016-09-22 09:55:50,501 browser._LogBrowserInfo:132    force_cube_complete
(INFO) 2016-09-22 09:55:50,501 browser._LogBrowserInfo:132    init_gl_position_in_vertex_shader
(INFO) 2016-09-22 09:55:50,501 browser._LogBrowserInfo:132    init_vertex_attributes
(INFO) 2016-09-22 09:55:50,501 browser._LogBrowserInfo:132    pack_parameters_workaround_with_pack_buffer
(INFO) 2016-09-22 09:55:50,501 browser._LogBrowserInfo:132    scalarize_vec_and_mat_constructor_args
(INFO) 2016-09-22 09:55:50,501 browser._LogBrowserInfo:132    unpack_alignment_workaround_with_unpack_buffer
(INFO) 2016-09-22 09:55:50,502 browser._LogBrowserInfo:132    unpack_overlapping_rows_separately_unpack_buffer
(INFO) 2016-09-22 09:55:50,502 browser._LogBrowserInfo:132    use_current_program_after_successful_link
(INFO) 2016-09-22 09:55:50,502 browser._LogBrowserInfo:132    use_virtualized_gl_contexts
(INFO) 2016-09-22 09:55:51,022 cache_temperature.EnsurePageCacheTemperature:55  PageCacheTemperature: any
(INFO) 2016-09-22 09:55:52,720 msr_power_monitor._CheckMSRs:94  Cannot monitor power: [Errno 2] No such file or directory
(INFO) 2016-09-22 09:55:52,721 msr_power_monitor.CanMonitorPower:119  Try running tools/telemetry/build/linux_setup_msr.py.
(INFO) 2016-09-22 09:56:39,208 msr_power_monitor._CheckMSRs:94  Cannot monitor power: [Errno 2] No such file or directory
(INFO) 2016-09-22 09:56:39,208 msr_power_monitor.CanMonitorPower:119  Try running tools/telemetry/build/linux_setup_msr.py.
(INFO) 2016-09-22 09:56:39,511 power.Close:104  Closing power monitors
(INFO) 2016-09-22 09:56:39,511 platform.IsMonitoringPower:308  IsMonitoringPower: False
[       OK ] 30s_datachannel_transfer (82180 ms)
[  PASSED  ] 1 test.

The VM size increased for windows7-single (which appeared to occur at a different CL): https://chromeperf.appspot.com/report?sid=62e42b54ddcdded49b56d775434293ca82d0a3fdb49ae3a2201c7d6e9aa2f10c

I looked through all webrtc CLs in the WebRTC roll list. Below are all the potential suspects that are related to data channel. 

ff96df5 Combining "SetTransportChannel" and "SetRtcpTransportChannel". 
Unlikely. Only changed the setup of transport channels. 

4501943 Add parameter to TransportController to not change ICE role on restart. by Taylor Brandstetter · 6 weeks ago
Unlikely. Only added an argument in the constructor which is not used. 

6f7a7f6 Log how often DTLS negotiation failed because of incompatible ciphersuites. by zhihuang 
Unlikely. This only logs a UMA metric when DTLS negotiation failed. 

a9d5eb4 Renaming BaseChannel methods and adding comments for added clarity. by Taylor Brandstetter
Unlikely. This CL mainly renamed a few methods. 

9a70d05 Refactor certificate stats collection, added SSLCertificateStats. 
Unlikely. This change is not data channel specific. 

77f77dd Fix setting the MTU for SCTP. by deadbeef · 6 weeks ago
Likely the cause. This fixed the MTU setting for SCTP but may cause more packets being fragmented. 

5e5708b Fixing inconsistency with behavior of `ClearGettingPorts`. by deadbeef 
Unlikely. 

c66c78a Fixing off-by-one error with max SCTP id. by Taylor Brandstetter 
Unlikely. It is unlikely that we have reached the max SCTP ID in tests consistently. 

15c987b Roll chromium_revision e3860bd297..938114be1e (412289:414059) by ehmaldonado 
Possible but small. The chromium change included a boringssl change. If it had caused the problem, it should have caused problems elsewhere.  

3d7ad3d RTCStats and RTCStatsReport added (webrtc/stats). by hbos · 6 weeks ago
Unlikely. Not data channel specific. 

We can create revert of CLs to verify the suspects.
 
rschoen@, Is there a way for us to run the test on a perf test bot without submitting the CL? 

By the way, this figure shows that the regression on Windows-7-single happened on a different range of commits. Can you do a bisection on the windows-7-single bot?
https://chromeperf.appspot.com/report?sid=62e42b54ddcdded49b56d775434293ca82d0a3fdb49ae3a2201c7d6e9aa2f10c

Oh dear. Yeah, performance regressions coming in from enormous WebRTC rolls are no fun, since Telemetry can't bisect into the roll. I'll add that to our suggested improvements for perf sheriffing. For browser tests this is easier, since they run on the FYI bots, and then we can see which WebRTC CL caused the regression in Chromium.

Re #13: A quick googling shows https://www.chromium.org/developers/telemetry/performance-try-bots might normally be what you want to debug this. The problem is just that you can't upload patches to third_party/webrtc :( This means you either need to do speculative reverts + rolls and see what happens on the bots, or run locally.

It appears the concept of working set memory is win-only, so you need to bisect on Windows. I suggest reading up on what it is and if it's even that bad, and looking at other memory metrics for win, before sinking tons of work into bisecting. Doing a manual bisect on Windows is probably a terrible experience because building is so slow. The first thing to try is to run the test locally before and after the roll. If you get a clear difference for vm_working_set_final_size_total a few runs before and after, you can start bisecting in third_party/webrtc + building + running a few times and averaging the working set memory readouts.

Note that memory metrics for WebRTC have been turned off since mid-september and will remain off until we have been able to rewrite the telemetry tests to TbMV2.

Patrik, 

Thanks for the info. 
I just realized that the memory tests have been removed recently. 
https://codereview.chromium.org/2325153002

I am running the test on a windows machine.
But because the memory metrics were removed/disabled, we cannot verify it with the bots any more. We can only verify it on my local machine.  
 

Re #16: Yeah, you're right. Good luck with the bisecting, and remember to be really thorough. Most times I've bisected anything locally I've simply ended up with results that don't make sense, so don't feel bad if you fail :)

Indeed I cannot verify the difference between before and after the roll. 
Probably it was because the issue was only  on win-x64-dual but the machine I used for testing is x64-single. 

The VM-size increase on x64-single happened at a different CL. Perhaps we should bisect on the x64-single. But it looked strange to me that the VM increase at CLs depends on the machine type. Maybe we have two different issues. 

Patrik, 

Any suggestions on what we should proceed here?
It could not be reproduced locally.  
We can speculatively revert some commits based on my analysis in #11 but we won't be able to see the metric changes as these metrics have been removed.
Plus, I am not sure if we should revert it even if that is the cause because it is a result of a bug fix.

Ok, then there's no way forward, so we should just close it.