All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=629329

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgyqWYpQoM


Bot(s) for this bug's original alert(s):

chromium-rel-win7-dual

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9006756028516609696

=== Auto-CCing suspected CL author lhchavez@chromium.org ===

Hi lhchavez@chromium.org, the bisect results pointed to your CL below as possibly
causing a regression. Please have a look at this info and see whether
your CL be related.


===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : arc: Use FOR_EACH_OBSERVER in InstanceHolder
Author  : lhchavez
Commit description:
  
Now that FOR_EACH_OBSERVER supports dependent types, we can use it
instead of its expansion.

BUG= 627209 
TEST=trybots

Review-Url: https://codereview.chromium.org/2147443005
Cr-Commit-Position: refs/heads/master@{#405174}
Commit  : 05b7d4f124a608289ff685341c146cb7e38f1fd6
Date    : Wed Jul 13 16:16:14 2016


===== TESTED REVISIONS =====
Revision         Mean     Std Dev    N  Good?
chromium@405157  36.8251  0.406983   5  good
chromium@405172  36.8997  0.166771   5  good
chromium@405173  36.7142  0.105625   5  good
chromium@405174  37.7332  0.260091   5  bad    <--
chromium@405176  37.8447  0.374415   5  bad
chromium@405179  37.6864  0.102325   5  bad
chromium@405185  37.8788  0.490485   5  bad
chromium@405209  37.6901  0.0983198  5  bad

Bisect job ran on: win_perf_bisect
Bug ID: 629329

Test Command: .\src\out\Release\performance_browser_tests.exe --test-launcher-print-test-stdio=always --enable-gpu
Test Metric: TabCapturePerformance/Capture
Relative Change: 2.35%
Score: 99.9

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_perf_bisect/builds/6709
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9006756028516609696


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5771527748845568

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

That change should not have affected any generated code at all :/

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9005789928013277808

Lets try another bisect.

===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : arc: Use FOR_EACH_OBSERVER in InstanceHolder
Author  : lhchavez
Commit description:
  
Now that FOR_EACH_OBSERVER supports dependent types, we can use it
instead of its expansion.

BUG= 627209 
TEST=trybots

Review-Url: https://codereview.chromium.org/2147443005
Cr-Commit-Position: refs/heads/master@{#405174}
Commit  : 05b7d4f124a608289ff685341c146cb7e38f1fd6
Date    : Wed Jul 13 16:16:14 2016


===== TESTED REVISIONS =====
Revision         Mean     Std Dev    N  Good?
chromium@405000  38.2703  0.0552996  5  good
chromium@405150  37.8741  0.207062   5  good
chromium@405167  37.8332  0.283748   5  good
chromium@405172  37.7404  0.0908711  5  good
chromium@405173  37.8868  0.200638   5  good
chromium@405174  38.5575  0.144521   5  bad    <--
chromium@405176  38.5816  0.321673   5  bad
chromium@405185  38.5636  0.124327   5  bad
chromium@405225  38.597   0.226742   5  bad
chromium@405300  39.0996  0.494786   5  bad

Bisect job ran on: win_perf_bisect
Bug ID: 629329

Test Command: .\src\out\Release\performance_browser_tests.exe --test-launcher-print-test-stdio=always --enable-gpu
Test Metric: TabCapturePerformance_webrtc/Capture
Relative Change: 2.17%
Score: 99.9

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_perf_bisect/builds/6760
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9005789928013277808


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5838690090221568

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

I'm worried that this change is still flagged. This is just replacing a macro expansion with the macro invocation. Moreover, it's Chrome OS only :(

Actually, looking at the revision log, the change right before yours (r405173) looks suspect: https://chromium.googlesource.com/chromium/src/+/351566a4e91d7f5c204c6c0881986f6bb385a04b

forshaw@: Could you PTAL?

I'm not sure why you consider mine to look suspect other than it being a Windows CL. That code should not be in a hot path of performance testing, unless the test is spawning a lot of sub-processes. If it was measuring performance over the entire process creation + webrtc testing then I'd suggest you've got a very bad test.

And even then the complex code this change would trigger on is not enabled by default, it's behind a finch flag. Further more the performance spike seems to be on the win7 bots where the only significant code (SetLowBoxToken) should _never_ activate even if the finch flag is enabled. 

Unfortunately I can't test atm, I've only got access go a Windows machine over RDP which isn't great for performance testing, and no access to Windows 7 anyway (the machine is Windows 10). But I'm very skeptical that my sandbox changes would affect the runtime performance of WebRTC.

Perf sheriff ping: reminder to follow up on possible performance issues

miu: Any ideas how we should follow up here?

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8999611409999767360

Fixit ping: forshaw, any change in your ability to test locally? miu, any ideas on next steps?

I've kicked off another bisect.

=== Auto-CCing suspected CL author lhchavez@chromium.org ===

Hi lhchavez@chromium.org, the bisect results pointed to your CL below as possibly
causing a regression. Please have a look at this info and see whether
your CL be related.


===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : arc: Use FOR_EACH_OBSERVER in InstanceHolder
Author  : lhchavez
Commit description:
  
Now that FOR_EACH_OBSERVER supports dependent types, we can use it
instead of its expansion.

BUG= 627209 
TEST=trybots

Review-Url: https://codereview.chromium.org/2147443005
Cr-Commit-Position: refs/heads/master@{#405174}
Commit  : 05b7d4f124a608289ff685341c146cb7e38f1fd6
Date    : Wed Jul 13 16:16:14 2016


===== TESTED REVISIONS =====
Revision         Mean     Std Dev    N  Good?
chromium@405157  36.7459  0.325871   5  good
chromium@405172  36.6636  0.127186   5  good
chromium@405173  36.7793  0.121059   5  good
chromium@405174  37.5359  0.0290113  5  bad    <--
chromium@405176  37.646   0.141541   5  bad
chromium@405179  37.6747  0.121716   5  bad
chromium@405185  37.6702  0.220884   5  bad
chromium@405209  37.7911  0.114122   5  bad

Bisect job ran on: win_perf_bisect
Bug ID: 629329

Test Command: .\src\out\Release\performance_browser_tests.exe --test-launcher-print-test-stdio=always --enable-gpu
Test Metric: TabCapturePerformance/Capture
Relative Change: 2.84%
Score: 99.9

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_perf_bisect/builds/6976
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8999611409999767360


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=6402540608946176

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

hmmm this really shouldn't be the bad change. I cannot see how this would be making windows performance any worse (or better, for that matter).

Looks like we missed the boat here. It would be nice to understand how the regression came about, but no bisect is confident at a single CL. So, it's probably infrastructure-related.

BTW--This issue is dwarfed by  bug 632292 .