Issue metadata
Sign in to add a comment
|
9.2%-65.2% regression in v8.runtimestats.browsing_desktop at 463583:463939 |
||||||||||||||||||||||
Issue descriptionSee the link to graphs below.
,
Apr 17 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982047940517183712
,
Apr 17 2017
=== Auto-CCing suspected CL author samans@chromium.org === Hi samans@chromium.org, the bisect results pointed to your CL, please take a look at the results. === BISECT JOB RESULTS === Perf regression found with culprit Suspected Commit Author : samans Commit : 2040988b1345e6ce738bd03ab7ccd5f4b73a956b Date : Tue Apr 11 23:58:35 2017 Subject: Use MojoCompositorFrameSink in RendererCompositorFrameSink Bisect Details Configuration: mac_10_11_perf_bisect Benchmark : v8.runtimestats.browsing_desktop Metric : GC:duration_avg/browse_media/browse_media_tumblr Change : 50.92% | 39.1448333333 -> 59.0778333333 Revision Result N chromium@463582 39.1448 +- 2.69617 6 good chromium@463761 38.1595 +- 2.71012 6 good chromium@463806 37.5863 +- 1.76321 6 good chromium@463828 39.109 +- 6.4595 9 good chromium@463839 41.4336 +- 24.6201 9 good chromium@463842 39.7322 +- 10.5463 6 good chromium@463844 42.6835 +- 20.884 6 good chromium@463845 64.6268 +- 3.93443 6 bad <-- chromium@463850 61.5225 +- 23.5189 6 bad chromium@463939 59.0778 +- 31.8078 6 bad To Run This Test src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=browse.media.tumblr v8.runtimestats.browsing_desktop Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8982047940517183712 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5802813147316224 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
Apr 18 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982012736166990240
,
Apr 18 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982012731177210624
,
Apr 18 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982012726645189712
,
Apr 18 2017
=== BISECT JOB RESULTS === Bisect was unable to run to completion The bisect was able to narrow the range, you can try running with: good_revision: 137707cfcb43771e8a3d6805c3f327c22a4f87d9 bad_revision : 2040988b1345e6ce738bd03ab7ccd5f4b73a956b If failures persist contact the team (see below) and report the error. Bisect Details Configuration: mac_10_11_perf_bisect Benchmark : v8.runtimestats.browsing_desktop_classic Metric : API:duration_avg/browse_news/browse_news_flipboard Change : 12.85% | 20.0585 -> 22.6353333333 Revision Result N chromium@463582 20.0585 +- 0.439137 6 good chromium@463761 19.9662 +- 0.471912 6 good chromium@463806 20.1952 +- 0.687108 6 good chromium@463828 20.1037 +- 0.61155 6 good chromium@463839 20.0077 +- 0.759914 6 good chromium@463842 20.316 +- 0.55729 6 good chromium@463845 22.5315 +- 0.888671 6 bad chromium@463850 22.574 +- 0.653185 6 bad chromium@463939 22.6353 +- 1.3015 6 bad To Run This Test src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=browse.news.flipboard v8.runtimestats.browsing_desktop_classic Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8982012731177210624 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5895402106126336 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
Apr 18 2017
=== Auto-CCing suspected CL author samans@chromium.org === Hi samans@chromium.org, the bisect results pointed to your CL, please take a look at the results. === BISECT JOB RESULTS === Perf regression found with culprit Suspected Commit Author : samans Commit : 2040988b1345e6ce738bd03ab7ccd5f4b73a956b Date : Tue Apr 11 23:58:35 2017 Subject: Use MojoCompositorFrameSink in RendererCompositorFrameSink Bisect Details Configuration: mac_10_11_perf_bisect Benchmark : v8.runtimestats.browsing_desktop_classic Metric : API:duration_avg/browse_news/browse_news_flipboard Change : 12.35% | 20.1933333333 -> 22.6863333333 Revision Result N chromium@463582 20.1933 +- 0.25644 6 good chromium@463761 20.1407 +- 0.551686 6 good chromium@463806 19.8822 +- 0.584287 6 good chromium@463828 20.0593 +- 0.780091 6 good chromium@463839 20.015 +- 0.551335 6 good chromium@463842 20.0005 +- 0.752549 6 good chromium@463844 20.4203 +- 0.748092 6 good chromium@463845 22.6343 +- 0.666751 6 bad <-- chromium@463850 22.5425 +- 0.597111 6 bad chromium@463939 22.6863 +- 0.905117 6 bad To Run This Test src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=browse.news.flipboard v8.runtimestats.browsing_desktop_classic Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8982012736166990240 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5774896933634048 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
Apr 18 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982001393087087568
,
Apr 18 2017
=== Auto-CCing suspected CL author samans@chromium.org === Hi samans@chromium.org, the bisect results pointed to your CL, please take a look at the results. === BISECT JOB RESULTS === Perf regression found with culprit Suspected Commit Author : samans Commit : 2040988b1345e6ce738bd03ab7ccd5f4b73a956b Date : Tue Apr 11 23:58:35 2017 Subject: Use MojoCompositorFrameSink in RendererCompositorFrameSink Bisect Details Configuration: mac_10_11_perf_bisect Benchmark : v8.runtimestats.browsing_desktop_classic Metric : API:duration_avg/browse_news/browse_news_flipboard Change : 13.39% | 20.2435 -> 22.9545 Revision Result N chromium@463582 20.2435 +- 0.519978 6 good chromium@463761 19.7652 +- 0.364866 6 good chromium@463806 20.4858 +- 0.952118 6 good chromium@463828 19.8268 +- 0.682254 6 good chromium@463839 19.9212 +- 0.543066 6 good chromium@463842 20.2373 +- 1.03412 6 good chromium@463844 20.1275 +- 0.654357 6 good chromium@463845 22.6527 +- 1.36966 6 bad <-- chromium@463850 22.8573 +- 0.659505 6 bad chromium@463939 22.9545 +- 0.801926 6 bad To Run This Test src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=browse.news.flipboard v8.runtimestats.browsing_desktop_classic Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8982012726645189712 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5870015963725824 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
Apr 18 2017
=== Auto-CCing suspected CL author samans@chromium.org === Hi samans@chromium.org, the bisect results pointed to your CL, please take a look at the results. === BISECT JOB RESULTS === Perf regression found with culprit Suspected Commit Author : samans Commit : 2040988b1345e6ce738bd03ab7ccd5f4b73a956b Date : Tue Apr 11 23:58:35 2017 Subject: Use MojoCompositorFrameSink in RendererCompositorFrameSink Bisect Details Configuration: mac_10_11_perf_bisect Benchmark : v8.runtimestats.browsing_desktop_classic Metric : API:duration_avg/browse_news/browse_news_flipboard Change : 12.65% | 20.4125 -> 22.9938333333 Revision Result N chromium@463582 20.4125 +- 0.630409 6 good chromium@463761 19.845 +- 0.883456 6 good chromium@463806 20.2095 +- 0.482425 6 good chromium@463828 20.007 +- 0.362886 6 good chromium@463839 20.2727 +- 0.675041 9 good chromium@463842 20.135 +- 0.746064 9 good chromium@463844 20.0932 +- 0.683956 9 good chromium@463845 22.1069 +- 2.84269 9 bad <-- chromium@463850 22.8382 +- 0.899617 6 bad chromium@463939 22.9938 +- 0.727447 6 bad To Run This Test src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=browse.news.flipboard v8.runtimestats.browsing_desktop_classic Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8982001393087087568 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5895402106126336 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
Apr 19 2017
,
Apr 19 2017
hablich: I don't know a lot about v8 and this particular test. Do you have any hypotheses about what could be going on? Are there any useful traces to look at?
,
Apr 20 2017
Sure. In this benchmark several important web pages (AFAIK the system health pageset) are interacted with. Please note that this only affects the desktop version of the pages in question. The benchmarks measures the time spent in various V8 components and the overall duration. The categories API and GC are also heavily influenced by Blink. No clue though why Optimize (that is V8 generating optimized code) is showing up. Our bots already collect traces right away. Checkout https://docs.google.com/presentation/d/1JBs1dYP_6IPZfAEYGZr2HcWPfvFZjV6bqe1A6NIUqt0/edit#slide=id.g2134c2c1c1_0_44 on how to access them.
,
Apr 25 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8981386695257550880
,
Apr 25 2017
hablich: Thank you for your explanation. A few more questions. 1) You said API and GC are heavily influenced by blink, but can you elaborate more on that? For example, how can blink make v8 spend more time doing garbage collection? (unless it's asking v8 to run more code?) 2) My CL doesn't touch blink. It could be that we ask blink to paint later than we used to, or maybe after painting's done we deliver the result to browser later than we used to, but other than that I'm not sure what else could be affected. Generally speaking, there is no change to renderer's main thread. Only impl thread. 3) How are these metrics reported? Are they sent to the browser? If so, can you tell me what IPC message is used? 4) Does v8 interact with the browser, and how? Ordering of v8 IPC messages with respect to compositing-related IPC messages could be affected.
,
Apr 26 2017
=== BISECT JOB RESULTS === Bisect failed for unknown reasons Please contact the team (see below) and report the error. Bisect Details Configuration: android_webview_arm64_aosp_perf_bisect Benchmark : v8.runtimestats.browsing_mobile Metric : Optimize:duration_avg/browse_media/browse_media_facebook_photos To Run This Test src/tools/perf/run_benchmark -v --browser=android-webview --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=browse.media.facebook.photos v8.runtimestats.browsing_mobile Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8981386695257550880 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5890243011543040 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
Apr 26 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8981289999657537712
,
Apr 26 2017
Re #16 1.) +jgruber regarding memory questions 3.) Those are reported via tracing 4.) Callbacks AFAIK. +cbruni for more information +nednguyen, I am wondering if this is again a case of strange bot behavior?
,
Apr 26 2017
hablich@, the bisect bots keep pointing to samans@' CL, so it's unlikely that this is a strange bot behavior. Can we just make a speculative revert of 2040988b1345e6ce738bd03ab7ccd5f4b73a956b & see if that makes the regression recover? I know a bunch of times, you can be working on area A which regresses Chrome in an area B which is super unrelated.
,
Apr 26 2017
my 2 cents: I don't trust the bisect here OR there is an unrelated issue going on with gc / optimization in v8.
,
Apr 26 2017
The revision ranges are too big to have any other good options. I vouch as well for speculative revert and see what happens in the next 2 days or so.
,
Apr 26 2017
A hypothesis: IIRC, GC happens at idle time, right? Idle time available is a function of BeginFrames, right? If BeginFrames shift in time a bit that could impact idle time and trigger GC more possibly, causing performance regressions?
,
Apr 26 2017
I created a revert and I'm running the benchmark that regressed in #3 to see how it goes. https://codereview.chromium.org/2835403004/
,
Apr 26 2017
fsamuel: Good point. If someone changes GC algorithm and GC:duration goes up, that's bad, but if some unrelated change allows v8 to run GC more often, that could be good actually.
,
Apr 26 2017
GC:duration went from 60 to 43 after revert.
,
Apr 26 2017
Not sure if you guys are aware of this other bug. Seems to be memory regression in v8 caused by the same CL. https://bugs.chromium.org/p/chromium/issues/detail?id=714007
,
Apr 26 2017
So..we're taking more time to do GC AND regressing memory? That's super weird. I guess GC is very BeginFrame timing sensitive...it would be helpful to get some insight from v8 folks. I'm trying to find v8 GC hooks but I really don't know this code at all.
,
Apr 26 2017
We're not even using mojo to send BeginFrames though, but it could be that SetNeedsBeginFrames is delivered earlier/later.
,
Apr 26 2017
Maybe it's also worth investigating what happens if we switch the BeginFrame to mojo?
,
Apr 26 2017
Sure. We know something else will regress but maybe this will get fixed.
,
Apr 26 2017
We are using mojo for DidReceiveCompositorFrameAck, right? It's possible that it's arriving out of order with respect to BeginFrame and causing weird interaction with the blink scheduler (confusing it into a bad state perhaps?).
,
Apr 27 2017
Regarding chromium-rel-mac11/v8.runtimestats.browsing_desktop/GC:duration_avg/browse_media regression. Before the change the benchmark was mostly doing 2 GCs and occasionally 3 GCs, after the change the benchmark is doing 3 GCs: https://chromeperf.appspot.com/report?sid=935c8d319a54df9dccc2e503935670341020b2ea9a83c544db84f02f2b360a16 There are two possible explanations: 1) GC timing: the change affects blink scheduler which affects GC tasks. 2) Memory leak or increased allocations in JS heap. Can the change trigger more allocations in JS heap? If not, I would ignore the regression.
,
Apr 27 2017
#33: Thank you for your input. I don't believe my CL can affect JS heap allocation. All I've done was basically switching from Chrome IPC to Mojo in RendererCompositorFrameSink, and since Mojo has a whole different system for transporting messages, it's expected that timing of BeginFrames would be affected, so the right explanation is most probably number 1.
,
Apr 28 2017
=== BISECT JOB RESULTS === Bisect failed for unknown reasons Please contact the team (see below) and report the error. Bisect Details Configuration: android_webview_arm64_aosp_perf_bisect Benchmark : v8.runtimestats.browsing_mobile Metric : Optimize:duration_avg/browse_media/browse_media_facebook_photos To Run This Test src/tools/perf/run_benchmark -v --browser=android-webview --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=browse.media.facebook.photos v8.runtimestats.browsing_mobile Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8981289999657537712 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5890243011543040 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
Apr 29 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8981008375309350560
,
Apr 29 2017
=== Auto-CCing suspected CL author hcarmona@chromium.org === Hi hcarmona@chromium.org, the bisect results pointed to your CL, please take a look at the results. === BISECT JOB RESULTS === Perf regression found with culprit Suspected Commit Author : hcarmona Commit : db210de154551f5b47357592bab4612303b7a381 Date : Wed Apr 12 00:00:20 2017 Subject: MD-Settings: A11y - Label settings dropdown menu Bisect Details Configuration: mac_pro_perf_bisect Benchmark : v8.runtimestats.browsing_desktop Metric : GC:duration_avg/browse_media/browse_media_tumblr Change : 64.77% | 34.5401666667 -> 56.9115 Revision Result N chromium@463588 34.5402 +- 6.91074 6 good chromium@463758 35.8442 +- 3.95084 6 good chromium@463843 36.6367 +- 7.90751 9 good chromium@463845 50.2722 +- 22.871 6 good chromium@463846 62.6695 +- 8.93134 6 bad <-- chromium@463849 53.5825 +- 37.999 14 bad chromium@463854 60.2403 +- 9.28539 9 bad chromium@463865 60.498 +- 12.0789 6 bad chromium@463886 59.8762 +- 8.62668 6 bad chromium@463928 56.9115 +- 5.30981 6 bad To Run This Test src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=browse.media.tumblr v8.runtimestats.browsing_desktop Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8981008375309350560 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5890243011543040 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
Apr 29 2017
This bisect is probably wrong. You can see at 463845 (my CL) the metric goes up to 50, but the bisect somehow blames the next revision.
,
Apr 30 2017
Issue 715836 has been merged into this issue.
,
Apr 30 2017
Issue 715774 has been merged into this issue.
,
Aug 10 2017
,
Aug 10 2017
Marking as WontFix per discussion that we had in this thread. |
|||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||
Comment 1 by hablich@chromium.org
, Apr 17 2017