All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=755445

(For debugging:) Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?sid=06f04e36f86d3d388f042365ce1f225584fd18ae3569b846b80e6dc90b86c249


Bot(s) for this bug's original alert(s):

linux-release

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8971229512832223744

Taking a look at this older rasterize_and_record_micro regression on Linux that doesn't seem to have recovered.

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8971229218891997584

Relevant graphs: https://chromeperf.appspot.com/report?sid=ad086f8b4cbe73bab24ca4aa824c566976192f2cca0b9a29a656ca5ab6d125d9&start_rev=432787&end_rev=440556

=== BISECT JOB RESULTS ===
NO Perf regression found

Bisect Details
  Configuration: linux_perf_bisect
  Benchmark    : rasterize_and_record_micro.top_25
  Metric       : rasterize_time/Blogger

Revision             Result                   N
chromium@433543      1.5079 +- 0.0518827      21      good
chromium@434229      1.5041 +- 0.0562478      21      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=Blogger rasterize_and_record_micro.top_25

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8971229218891997584


For feedback, file a bug with component Speed>Bisection

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8971217692925372464

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8971217680463369632

=== BISECT JOB RESULTS ===
NO Perf regression found

Bisect Details
  Configuration: linux_perf_bisect
  Benchmark    : rasterize_and_record_micro.top_25
  Metric       : rasterize_time/Wordpress

Revision             Result                    N
chromium@433543      1.61395 +- 0.0429296      21      good
chromium@434229      1.6239 +- 0.0764186       21      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=Wordpress rasterize_and_record_micro.top_25

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8971217692925372464


For feedback, file a bug with component Speed>Bisection

=== BISECT JOB RESULTS ===
Perf regression found but unable to narrow commit range

Build failures prevented the bisect from narrowing the range further.


Bisect Details
  Configuration: linux_perf_bisect
  Benchmark    : rasterize_and_record_micro.top_25
  Metric       : rasterize_time/Pinterest
  Change       : 0.77% | 2.72307142857 -> 2.74407142857

Suspected Commit Range
  3 commits in range
  https://chromium.googlesource.com/chromium/src/+log/9077bce9ccad2863d39424b04b9fd490e3f3ba4a..4e835c3b0ff28e8214bb41cb1995d9fb33b7fbe7


Revision             Result                    N
chromium@433543      2.72307 +- 0.0547625      14       good
chromium@433715      2.72693 +- 0.0620236      14       good
chromium@433800      2.72971 +- 0.0411929      14       good
chromium@433844      2.72122 +- 0.032887       9        good
chromium@433865      2.72557 +- 0.0796331      14       good
chromium@433875      2.7261 +- 0.0636695       21       good
chromium@433881      2.72017 +- 0.0218365      6        good
chromium@433884      2.72493 +- 0.0905369      14       good
chromium@433885      ---                       ---      build failure
chromium@433886      ---                       ---      build failure
chromium@433887      2.74971 +- 0.0811471      14       bad
chromium@434229      2.74407 +- 0.0663998      14       bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=Pinterest rasterize_and_record_micro.top_25

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8971217680463369632


For feedback, file a bug with component Speed>Bisection

=== BISECT JOB RESULTS ===
Bisect was unable to run to completion

Error: INFRA_FAILURE

The bisect was able to narrow the range, you can try running with:
  good_revision: 2459701934af4c8297ee26d351a1fbf9596c6bd5
  bad_revision : 92e90e89cf3af95e0158192205896c6bc4a52d14

If failures persist contact the team (see below) and report the error.


Bisect Details
  Configuration: linux_perf_bisect
  Benchmark    : rasterize_and_record_micro.top_25
  Metric       : rasterize_time/rasterize_time

Revision             Result                    N
chromium@433543      2.48025 +- 0.0189201      14      good
chromium@433886      2.48441 +- 0.0184515      14      good
chromium@434059      2.48875 +- 0.0631051      14      good
chromium@434080      2.48501 +- 0.0133422      14      good
chromium@434101      2.49376 +- 0.0218251      14      bad
chromium@434144      2.4914 +- 0.0367327       14      bad
chromium@434229      2.49278 +- 0.0392169      14      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests rasterize_and_record_micro.top_25

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8971229512832223744


For feedback, file a bug with component Speed>Bisection

vmiura, vmpstr, wkorman: bisect can't reproduce this. we did have some hardware fixes on linux, but I'm not sure if they overlap with the timeline. What should we do with this bug?

When were the hardware change(s) roughly?

The original regression was around Nov 2016. Change log:

https://chromium.googlesource.com/chromium/src/+log/2947703c1aad639f2cce9d3b459c04c58f9cbefd%5E..24489dbd083c57720618e2c84830cd94fd0bf877?pretty=fuller&n=1000

Six changes affecting cc as potential candidates on manual review:

https://chromium.googlesource.com/chromium/src/+/6182cb1b36dacbcb4cfef9ecb77fe1ee9dbd07b5
Remove redundant signals_check_notifier_ schedule.

https://chromium.googlesource.com/chromium/src/+/e3d2888a771050099facf2b18c3d72b4922c9581
Display Compositor: Towards allocating SurfaceIds in the client

https://chromium.googlesource.com/chromium/src/+/4af1869e97cdf585994c422c22423f6d6fcf68ce
Introduce Display Compositor mojo interface. Use InProcessContextProvider.

https://chromium.googlesource.com/chromium/src/+/3e64b9d5c446a5a545402f356fd8e19f7df87e39
cc: Remove indirection in calls to RenderSurfaceImpl::AppendQuads

https://chromium.googlesource.com/chromium/src/+/cc9c70f3fd05091ebb4966ee2f70efed0532fd3e
stop using SkXfermode -- use SkBlendMode instead

https://chromium.googlesource.com/chromium/src/+/e27fa89ee10d287ece8b775864fa609e3a78e062
Revert "Revert "Change call-sites now that SkCanvas is not ref-counted""

Oh, I didn't look at the graph, hardware changes were early summer, so probably unrelated.

Sort of blocked on  issue 767987  being resolved though I'll look at other result output formats.

Discussed briefly w/ enne@ in person. Syncing back to r433543 locally to try bisecting skia deps-rolls. Two possible example changes:

https://skia-review.googlesource.com/c/skia/+/5100
https://skia-review.googlesource.com/c/skia/+/5107

I can't repro locally. I synced back to the involved changes, so that I'd be using the same benchmark and page set, and bisected across most of the skia deps-rolls running just the Wordpress story. I then checked the before-regression r433543 and last-within-regression r434229 commits.

Results below, but they don't show anything like the 33% regression associated with this issue. For the full page set I also looked at individual stories. Benchmark results.html for r434229 and r433543 are also attached for posterity.

At this point unless there are other ideas I think we should WontFix. I know these benchmarks are hard to run correctly/effectively locally, and we already haven't identified a breaking change with the bisect perf bots.

= WordPress only

commit  | rasterize_time

r434229 | 3.263 / 3.121
r434215 | 3.104
r434211 | 3.108
r434208 | 3.101
r434206 | 3.085 / 3.063 / 3.094
r434021 | 3.09
r433846 | 3.09
r433543 | 3.30 / 3.00

= Full Page Set

commit  | rasterize_time

r434229 | 4.494
r433543 | 4.49

Deleted: results.tar.gz 292 KB

results.tar.gz 292 KB Download

Please reopen if other suggestions or things to look into.

That sounds good to me.  Thanks for the investigation!