New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 769710 link

Starred by 3 users

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug-Regression

Blocked on:
issue 768384



Sign in to add a comment

9.7%-327% regression in v8.runtimestats.browsing_mobile at 499617:500579

Project Member Reported by hablich@chromium.org, Sep 28 2017

Issue description

See the link to graphs below.
 
Project Member

Comment 1 by 42576172...@developer.gserviceaccount.com, Sep 28 2017

All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=769710

(For debugging:) Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?sid=9de6bf410ab3fd9a1799dde071ebdd825568f64be09514698a4e7e04ecd60a25


Bot(s) for this bug's original alert(s):

android-nexus5
android-nexus5X
android-nexus7v2
Project Member

Comment 3 by 42576172...@developer.gserviceaccount.com, Sep 29 2017


=== BISECT JOB RESULTS ===
Perf regression found but unable to narrow commit range

Build failures prevented the bisect from narrowing the range further.


Bisect Details
  Configuration: android_nexus5_perf_bisect
  Benchmark    : v8.runtimestats.browsing_mobile
  Metric       : Parse-Background:duration_avg/browse_social/browse_social_twitter
  Change       : 69.75% | 39.4374285714 -> 66.9444285714

Suspected Commit Range
  8 commits in range
  Unknown depot, please contact team to have this added.


Revision                                 Result                  N
chromium@499810                          39.4374 +- 117.506      14       good
chromium@500195                          34.9155 +- 103.785      14       good
chromium@500292                          37.5826 +- 151.646      21       good
chromium@500340                          35.3499 +- 140.749      14       good
chromium@500343                          40.1308 +- 128.714      14       good
chromium@500344                          55.4914 +- 94.874       14       good
chromium@500344,catapult@c5a474f50f      17.0195 +- 59.0271      6        good
chromium@500344,catapult@724bb776bc      ---                     ---      build failure
chromium@500344,catapult@80c47581b1      ---                     ---      build failure
chromium@500344,catapult@fc8c846a47      ---                     ---      build failure
chromium@500344,catapult@4f611510b9      ---                     ---      build failure
chromium@500344,catapult@25e5c12416      ---                     ---      build failure
chromium@500344,catapult@53e727fc21      ---                     ---      build failure
chromium@500344,catapult@14a576c888      ---                     ---      build failure
chromium@500344,catapult@442f46edf7      66.1274 +- 27.5778      14       bad
chromium@500345                          67.0887 +- 43.7986      20       bad
chromium@500346                          68.6531 +- 49.5231      21       bad
chromium@500353                          70.5243 +- 79.6355      21       bad
chromium@500364                          70.6189 +- 37.2148      14       bad
chromium@500387                          70.9733 +- 35.3583      14       bad
chromium@500579                          66.9444 +- 32.0269      14       bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=browse.social.twitter v8.runtimestats.browsing_mobile

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8967203275170163264


For feedback, file a bug with component Speed>Bisection
Cc: simonhatch@chromium.org
Owner: sullivan@chromium.org
Status: Assigned (was: Untriaged)
Seems like some broken catapult roll.
Owner: xunji...@chromium.org
This is the catapult roll:

Roll src/third_party/catapult/ c9667ecd2..29f450ae4 (16 commits)

https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/c9667ecd29cb..29f450ae4dcb

$ git log c9667ecd2..29f450ae4 --date=short --no-merges --format='%ad %ae %s'
2017-09-07 littlecvr [Telemetry] Add ChromeOS to desktop platform list
2017-09-07 dtu [pinpoint] Separate Execution exceptions from result_values.
2017-09-07 lalitm Update memtrack binary version for ARMv7 devices
2017-09-07 nednguyen Revert of Smoke test for heap profiler. (patchset #2 id:20001 of https://codereview.chromium.org/3010173002/ )
2017-09-06 rnephew [Telemetry] Fully get rid of PermanentlyDisableBenchmark.
2017-09-06 benjhayden Add traceUrls to CSVs.
2017-09-06 benjhayden Format traceUrls in generic-set-span.
2017-09-06 benjhayden Remove incorrect test testGet_WithFinish_LabelsBugWithLowestMilestonePossible
2017-09-06 benjhayden Produce all statistics in CSVBuilder.
2017-09-06 sullivan Add CORS headers for whitelisted origins.
2017-09-06 xunjieli [wpr-go] Update comment in telemtry/bin/update_wpr_go_binary
2017-09-06 simonhatch Dashboard - Add calls to graph_revisions and find_anomalies in add_histograms_queue
2017-09-06 loloangela Fix errors related to invalid-name pt. 10
2017-09-06 dtu [pinpoint] Limit executions to one test run each + device sharding.
2017-09-06 xunjieli [Telemetry] Use --ignore-certificate-errors-spki-list to bypass cert errors
2017-09-06 kraynov Smoke test for heap profiler.


Helen, do you think this could be related to "[Telemetry] Use --ignore-certificate-errors-spki-list to bypass cert errors" (https://chromiumcodereview.appspot.com/3003143002)?

It's unlikely due to --ignore-certificate-errors-spki-list change. "--ignore-certificate-errors-spki-list" is supposed to reduce the time we take for cert verification. Some loading metrics (timeToFirstContentfulPaint) regressed after WprGo migration because WprGo introduced more SSL certs. --ignore-certificate-errors-spki-list helped with that. 

The "Parse-Background:duration_avg/browse_social/browse_social_twitter" metric 
seems to be very noisy around the time when this alert is filed. For example, there was a huge decrease at r489985. The alert was triggered when the metric climbed back. 




Screenshot from 2017-09-29 10:03:39.png
23.3 KB View Download
Owner: ----
Status: Untriaged (was: Assigned)
Cc: xunji...@chromium.org
Owner: sullivan@chromium.org
Status: Assigned (was: Untriaged)
Assigning to sullivan@ for further triage because it seems to be a problem with catapult?
Kicking off another bisect on a clearer metric. Simon, any ideas how the catapult roll could possibly cause a build failure? Nothing in there should be built into chrome.
Blockedon: 768384
Yeah that's not build failure at all, that's totally wrong. What seems to be happening is that it can't decide if the results are good are bad in that range.

Blocking on  crbug.com/768384  for the output.
Project Member

Comment 12 by 42576172...@developer.gserviceaccount.com, Oct 10 2017

Cc: sunn...@chromium.org
Owner: sunn...@chromium.org

=== Auto-CCing suspected CL author sunnyps@chromium.org ===

Hi sunnyps@chromium.org, the bisect results pointed to your CL, please take a look at the
results.


=== BISECT JOB RESULTS ===
Perf regression found with culprit

Suspected Commit
  Author : Sunny Sachanandani
  Commit : 2eb3c8ac7ffc6840dc07f5e8312576e107eece23
  Date   : Wed Sep 06 02:01:21 2017
  Subject: cc: Flush after every tile task set completes.

Bisect Details
  Configuration: android_nexus5X_perf_bisect
  Benchmark    : v8.runtimestats.browsing_mobile
  Metric       : Optimize:duration_avg/browse_tech/browse_tech_discourse_infinite_scroll
  Change       : 31.20% | 143.777166667 -> 188.639833333

Revision             Result                  N
chromium@499616      143.777 +- 24.0137      6      good
chromium@499747      150.603 +- 22.1316      6      good
chromium@499812      156.189 +- 24.9525      5      good
chromium@499829      156.087 +- 28.4887      6      good
chromium@499837      152.723 +- 16.4984      6      good
chromium@499841      149.459 +- 15.1239      6      good
chromium@499843      152.976 +- 25.1839      6      good
chromium@499844      147.542 +- 26.7357      6      good
chromium@499845      198.654 +- 13.4137      6      bad       <--
chromium@499877      188.64 +- 24.3733       6      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=browse.tech.discourse.infinite.scroll v8.runtimestats.browsing_mobile

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8966117537284670800


For feedback, file a bug with component Speed>Bisection
Project Member

Comment 13 by 42576172...@developer.gserviceaccount.com, Oct 18 2017

Cc: ericrk@chromium.org
 Issue 776147  has been merged into this issue.

Sign in to add a comment