I've got a response from Android Galaxy S5 Perf (1) [https://uberchromegw.corp.google.com/i/chromium.perf/builders/Android%20Galaxy%20S5%20Perf%20%281%29/builds/3750]

This failure seems to be WPR related:

(ERROR) 2016-08-31 13:34:32,725 story_runner._UpdateAndCheckArchives:381  The story set archives for some stories are missing.
Someone forgot to check them in, uploaded them to the wrong cloud storage bucket, or they were deleted.
To fix this, record those stories using record_wpr.
To ignore this warning and run against live sites, pass the flag --use-live-sites.
(ERROR) 2016-08-31 13:34:32,725 story_runner._UpdateAndCheckArchives:390  stories missing archives: long_running:tools:gmail-background, long_running:tools:gmail-foreground
(INFO) 2016-08-31 13:34:32,725 exception_formatter.PrintFormattedException:19  Try printing formatted exception: None None None

Randy, any chance your CL https://codereview.chromium.org/2272993003 is related to this failure? It seems to be the only system health CL within the range...

Yes, this is definitely https://codereview.chromium.org/2272993003. I'm going to revert it.

Revert landed: https://codereview.chromium.org/2302803002/.

Randy: Please add WPR recordings to the patch and re-land it.

Ned: This demonstrates why we need SH stories on the regular CQ so badly ;-)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/50107db100baf17574adf73954af24da40532a77

commit 50107db100baf17574adf73954af24da40532a77
Author: petrcermak <petrcermak@chromium.org>
Date: Thu Sep 01 10:39:10 2016

Revert of [Telemetry] Convert long running user stories to the System Health format. (patchset #4 id:60001 of https://codereview.chromium.org/2272993003/ )

Reason for revert:
The patch broke perf bots because it doesn't contain WPR recordings.

Original issue's description:
> [Telemetry] Convert long running user stories to the System Health format.
>
> BUG= 639315 
> CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.perf:android_s5_perf_cq;master.tryserver.chromium.perf:linux_perf_cq;master.tryserver.chromium.perf:mac_retina_perf_cq;master.tryserver.chromium.perf:winx64_10_perf_cq
>
> Committed: https://crrev.com/13d57aa167188a0ea4388634fcf4589d892002ff
> Cr-Commit-Position: refs/heads/master@{#415702}

TBR=charliea@chromium.org,nednguyen@google.com,perezju@chromium.org,rnephew@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= 639315 , 643110 

Review-Url: https://codereview.chromium.org/2302803002
Cr-Commit-Position: refs/heads/master@{#415926}

[modify] https://crrev.com/50107db100baf17574adf73954af24da40532a77/tools/perf/benchmarks/system_health_smoke_test.py
[modify] https://crrev.com/50107db100baf17574adf73954af24da40532a77/tools/perf/page_sets/data/system_health_desktop.json
[modify] https://crrev.com/50107db100baf17574adf73954af24da40532a77/tools/perf/page_sets/data/system_health_mobile.json
[delete] https://crrev.com/a946e92eaa55cb6da801ada49dfdd51d786fa06d/tools/perf/page_sets/system_health/long_running_stories.py

Android Nexus 6 Perf bot (https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus6%20Perf%20%281%29) has just turned green. Hopefully it is the first of many.

It looks like the fix didn't work as planned.

https://codereview.chromium.org/2297783005 has turned the bots red again. The cause is less obvious this time, the logs don't complain about missing WPR files anymore.

long_running:tools:gmail-background fails but I can't understand why from the logs (https://uberchromegw.corp.google.com/i/chromium.perf/builders/Android%20Galaxy%20S5%20Perf%20%281%29/builds/3761/steps/system_health.memory_mobile/logs/stdio)

I will revert the CL in the meantime (sorry).

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ff39c2e3c16c91a7beff4a959c918b344286b010

commit ff39c2e3c16c91a7beff4a959c918b344286b010
Author: picksi <picksi@chromium.org>
Date: Fri Sep 02 09:38:52 2016

Revert of Reland [Telemetry] Convert long running user stories to the System Health format. (patchset #1 id:1 of https://codereview.chromium.org/2297783005/ )

Reason for revert:
The long running gmail test enabled by the CL is failing and turning all the Android perf bots red. See  crbug.com/643110 /

Original issue's description:
> Reland [Telemetry] Convert long running user stories to the System Health format.
>
> BUG= 639315 
>
> Committed: https://crrev.com/1e7e98e048508e3c158244c409830f5a4ebc0630
> Cr-Commit-Position: refs/heads/master@{#415996}

TBR=nednguyen@google.com,petrcermak@chromium.org,perezju@chromium.org,charliea@chromium.org,rnephew@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= 639315 , 643110 

Review-Url: https://codereview.chromium.org/2294913010
Cr-Commit-Position: refs/heads/master@{#416224}

[modify] https://crrev.com/ff39c2e3c16c91a7beff4a959c918b344286b010/tools/perf/benchmarks/system_health_smoke_test.py
[modify] https://crrev.com/ff39c2e3c16c91a7beff4a959c918b344286b010/tools/perf/page_sets/data/system_health_desktop.json
[delete] https://crrev.com/2fcb04a7eabd49c59a9b25edd7303973d01523b1/tools/perf/page_sets/data/system_health_desktop_028.wpr.sha1
[modify] https://crrev.com/ff39c2e3c16c91a7beff4a959c918b344286b010/tools/perf/page_sets/data/system_health_mobile.json
[delete] https://crrev.com/2fcb04a7eabd49c59a9b25edd7303973d01523b1/tools/perf/page_sets/data/system_health_mobile_042.wpr.sha1
[delete] https://crrev.com/2fcb04a7eabd49c59a9b25edd7303973d01523b1/tools/perf/page_sets/system_health/long_running_stories.py

Randy: next time you land this, can you make a patch set in which you don't disable this test in system_health_smoke_test, then verify linux_android_rel_ng is green, then disable it & land the new patchset?

Most all android bots are still red or purple on the waterfall, was the waterfall observed as green after the revert landed this week?  Should I open new issues to address the failures this week?

The redness from the long_running system health story should have passed. Any current redness is caused by a sperate issue.


It looks like the test this system health test was based on (memory.long_running_idle_gmail_background_tbmv2) has been failing consistently on android bots for quite awhile now.

I will be disabling this test on android for now and attempting to reland.


https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5%20Perf%20%282%29?numbuilds=200

The last time it passed consistently on a N5 bot was Aug 26. I'll be kicking off a RC bisect for it.

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9002249260581550688

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1f38554c02bb4e0c9de35a53ce617931d0c3da08

commit 1f38554c02bb4e0c9de35a53ce617931d0c3da08
Author: catapult-deps-roller <catapult-deps-roller@chromium.org>
Date: Wed Sep 07 01:21:29 2016

Roll src/third_party/catapult/ 6c34841e9..cb0d9d7c3 (4 commits).

https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/6c34841e91ea..cb0d9d7c3b28

$ git log 6c34841e9..cb0d9d7c3 --date=short --no-merges --format='%ad %ae %s'
2016-09-06 rnephew [BattOr] Add message to ChromeClockSyncError when Chrome fails to clock sync.
2016-09-06 alexandermont Add video playback animation expectation to user model.
2016-09-06 alexandermont Update RAIL power metric to be consistent with other power metrics.
2016-09-06 aiolos Clean up after decorators change. This should land after https://codereview.chromium.org/2314113002/

BUG= 643110 

TBR=catapult-sheriff@chromium.org

Review-Url: https://codereview.chromium.org/2312403002
Cr-Commit-Position: refs/heads/master@{#416812}

[modify] https://crrev.com/1f38554c02bb4e0c9de35a53ce617931d0c3da08/DEPS

===== BISECT JOB RESULTS =====
Status: failed


===== TESTED REVISIONS =====
Revision         Exit Code  Std Dev  N  Good?
chromium@414687  0          N/A      4  good
chromium@414702  0          N/A      4  good
chromium@414717  1          N/A      4  bad

Bisect job ran on: android_nexus5_perf_bisect
Bug ID: 643110

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --also-run-disabled-tests memory.long_running_idle_gmail_background_tbmv2
Test Metric: memory:chrome:renderer_processes:reported_by_chrome:v8:allocated_by_malloc:effective_size_avg/memory:chrome:renderer_processes:reported_by_chrome:v8:allocated_by_malloc:effective_size_avg
Relative Change: Zero to non-zero
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus5_perf_bisect/builds/4090
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9002249260581550688


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5823055962570752

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9002161171146633120

===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : DevTools: merge devtools target with devtools host, part 1
Author  : pfeldman
Commit description:
  
BUG= 627517 
TBR=

CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:closure_compilation

Review-Url: https://codereview.chromium.org/2263843002
Cr-Commit-Position: refs/heads/master@{#414707}
Commit  : a9e7ddae8f1174e71c2e999eec4813ee24c1f41a
Date    : Fri Aug 26 14:37:20 2016


===== TESTED REVISIONS =====
Revision         Exit Code  Std Dev  N  Good?
chromium@414687  0          N/A      2  good
chromium@414702  0          N/A      2  good
chromium@414706  0          N/A      2  good
chromium@414707  1          N/A      2  bad    <--
chromium@414708  1          N/A      2  bad
chromium@414710  1          N/A      2  bad
chromium@414717  1          N/A      2  bad

Bisect job ran on: android_nexus5_perf_bisect
Bug ID: 643110

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --also-run-disabled-tests memory.long_running_idle_gmail_background_tbmv2
Test Metric: memory:chrome:renderer_processes:reported_by_chrome:v8:heap:code_space:effective_size_std/memory:chrome:renderer_processes:reported_by_chrome:v8:heap:code_space:effective_size_std
Relative Change: Zero to non-zero
Score: 99.9

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus5_perf_bisect/builds/4098
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9002161171146633120


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5907494533070848

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!