I will own this bug since it's blocking effort to reduce telemetry_perf_unittest flake on CQ

https://datastudio.google.com/c/reporting/12dYEpcepJ5_6ZOhprbd5GpDNooiUJONV/page/AYfX

Any update on this? telemetry_eprf_unittests are once again the top source of flaky failures, and I still have a lot of trouble figuring out what's happening in the logs.

Deleted: Screen Shot 2018-10-10 at 1.01.31 PM.png 78.3 KB

	Screen Shot 2018-10-10 at 1.01.31 PM.png 78.3 KB View Download

Can you give me the log of the failure? Fixing this will require some substantial change, and I am swarmed digging into webkit layout tests these days

Raw swarming output:
https://chromium-swarm.appspot.com/task?id=40768a1e68c19e10&refresh=10&show_raw=1

I've already dug through and filed:
https://bugs.chromium.org/p/chromium/issues/detail?id=894092

> Fixing this will require some substantial change
I've already linked the code that needs to be changed in the opening comment. This is 2-3, relatively small CLs. If you don't want to do it, then I will.

Ah yes. Feel free to go ahead & steal this bug from me :-)

The place to plumb the typ flag in Telemetry unittest runner is https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/testing/run_tests.py?rcl=86bdcbf37ffcfaa2c9064a14e24f6195a56baefe&l=170

For gpu Telemetry test runner, the plumbing of flag is: https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/testing/run_browser_tests.py?rcl=86bdcbf37ffcfaa2c9064a14e24f6195a56baefe&l=299

The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/cd3378c32017c99062cd467a3ff601e0b57f0c23

commit cd3378c32017c99062cd467a3ff601e0b57f0c23
Author: erikchen <erikchen@chromium.org>
Date: Wed Oct 10 22:02:32 2018

Reduce verbosity of typ output for catapult tests.

Verbosity was increased back in 2016 to help debug a telemetry issue, which has
since been fixed.

Increased verbosity makes it difficult for non-experts to read the log to
understand or even find the source of a test failure. This CL reduces the
verbosity back to standard levels, which only emits output for failed test runs.

Bug: chromium:891898,  chromium:457446 
Change-Id: I3bf0b115eeca011148ae093f287ead3edaeb6c9c
NOTRY=true
NOPRESUBMIT=true
Reviewed-on: https://chromium-review.googlesource.com/c/1274286
Commit-Queue: Ned Nguyen <nednguyen@google.com>
Auto-Submit: Erik Chen <erikchen@chromium.org>
Reviewed-by: Ned Nguyen <nednguyen@google.com>

[modify] https://crrev.com/cd3378c32017c99062cd467a3ff601e0b57f0c23/telemetry/telemetry/testing/run_tests.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/11e84b285a9077d5624fd1f49e7412fb35a974c0

commit 11e84b285a9077d5624fd1f49e7412fb35a974c0
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Thu Oct 11 02:37:21 2018

Roll src/third_party/catapult fc2aa094eda4..cd3378c32017 (2 commits)

https://chromium.googlesource.com/catapult.git/+log/fc2aa094eda4..cd3378c32017


git log fc2aa094eda4..cd3378c32017 --date=short --no-merges --format='%ad %ae %s'
2018-10-10 erikchen@chromium.org Reduce verbosity of typ output for catapult tests.
2018-10-10 benjhayden@chromium.org Add service worker for v2spa.


Created with:
  gclient setdep -r src/third_party/catapult@cd3378c32017

The AutoRoll server is located here: https://autoroll.skia.org/r/catapult-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG=chromium:891898, chromium:457446 
TBR=sullivan@chromium.org

Change-Id: I13e3b982e92c085cb9cf835b176bed84b6df2f57
Reviewed-on: https://chromium-review.googlesource.com/c/1274970
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#598641}
[modify] https://crrev.com/11e84b285a9077d5624fd1f49e7412fb35a974c0/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/7c50ac5ddf5b23ed1de01b542eb8c3adf7839665

commit 7c50ac5ddf5b23ed1de01b542eb8c3adf7839665
Author: Ben Pastene <bpastene@chromium.org>
Date: Thu Oct 18 18:27:58 2018

Reenable telemetry test timings.

This was removed in https://chromium-review.googlesource.com/c/catapult/+/1274286,
making it difficult to debug telemetry_unittests timeouts.

Bug: chromium:891898
Change-Id: I229a3a79363117386b60b1b75af677c8856b394d
Reviewed-on: https://chromium-review.googlesource.com/c/1289229
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Ben Pastene <bpastene@chromium.org>

[modify] https://crrev.com/7c50ac5ddf5b23ed1de01b542eb8c3adf7839665/telemetry/telemetry/testing/run_tests.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/24d4abac024c96a27050d7ba6f86e8efe3bc8a99

commit 24d4abac024c96a27050d7ba6f86e8efe3bc8a99
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Thu Oct 18 21:20:33 2018

Roll src/third_party/catapult 1922eb00bbd9..7c50ac5ddf5b (1 commits)

https://chromium.googlesource.com/catapult.git/+log/1922eb00bbd9..7c50ac5ddf5b


git log 1922eb00bbd9..7c50ac5ddf5b --date=short --no-merges --format='%ad %ae %s'
2018-10-18 bpastene@chromium.org Reenable telemetry test timings.


Created with:
  gclient setdep -r src/third_party/catapult@7c50ac5ddf5b

The AutoRoll server is located here: https://autoroll.skia.org/r/catapult-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG=chromium:891898
TBR=sullivan@chromium.org

Change-Id: Ib547b3179bf070844b75250f963ee7b950d1c221
Reviewed-on: https://chromium-review.googlesource.com/c/1289549
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#600913}
[modify] https://crrev.com/24d4abac024c96a27050d7ba6f86e8efe3bc8a99/DEPS

Right now, INFO logs are still swamping the output file, e.g.:

https://chromium-swarm.appspot.com/task?id=40c0b002c464b310&refresh=10&show_raw=1

In the example above, I believe that there's an actual bug where the scroll-end intervention [which ignores touchend events] is causing a timeout in a telemetry user story. https://bugs.chromium.org/p/chromium/issues/detail?id=898844

However, the large amount of extraneous information has made this very difficult to spot.

I'm going to continue to clean up the log files so we can easily figure out what's going wrong.

The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/a45184bee57682468bbae8ca321d8c0abc1bbac4

commit a45184bee57682468bbae8ca321d8c0abc1bbac4
Author: erikchen <erikchen@chromium.org>
Date: Mon Oct 29 16:53:30 2018

Reduce verbosity of logging for ADB commands.

By default, every single ADB command was being emitted to the swarming log. This
makes it very difficult to read the swarming log and figure out what's going on.
This CL reduces the verbosity to debug, which prevents the commands from making
it to the swarming log.

Bug: chromium:891898
Change-Id: I926dc9fda0b23fac132ce3ec975e12f37f4a8893
Reviewed-on: https://chromium-review.googlesource.com/c/1300153
Reviewed-by: John Budorick <jbudorick@chromium.org>
Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org>
Commit-Queue: Erik Chen <erikchen@chromium.org>

[modify] https://crrev.com/a45184bee57682468bbae8ca321d8c0abc1bbac4/devil/devil/utils/cmd_helper.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b3b7012b37e94bc9bf3c35a2a88a71ded0e174c9

commit b3b7012b37e94bc9bf3c35a2a88a71ded0e174c9
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Mon Oct 29 20:22:59 2018

Roll src/third_party/catapult 625dca84765d..25da2e8be18d (2 commits)

https://chromium.googlesource.com/catapult.git/+log/625dca84765d..25da2e8be18d


git log 625dca84765d..25da2e8be18d --date=short --no-merges --format='%ad %ae %s'
2018-10-29 dtu@chromium.org [pinpoint] Temporarily remove auth from gerrit_service.GetChange().
2018-10-29 erikchen@chromium.org Reduce verbosity of logging for ADB commands.


Created with:
  gclient setdep -r src/third_party/catapult@25da2e8be18d

The AutoRoll server is located here: https://autoroll.skia.org/r/catapult-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG= chromium:899427 ,chromium:891898
TBR=sullivan@chromium.org

Change-Id: I78a868219581286ce001a19c20005054e6a50c64
Reviewed-on: https://chromium-review.googlesource.com/c/1305893
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#603604}
[modify] https://crrev.com/b3b7012b37e94bc9bf3c35a2a88a71ded0e174c9/DEPS

This issue has an owner, a component and a priority, but is still listed as untriaged or unconfirmed. By definition, this bug is triaged. Changing status to "assigned". Please reach out to me if you disagree with how I've done this.