All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=903204

(For debugging:) Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?sid=61ebbcae0219325d41996d7fa112ee22311c1dd71118db12ec45dcacfbf3cdb7


Bot(s) for this bug's original alert(s):

Android Nexus6 WebView Perf

rendering.mobile - Benchmark documentation link:
  https://bit.ly/rendering-benchmarks

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/15a967c7e40000

📍 Found a significant difference after 1 commit.
https://pinpoint-dot-chromeperf.appspot.com/job/15a967c7e40000

Telemetry: finally move rendering to TBMv2 by chiniforooshan@chromium.org
https://chromium.googlesource.com/chromium/src/+/05af02c8afe6a0fdc3d5e32ff88632fa15f5c567
tasks_per_frame_total_all: 14.2 → 14.7 (+0.504)

Understanding performance regressions:
  http://g.co/ChromePerformanceRegressions

Benchmark documentation link:
  https://bit.ly/rendering-benchmarks

Although generally we do expect changes in tasks_per_frame_* metrics from my CL, there is something else going on here. In this case, renderer compositor frame times should be similar to display compositor frame times and so the 14.2 -> 14.7 difference is strange.

hmmm... there is no avg_surface_fps or frame_lengths after my CL:

https://00e9e64bac0471207635a887df23c2e1f60216de7f3225e3ba-apidata.googleusercontent.com/download/storage/v1/b/results2-public/o/15a967c7e40000.html?qk=AD5uMEtHXkAtjFRfoQ4Vwd295PvodtTqWc7n2JRoPO_DeVw6J__n4-_KmctYmvPWy_7zhb0xBOZQvAWG69_21_DN2_hqQv3Z8lxvQptYSyYFtJYk-tI-Cvga0AeAyy3urIYieS1qfqI_3jki_kn2qaG-a3g2VJMfG34U_tqz00tQO8VG54w9VaV8SIusatNWSmMMl8zTl7PlzPlg6hoUYmNbUo17M3lFPSwiSX-PoDGhgatmFfOEmw2ZnVJOhE_70BKrvvI0xagXDhivuKG_hsciJs8jV6YbtFZGuWaRrPe7FRtTShg_-bj01oj8x-tooMi0ZZ2uUmOl078QiRBxMOVJdU9qD8WEwIEac2ITzxONppICIWCNotHDxAxSLWYOk_UzQqGy6BqTJcT95GP9K9_Khcg0Eh5G6rUU5Zo9SkyPAJjj4N5tHG09K1TCVr5JrxlCEDzSIVZpbGotaM7zwvx5EQm1s4yGTTtZm3Oxwn-sXllwtX1rar8z0a6nGPiURqmQPtBfqXdbnAsF2mB3D4K7Ice4CZ9O7RqBFZ_0baXi8nxEkvVpXGgjdE227tdG0OLOXbYGlJKbAaDDZ8kiu3cVahdRuJSonQDR2maslYS-NdXG64I_dhEaHlYfs0rVAmhPNO3g0A_REfy2NGSR9WO9Zu5L-GJNn_omXf3BzOkordEjM6jl4F4mfkOhBs9HbFvibW8m5c_6cN3m3kRsh94BexIICko2ws6VMfInR3b3wfbGa7I3MuU8FuQnsKAfYkic2vQDotlwmXzy6GEGRgsW2AmAWQjhFdmG5xe4eTWAT6FWPfoxre4

This can be seen in the perf dash, too:
https://chromeperf.appspot.com/report?sid=a0b550656f0888074d4617e9f47af80e8f0209c79ec31315f54fec2405b7e338

So, surface flinger data is missing and we are falling back to timestamps coming from the browser process for all frame related metrics, which can explain the change.

I'll try to understand why and send a fix.

Fix: crrev.com/c/1327150

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4726135f893e414928888cdfb2c36f852d0a0ff7

commit 4726135f893e414928888cdfb2c36f852d0a0ff7
Author: Ehsan Chiniforooshan <chiniforooshan@chromium.org>
Date: Thu Nov 08 20:13:27 2018

Telemetry: do display tracing in rendering.mobile

Bug: 903204
Change-Id: I7d2cbff9184eb9f0fa93bfe14708ca719f145e7a
Reviewed-on: https://chromium-review.googlesource.com/c/1327150
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Ehsan Chiniforooshan <chiniforooshan@chromium.org>
Cr-Commit-Position: refs/heads/master@{#606580}
[modify] https://crrev.com/4726135f893e414928888cdfb2c36f852d0a0ff7/tools/perf/benchmarks/rendering.py

The "Android Nexus6 WebView Perf" is failing and so the graphs are not begin updated. The error I see in logs is:

  File "/b/swarming/w/ir/third_party/catapult/common/py_utils/py_utils/atexit_with_log.py", line 13, in _WrappedFn
    function(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/chrome_tracing_agent.py", line 297, in _RemoveTraceConfigFile
    self._trace_config_file, force=True, rename=True, as_root=True)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper
    retry_if_func=retry_if_func)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 158, in Run
    error_log_func=error_log_func)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 198, in JoinAll
    self._JoinAll(watcher, timeout)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 170, in _JoinAll
    thread.ReraiseIfException()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 93, in run
    self._ret = self._func(*self._args, **self._kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 151, in <lambda>
    child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs),
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1873, in RemovePath
    self.RunShellCommand(args, as_root=as_root, check_return=True)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper
    return impl()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1159, in RunShellCommand
    output = handle_large_output(cmd, large_output)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1128, in handle_large_output
    return handle_large_command(cmd)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1110, in handle_large_command
    return handle_check_return(cmd)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1101, in handle_check_return
    return run(cmd)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1097, in run
    return self.adb.Shell(cmd)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 546, in Shell
    command, output, status=status, device_serial=self._device_serial)
AdbShellCommandFailedError: (device: ZX1G22Q2S7) shell command run via adb failed on the device:
  command: rm -f /data/local/chrome-trace-config.json
  exit status: 1
  output:
  - rm: /data/local/chrome-trace-config.json: Permission denied

(ERROR) 2018-11-13 01:54:57,895 forwarder._UnmapDevicePortLocked:334  `/b/swarming/w/ir/third_party/catapult/devil/bin/deps/linux2/x86_64/forwarder_host --adb=/b/swarming/w/ir/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb --serial-id=ZX1G22Q2S7 --unmap 39766` exited with 1:
[1113/015457.895379:ERROR:host_forwarder_main.cc(150)] ERROR: could not unmap port.
ERROR: Existing controllers:

And this error, too:

Exception raised when cleaning story run: 

Traceback (most recent call last):
  _RunStoryAndProcessErrorIfNeeded at /b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/story_runner.py:153
    test.DidRunStory(state.platform, results)
  DidRunStory at /b/swarming/w/ir/third_party/catapult/telemetry/telemetry/web_perf/timeline_based_measurement.py:296
    trace_result, _ = platform.tracing_controller.StopTracing()
  StopTracing at /b/swarming/w/ir/third_party/catapult/telemetry/telemetry/core/tracing_controller.py:53
    return self._tracing_controller_backend.StopTracing()
  StopTracing at /b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py:157
    '\n'.join(raised_exception_messages))
TracingException: Exceptions raised when trying to stop tracing:
Traceback (most recent call last):
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py", line 135, in StopTracing
    agent.StopAgentTracing()
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/chrome_tracing_agent.py", line 202, in StopAgentTracing
    self._RemoveTraceConfigFile()
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/chrome_tracing_agent.py", line 297, in _RemoveTraceConfigFile
    self._trace_config_file, force=True, rename=True, as_root=True)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper
    retry_if_func=retry_if_func)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 158, in Run
    error_log_func=error_log_func)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 198, in JoinAll
    self._JoinAll(watcher, timeout)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 170, in _JoinAll
    thread.ReraiseIfException()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 93, in run
    self._ret = self._func(*self._args, **self._kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 151, in <lambda>
    child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs),
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1869, in RemovePath
    args.append(device_path if not rename else _RenamePath(device_path))
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1858, in _RenamePath
    ['mv', path, dest], as_root=as_root, check_return=True)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper
    return impl()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1159, in RunShellCommand
    output = handle_large_output(cmd, large_output)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1128, in handle_large_output
    return handle_large_command(cmd)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1110, in handle_large_command
    return handle_check_return(cmd)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1101, in handle_check_return
    return run(cmd)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1097, in run
    return self.adb.Shell(cmd)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 530, in Shell
    output = self._RunDeviceAdbCmd(args, timeout, retries, check_error=False)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 313, in _RunDeviceAdbCmd
    check_error=check_error)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper
    return impl()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 290, in _RunAdbCmd
    raise device_errors.DeviceUnreachableError(device_serial)
DeviceUnreachableError: ZX1G22Q2S7

Traceback (most recent call last):
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py", line 145, in StopTracing
    agent.CollectAgentTraceData(builder)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/display_tracing_agent.py", line 30, in CollectAgentTraceData
    surface_flinger_trace_data = self._platform_backend.StopDisplayTracing()
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/platform/android_platform_backend.py", line 190, in StopDisplayTracing
    refresh_period, timestamps = self._surface_stats_collector.Stop()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/perf/surface_stats_collector.py", line 47, in Stop
    (refresh_period, timestamps) = self._GetDataFromThread()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/perf/surface_stats_collector.py", line 95, in _GetDataFromThread
    raise ret
DeviceUnreachableError: ZX1G22Q2S7

+dtu +nednguyen in case they have any ideas about the error in c#9?

Those are just device error, normally we contact lab team and ask them to fix/replace such device.

Thank you!

+Brad/John for the device failures

 Issue 906552  has been merged into this issue.

The bot is still failing:

https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/Android%20Nexus6%20WebView%20Perf

Which bot has the failing N6 device?  Any chance you would like us to upgrade to a newer and more reliable device?

ZX1G22Q2S7 is on build204-b7. I just checked and the N6 fleet is in pretty bad shape physically. We need to pull the following devices for dangerous battery issues:

build202-b7--device{2,5}
build203-b7--device{1,2,4,5,6}
build204-b7--device{2,3,6}

Re:ZX1G22Q2S7 going to re-flash that device now (it's battery looks ok)

+cc vhang for any repair/replace options we might have for the devices that were pulled.

the best replacement option is to pick another newer device IMO.  Pixel 2s?

Indeed, the intended replacement for this N6 bot has Pixel 2s and is currently in fyi waterfall:
https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/android-pixel2_webview-perf

however it's not yet ready.

Any idea what are the recent purple infra failures on it? e.g.:
https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/android-pixel2_webview-perf/777

Filed issue 907852 for the Infra Falure on those new webview bots.