Temporarily assign to Ethan to disable these test on nexus-5x. I will investigate the adb cert failure later.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e89802a4b539562b96d004ae22ab0af163ba08a1

commit e89802a4b539562b96d004ae22ab0af163ba08a1
Author: eakuefner <eakuefner@chromium.org>
Date: Thu Mar 10 15:37:11 2016

[Telemetry] Disable some tests on Nexus 5X reference

BUG= 593152 
CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:winx64_10_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq;tryserver.chromium.perf:linux_perf_cq

Review URL: https://codereview.chromium.org/1776473008

Cr-Commit-Position: refs/heads/master@{#380412}

[modify] https://crrev.com/e89802a4b539562b96d004ae22ab0af163ba08a1/tools/perf/benchmarks/blink_style.py
[modify] https://crrev.com/e89802a4b539562b96d004ae22ab0af163ba08a1/tools/perf/benchmarks/page_cycler.py
[modify] https://crrev.com/e89802a4b539562b96d004ae22ab0af163ba08a1/tools/perf/benchmarks/power.py
[modify] https://crrev.com/e89802a4b539562b96d004ae22ab0af163ba08a1/tools/perf/benchmarks/smoothness.py
[modify] https://crrev.com/e89802a4b539562b96d004ae22ab0af163ba08a1/tools/perf/benchmarks/v8.py

 Issue 593960  has been merged into this issue.

Lots of tests still failing:

power.tough_ad_cases.reference
page_cycler.intl_ko_th_vi.reference
smoothness.top_25_smooth.reference
oilpan_gc_times.sync_scroll.key_mobile_sites_smooth.reference
smoothness.fling.simple_mobile_sites
smoothness.fling.simple_mobile_sites.reference
smoothness.gpu_rasterization.top_25_smooth smoothness.gpu_rasterization.top_25_smooth.reference smoothness.tough_texture_upload_cases smoothness.tough_texture_upload_cases.reference failed thread_times.key_noop_cases
thread_times.key_noop_cases.reference
thread_times.polymer
thread_times.polymer.reference

This is so systemic that it must be an 5X issue rather than a Clank issue? Disabling all these tests feels too extreme.

Both Perf (2) and Perf (3) are failing in the same way. Perf (1) is green.

Ethan, are you working on the cert issue?

There are two set of failures here: 
1) Failure due to CertInstallError('Cert Install Failed')
2) Failure due to test timeout.

Retitle this bug to be about CertInstallError & file bug 594136 about test timeout.

I will look at the CertInstallError. 

Hopefully https://codereview.chromium.org/1790563002/ will fix it.

Though I am still a little bit unsure why we don't see this problem with other M devices on the waterfall.

Hmhh, looks like the web-page-replay change is killing all nexus5 bots:

https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5X%20Perf%20%281%29/builds/89
https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5X%20Perf%20%282%29/builds/54

https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5X%20Perf%20%283%29/builds/85

I will revert the change.

The CertInstallError fix in #10 probably work since the nexus5 log no longer contains such error, but look like the cert removal step fails:

(ERROR) 2016-03-11 11:21:40,442 network_controller_backend._RemoveTestCa:142  Error trying to remove certificate authority from target platform.
Traceback (most recent call last):
  File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/platform/network_controller_backend.py", line 138, in _RemoveTestCa
    self._platform_backend.RemoveTestCa()
  File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/platform/android_platform_backend.py", line 558, in RemoveTestCa
    self._device_cert_util.remove_cert()
  File "/b/build/slave/Android_Nexus5X_Perf__1_/build/src/third_party/catapult/telemetry/third_party/webpagereplay/adb_install_cert.py", line 151, in remove_cert
    raise CertRemovalError('Cert Removal Failed')
CertRemovalError: Cert Removal Failed
[  PASSED  ] 30 tests.


For some reason, the benchmark that run right after this timed out. So I think it's probably the certificate removal error that causes all the subsequent runs to fail.

Please let me know if anyone needs an N5X for local testing to make this process go easier. (There are also N5X trybots)

Ok, the problem was that during the step of removing certificate, we run the removing command with:

> adb shell su 0 rm /system/etc/security/cacerts/d07fb5db.0 
rm ro /system/etc/security/cacerts/d07fb5db.0 (y/N) # the command ask for 'Y' to confirm

Hence Telemetry hang here, killing all the subsequent runs. The immediate fix here is to replace the "adb shell su 0 rm" command with "adb shell su 0 rm -f".

Good catch. Do you know why this wasn't impacting other bots?

Likely this is just an L->M thing as in https://github.com/chromium/web-page-replay/issues/60

*Note that this problems does not happen on other bots because "adb shell su 0 rm" inherently only requires confirmation prompt since Marshmallow.

The certificate removal is still failing. Running this locally shows:
"
$ adb shell su 0 rm -f /system/etc/security/cacerts/d07fb5db.0
rm: /system/etc/security/cacerts/d07fb5db.0: Read-only file system
"
Maybe I need extra step of changing the permission of certificate file for android M, will investigate more.


The good news here is although the benchmarks are failing at this final step, all the benchmark values produced from earlier steps should still be sent to the dashboard, hence there is no discontinuity of sending data to the dashboard.

Issue 595703 has been merged into this issue.

It looks to me that starting from Android M, the certificate injection by webpagereplay no longer works, probably due to a change in security restriction.

From the log:

https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/profiler-file-id_0-2016-03-17_07-49-2165733.png


03-17 14:40:46.885 24217 24264 I X509Util: Failed to validate the certificate chain, error: java.security.cert.CertPathValidatorException: Trust anchor for certification path not found.
	03-17 14:40:46.927   916  3623 I ActivityManager: Start proc 24342:com.google.android.apps.chrome:sandboxed_process1/u0i109 for service com.google.android.apps.chrome/org.chromium.content.app.SandboxedProcessService1
	03-17 14:40:46.959 24355 24355 W com.google.android.apps.chrome:sandboxed_process1: type=1400 audit(0.0:313): avc: denied { search } for comm=4173796E635461736B202331 name="com.google.android.apps.chrome" dev="dm-2" ino=376494 scontext=u:r:isolated_app:s0:c512,c768 tcontext=u:object_r:app_data_file:s0:c512,c768 tclass=dir permissive=0
	03-17 14:40:46.959 24355 24355 W com.google.android.apps.chrome:sandboxed_process1: type=1400 audit(0.0:314): avc: denied { search } for comm=4173796E635461736B202331 name="com.google.android.apps.chrome" dev="dm-2" ino=376494 scontext=u:r:isolated_app:s0:c512,c768 tcontext=u:object_r:app_data_file:s0:c512,c768 tclass=dir permissive=0
	03-17 14:40:46.959 24355 24355 W com.google.android.apps.chrome:sandboxed_process1: type=1400 audit(0.0:315): avc: denied { search } for comm=4173796E635461736B202331 name="com.google.android.apps.chrome" dev="dm-2" ino=376494 scontext=u:r:isolated_app:s0:c512,c768 tcontext=u:object_r:app_data_file:s0:c512,c768 tclass=dir permissive=0
	03-17 14:40:46.959 24355 24355 W com.google.android.apps.chrome:sandboxed_process1: type=1400 audit(0.0:316): avc: denied { search } for comm=4173796E635461736B202331 name="com.google.android.apps.chrome" dev="dm-2" ino=376494 scontext=u:r:isolated_app:s0:c512,c768 tcontext=u:object_r:app_data_file:s0:c512,c768 tclass=dir permissive=0
	03-17 14:40:46.959 24355 24355 W com.google.android.apps.chrome:sandboxed_process1: type=1400 audit(0.0:317): avc: denied { search } for comm=4173796E635461736B202331 name="com.google.android.apps.chrome" dev="dm-2" ino=376494 scontext=u:r:isolated_app:s0:c512,c768 tcontext=u:object_r:app_data_file:s0:c512,c768 tclass=dir permissive=0
	03-17 14:40:46.959 24355 24355 W com.google.android.apps.chrome:sandboxed_process1: type=1400 audit(0.0:318): avc: denied { search } for comm=4173796E635461736B202331 name="com.google.android.apps.chrome" dev="dm-2" ino=376494 scontext=u:r:isolated_app:s0:c512,c768 tcontext=u:object_r:app_data_file:s0:c512,c768 tclass=dir permissive=0
	03-17 14:40:46.964 24342 24355 W FileUtils: Failed to chmod(/data/user/0/com.google.android.apps.chrome/app_chrome): android.system.ErrnoException: chmod failed: EACCES (Permission denied)
	03-17 14:40:46.964 24342 24355 W ContextImpl: Unable to create files subdir /data/user/0/com.google.android.apps.chrome/cache
	03-17 14:40:46.967 24342 24342 I cr_ChildProcessService: Creating new ChildProcessService pid=24342
	03-17 14:40:46.971 24342 24356 I cr_LibraryLoader: Using linker: org.chromium.base.library_loader.ModernLinker
	03-17 14:40:46.986 24342 24356 I cr_LibraryLoader: Loading chrome from within /data/app/com.google.android.apps.chrome-1/base.apk
	03-17 14:40:47.019 24342 24356 I cr_LibraryLoader: Time to load native libraries: 48 ms (timestamps 5510-5558)
	03-17 14:40:47.020 24342 24356 I cr_LibraryLoader: Expected native library version number "48.0.2564.95", actual native library version number "48.0.2564.95"
	03-17 14:40:47.020 24342 24356 I chromium: [INFO:library_loader_hooks.cc(130)] Chromium logging enabled: level = 0, default verbosity = 0
	03-17 14:40:48.261   607   607 E QCALOG  : [MessageQ] ProcessNewMessage: [LOWI-SERVER] unknown deliver target [OS-Agent]
	03-17 14:40:55.391   916  3623 I AccountManagerService: getTypesVisibleToCaller: isPermitted? true
	03-17 14:40:55.396 24217 24388 E cr_omaha: Failed to contact server: 
	03-17 14:41:45.370  3227  3458 I Keyboard.Facilitator.LanguageModelFlusher: run()
	03-17 14:41:45.371  3227  3458 I Keyboard.Facilitator: flushDynamicLanguageModels()
	03-17 14:41:45.386  3469  3469 I ConfigService: onCreate

(https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5X%20Perf%20%283%29/builds/99/steps/tracing.tracing_with_debug_overhead.reference/logs/stdio)


So for now, I will just skip Certificate installation on Android M & switch telemetry to use --disable-certificate-error

This problem is fixed now:
https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5X%20Perf%20%282%29/builds/139/steps/memory.top_10_mobile/logs/stdio

https://codereview.chromium.org/1824623002/ for reverting Ethan's disable patch.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/da0ce8d5f44c0bee79fff8a39a500548d06df79b

commit da0ce8d5f44c0bee79fff8a39a500548d06df79b
Author: nednguyen <nednguyen@google.com>
Date: Mon Mar 21 17:10:30 2016

Revert of [Telemetry] Disable some tests on Nexus 5X reference (patchset #1 id:1 of https://codereview.chromium.org/1776473008/ )

Reason for revert:
CertInstallationError problem is fixed.

Original issue's description:
> [Telemetry] Disable some tests on Nexus 5X reference
>
> BUG= 593152 
> CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:winx64_10_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq;tryserver.chromium.perf:linux_perf_cq
>
> Committed: https://crrev.com/e89802a4b539562b96d004ae22ab0af163ba08a1
> Cr-Commit-Position: refs/heads/master@{#380412}

TBR=aiolos@chromium.org,simonhatch@chromium.org,nednguyen@chromium.org,eakuefner@chromium.org
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG= 593152 

Review URL: https://codereview.chromium.org/1824623002

Cr-Commit-Position: refs/heads/master@{#382311}

[modify] https://crrev.com/da0ce8d5f44c0bee79fff8a39a500548d06df79b/tools/perf/benchmarks/blink_style.py
[modify] https://crrev.com/da0ce8d5f44c0bee79fff8a39a500548d06df79b/tools/perf/benchmarks/page_cycler.py
[modify] https://crrev.com/da0ce8d5f44c0bee79fff8a39a500548d06df79b/tools/perf/benchmarks/power.py
[modify] https://crrev.com/da0ce8d5f44c0bee79fff8a39a500548d06df79b/tools/perf/benchmarks/smoothness.py
[modify] https://crrev.com/da0ce8d5f44c0bee79fff8a39a500548d06df79b/tools/perf/benchmarks/v8.py

Re-opening this again as aeference tests are still failing on Nexus5x.

https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5X%20Perf%20(1)

They are failing due to different reason. (see  issue 596464 )