CQ: cheets_SafetyNetTest failures |
||||||||||
Issue descriptionThis test has failed on two CQ runs: https://uberchromegw.corp.google.com/i/chromeos/builders/master-paladin/builds/17958 https://uberchromegw.corp.google.com/i/chromeos/builders/master-paladin/builds/17959 It failed on auron_yuna both times: https://luci-milo.appspot.com/buildbot/chromeos/auron_yuna-paladin/2383 https://luci-milo.appspot.com/buildbot/chromeos/auron_yuna-paladin/2384 And on caroline once: https://luci-milo.appspot.com/buildbot/chromeos/caroline-paladin/2845 START cheets_SafetyNetTest cheets_SafetyNetTest timestamp=1520616120 localtime=Mar 09 09:22:00 FAIL cheets_SafetyNetTest cheets_SafetyNetTest timestamp=1520616165 localtime=Mar 09 09:22:45 Did not pass SafetyNet Basic Integrity check: { END FAIL cheets_SafetyNetTest cheets_SafetyNetTest timestamp=1520616165 localtime=Mar 09 09:22:45 Example failure: https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/182225371-chromeos-test/chromeos6-row2-rack21-host17/debug/ 03/09 09:22:25.917 DEBUG| global_hooks:0056| [u'/usr/bin/adb -s localhost:22 forward tcp:9008 tcp:9008'] 03/09 09:22:27.787 INFO |cheets_SafetyNetTe:0054| Output: { "nonce": "REDACTED", "timestampMs": 1520616147655, "apkPackageName": "org.chromium.arc.testapp.safetynet", "apkDigestSha256": "REDACTED", "ctsProfileMatch": false, "apkCertificateDigestSha256": [ "REDACTED" ], "basicIntegrity": false } 03/09 09:22:27.787 DEBUG| test:0410| Test failed due to Did not pass SafetyNet Basic Integrity check: { "nonce": "REDACTED", "timestampMs": 1520616147655, "apkPackageName": "org.chromium.arc.testapp.safetynet", "apkDigestSha256": "REDACTED", "ctsProfileMatch": false, "apkCertificateDigestSha256": [ "REDACTED" ], "basicIntegrity": false }. Exception log follows the after_iteration_hooks. (REDACTions are mine.) Also note that android stress failed on veyron_minnie in one of these runs. This may indicated a recent flaky android rev (not sure yet): https://luci-milo.appspot.com/buildbot/chromeos/veyron_minnie-paladin/5480 There is no common CL between the two runs, so this is a bug in ToT. --> sheriff for finding owner. I'm going to go chase the theory that this is a bad android rev last night.
,
Mar 9 2018
+ ARC Constable. Can you help us determine when the last ARC roll was, and if it is correlated with this failure?
,
Mar 9 2018
norvez@ points out that this was a KI on the android PFQ: b/73909648 But the last CL on that bug is one enabling this test again. Maybe this failure is flaky? (It's only failed 2 / last 3 runs in the CQ).
,
Mar 9 2018
taking a look at it. The test could be flaky, but: - the tests only calls an Android API - Android collects some information sends it to the server - and the server responds with either "looks good" or "problem". perhaps the server is flaky, but I doubt it. I got in contact with the person responsible for SafetyNet and told me to send the nonce and other info. And she might be able to tell us with signal (file/driver/etc.) is the offending one.
,
Mar 9 2018
Was this test recently enabled in the CQ? In the android PFQ? This bug is currently an emergency. If this test was recently enabled, this is a signal that the test is not ready to be part of the CQ. Our first response should be to remove it from the CQ so that CQ (and hence cros developers) are unblocked.
,
Mar 9 2018
We can have follow up discussion about this test later -- dependence on external services should be kept to a minimum in autotest integration tests.
,
Mar 9 2018
,
Mar 9 2018
I have no reason to believe this is an infra issue. This test became very flaky within the last day or so: https://stainless.corp.google.com/search?view=matrix&row=builder_name&col=queued_date&first_date=2018-02-28&last_date=2018-03-09&builder_name=.*-paladin&test=%5Echeets_SafetyNetTest&exclude_cts=true&exclude_not_run=false&exclude_non_release=false&exclude_au=true&exclude_acts=true&exclude_retried=false&exclude_non_production=true I was wrong about this test being enabled recently. So this brings us back to a bad android roll theory...
,
Mar 9 2018
,
Mar 9 2018
Update: SafetyNet server side recently added new signals for Chromebooks in dev mode. I asked them and told me that the failures that we have are related to that. https://b.corp.google.com/issues/62040924 https://critique.corp.google.com/#review/186460333 https://critique.corp.google.com/#review/187483371 I think one option is to disable SafetyNet (move it from bvt-arc to bvt-perbuild), and fix all the new signal triggers. Once all are fixed, we move the test back to bvt-perbuild. thoughts?
,
Mar 9 2018
sgtm. Please prepare that CL, chump it.
,
Mar 9 2018
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/autotest-cheets/+/5cf6090a17c4f5bf69715155b55d740cc196f4f7 commit 5cf6090a17c4f5bf69715155b55d740cc196f4f7 Author: Ricardo Quesada <ricardoq@google.com> Date: Fri Mar 09 21:51:37 2018
,
Mar 9 2018
#12 removed the test from CQ, while it's being fixed. Tree can be opened now.
,
Mar 10 2018
closing it as fixed. CL will be reverted once we fixed the issue correctly. Android bug: b/74444876
,
Mar 14 2018
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/autotest-cheets/+/0cf630d31eae8064246817cb9cd7c38e3ccf9db4 commit 0cf630d31eae8064246817cb9cd7c38e3ccf9db4 Author: Ricardo Quesada <ricardoq@google.com> Date: Wed Mar 14 17:27:11 2018
,
Mar 15 2018
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/autotest-cheets/+/71021b7a581786341ae3d435428cd14f46a4d50a commit 71021b7a581786341ae3d435428cd14f46a4d50a Author: Ricardo Quesada <ricardoq@google.com> Date: Thu Mar 15 02:42:24 2018 |
||||||||||
►
Sign in to add a comment |
||||||||||
Comment 1 by pprabhu@chromium.org
, Mar 9 2018