New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 811878 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Last visit 22 days ago
Closed: Feb 2018
Cc:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

cheets_KeyboardTest fails frequently on multiple boards: retry_count: 2, FAIL: Unhandled AssertionError

Project Member Reported by khmel@chromium.org, Feb 13 2018

Issue description

For example: https://luci-milo.appspot.com/buildbot/chromeos/quawks-paladin/2174


02/13 05:37:11.298 WARNI|               arc:0460| Iteration 1 failed, taking a screenshot.
02/13 05:37:11.609 DEBUG|              test:0420| after_iteration_hooks completed
02/13 05:37:11.615 WARNI|              test:0637| The test failed with the following exception
Traceback (most recent call last):
  File "/usr/local/autotest/common_lib/test.py", line 631, in _exec
    _call_test_function(self.execute, *p_args, **p_dargs)
  File "/usr/local/autotest/common_lib/test.py", line 837, in _call_test_function
    raise error.UnhandledTestFail(e)
UnhandledTestFail: Unhandled AssertionError: 
Traceback (most recent call last):
  File "/usr/local/autotest/common_lib/test.py", line 831, in _call_test_function
    return func(*args, **dargs)
  File "/usr/local/autotest/common_lib/test.py", line 495, in execute
    dargs)
  File "/usr/local/autotest/common_lib/test.py", line 362, in _call_run_once_with_retry
    postprocess_profiled_run, args, dargs)
  File "/usr/local/autotest/common_lib/test.py", line 400, in _call_run_once
    self.run_once(*args, **dargs)
  File "/usr/local/autotest/tests/cheets_KeyboardTest/cheets_KeyboardTest.py", line 53, in run_once
    self.keyboard.playback(full_path)
  File "/usr/local/autotest/cros/input_playback/keyboard.py", line 45, in playback
    input_type=_KEYBOARD)
  File "/usr/local/autotest/cros/input_playback/input_playback.py", line 504, in blocking_playback
    self.playback(filepath, input_type)
  File "/usr/local/autotest/cros/input_playback/input_playback.py", line 478, in playback
    assert(input_type in
 

Comment 1 by khmel@chromium.org, Feb 13 2018

Hi Chung-yih,
Could you please take a look.

This is potentially P0 once this is reason for multiple CQ failures.

Comment 2 by khmel@chromium.org, Feb 13 2018

Cc: lgcheng@google.com
+Long
Cc: akes...@chromium.org pho...@chromium.org snanda@chromium.org

Comment 4 by pho...@chromium.org, Feb 13 2018

Labels: -Pri-1 Pri-0
carilen just failed with this as well, https://luci-milo.appspot.com/buildbot/chromeos/caroline-paladin/2641
Cc: ihf@chromium.org

Comment 6 by snanda@chromium.org, Feb 13 2018

For c#0 should cheets tests be run on quawks (a BYT platform)?

Comment 7 by ihf@chromium.org, Feb 13 2018

I think it should not be cheets, but it clearly still is.
~/trunk/src/private-overlays/overlay-quawks-private

Comment 8 by ihf@chromium.org, Feb 13 2018

Owner: ihf@chromium.org
Status: Started (was: Untriaged)
I'll try to get rid of it.

Comment 9 by ihf@chromium.org, Feb 13 2018

I went through all BYT boards and also found squawks. Hope that is all.
Project Member

Comment 10 by bugdroid1@chromium.org, Feb 13 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/overlay-quawks-private/+/cc856239eaaa0296cfb1e0e4a5764805f23911f0

commit cc856239eaaa0296cfb1e0e4a5764805f23911f0
Author: Ilja H. Friedel <ihf@chromium.org>
Date: Tue Feb 13 22:16:31 2018

Comment 11 by ihf@chromium.org, Feb 13 2018

Components: Platform>ARC
Labels: -Pri-0 M-66 OS-Chrome Pri-1
the point of failure does not look relevant to cheets (input_playback.py is failing to find the keyboard input device (of Chrome OS, not the Android container). I'm not sure if it is a good thing to hide the failure.
Apparently the failures in #13 are only seen in the CQ, which makes me suspect some change in the queue, e.g.,
https://chromium-review.googlesource.com/c/chromiumos/third_party/autotest/+/907949

Comment 15 by lgcheng@google.com, Feb 13 2018

Yeah, I had sometime looking into the log. Same issue also happens on caroline.

Comment 16 by lgcheng@google.com, Feb 13 2018

Re #14 very likely

Logs for successful run looks like:
input_playback:0161| Emulating keyboard Emulated Keyboard 
(/usr/local/autotest/cros/input_playback/keyboard.prop).
02/12 21:18:11.381 DEBUG|             utils:0214| Running 'grep "interrupt to exit" /tmp/tmpxCLFNJ | wc -l'
02/12 21:18:11.398 DEBUG|             utils:0214| Running 'cat /sys/class/input/event9/device/name'
02/12 21:18:11.410 INFO |    input_playback:0412| Found keyboard: Emulated Keyboard at /dev/input/event9.

While failure run looks like:
02/13 05:22:18.660 INFO |    input_playback:0161| Emulating keyboard Emulated Keyboard (/usr/local/autotest/cros/input_playback/keyboard.prop).
02/13 05:22:18.787 DEBUG|             utils:0214| Running 'grep "interrupt to exit" /tmp/tmphnVMpq | wc -l'
02/13 05:22:18.802 DEBUG|    input_playback:0219| /dev/input/event9: Invalid argument

crrev.com/c/907949 couldn't have caused the latest caroline failure of cheets_KeyboardTest - it wasn't in the CQ at the time. Are there multiple bad CLs then?

Comment 18 by lgcheng@google.com, Feb 13 2018

Re# 17 Could be
This run of caroline(2/12): https://uberchromegw.corp.google.com/i/chromeos/builders/caroline-paladin/builds/2638
Fails with sympothm input_playback:0219| /dev/input/event9: Invalid argument
which is introduced in crrev.com/c/907949.

Latest run is different failure.

Comment 19 by lgcheng@google.com, Feb 13 2018

Like mentioned in 
https://bugs.chromium.org/p/chromium/issues/detail?id=809827

crrev.com/c/907949 is to ignore the error of find_connected_inputs in input_playback. However simply ignoring the error will result self.devices empty and cause
assert(input_type in self.devices) fail
Labels: -Pri-1 Pri-0
Summary: cheets_KeyboardTest fails frequently on multiple boards: retry_count: 2, FAIL: Unhandled AssertionError (was: quawks-paladin frequently failing cheets_KeyboardTest: retry_count: 2, FAIL: Unhandled AssertionError)
Project Member

Comment 22 by bugdroid1@chromium.org, Feb 14 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/overlay-quawks-private/+/b29d0fabc7933406275c91a80e5c92968be94ccb

commit b29d0fabc7933406275c91a80e5c92968be94ccb
Author: Ilja H. Friedel <ihf@google.com>
Date: Wed Feb 14 01:13:37 2018

Cc: adlr@chromium.org snanda@google.com
 Issue 812018  has been merged into this issue.
Project Member

Comment 25 by bugdroid1@chromium.org, Feb 14 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/autotest-cheets/+/50052f737483b1c44358e8a19d89e6b2d267d39c

commit 50052f737483b1c44358e8a19d89e6b2d267d39c
Author: Aviv Keshet <akeshet@chromium.org>
Date: Wed Feb 14 01:21:19 2018

Cc: risan@chromium.org hirono@chromium.org
+constables fyi
Cc: kathrelk...@chromium.org

Comment 28 by ihf@chromium.org, Feb 14 2018

Labels: -Pri-0 Pri-1
Owner: ----
Status: Available (was: Started)
I will remove the containers when Bernie gives the green light. Not going to debug the actual failure.

No more emergency as the CQ should be unblocked with #25.
Cc: yhanada@chromium.org kinaba@chromium.org hidehiko@chromium.org
FYI to ARC Keyboard expert.
Owner: hirono@chromium.org
Status: Assigned (was: Available)
Assigning to the non-PST ARC constable.

Can we simply re-enable the test (cheets_KeyboardTest) and close this bug?

* The cause of the "Unhandled AssertionError:" failure is already identified to be a bad CL:
it was happening if and only if crrev.com/c/907949 was in the CQ batch (R66-10401.0.0-rc1, R66-10400.0.0-rc1, R66-10399.0.0-rc2, R66-10398.0.0-rc2, R66-10389.0.0-rc4).

https://stainless.corp.google.com/search?view=list&first_date=2018-02-01&last_date=2018-02-20&test=cheets_KeyboardTest&status=FAIL&status=ERROR&exclude_cts=false&exclude_not_run=false&exclude_non_release=false&exclude_au=true&exclude_acts=true&exclude_retried=true&exclude_non_production=true

* The other failure mode "Timed out waiting for unnamed condition" is already worked on at crbug.com/809827 (where crrev.com/c/907949 came from). We don't need two duplicated bugs. Moreover this failure is not specific to the cheets_KeyboardTest test.

Comment 31 by ihf@chromium.org, Feb 15 2018

Revert SGTM.
Status: Started (was: Assigned)
Created the revert.
https://chrome-internal-review.googlesource.com/c/chromeos/autotest-cheets/+/571018

Project Member

Comment 33 by bugdroid1@chromium.org, Feb 15 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/autotest-cheets/+/f6e22a774aae1bc0fcca18f38e80fb18bdbdbdc7

commit f6e22a774aae1bc0fcca18f38e80fb18bdbdbdc7
Author: Daichi Hirono <hirono@google.com>
Date: Thu Feb 15 16:58:13 2018

Status: Fixed (was: Started)
Repeating my comment, for future verifiers:

the "Timed out waiting for unnamed condition" error from cheets_KeyboardTest is separately tracked at crbug.com/809827.
Project Member

Comment 36 by bugdroid1@chromium.org, Jul 21

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/overlay-squawks-private/+/a884ac0004aa8f3bf1791b1c72a9f7de7f09cea3

commit a884ac0004aa8f3bf1791b1c72a9f7de7f09cea3
Author: Ilja H. Friedel <ihf@chromium.org>
Date: Sat Jul 21 01:34:10 2018

Sign in to add a comment