Derek can you take a look?

Ping? Any progress on this?

It appears that this test is broken on many boards:  It is consistently failing on daisy, x86-alex and oak as well.

How do we typically limit which boards a given test will run on? Is it via a DEPENDENCIES entry? Glancing around, I don't see one that would obviously correspond to dark resume support. Should I add one to server/hosts/cros_label.py that e.g. looks for powerd prefs enabling dark resume?

Hmm, but I guess that this test also requires i915, so maybe it's best if it just depends on board labels. Where should it be running right now? Dark resume is disabled on samus, right?

Derek is out this week.  Sean, could you PTAL?

Oh, heh. A label already exists but it's using a different name (and just a hardcoded list right now):

class LucidSleepLabel(base_label.BaseLabel):
    """Label that determines if device has support for lucid sleep."""

    # TODO(kevcheng): See if we can determine if this label is applicable a
    # better way (crbug.com/592146).
    _NAME = 'lucidsleep'
    LUCID_SLEEP_BOARDS = ['samus', 'lulu']

    def exists(self, host):
        board = host.get_board().replace(ds_constants.BOARD_PREFIX, '')
        return board in self.LUCID_SLEEP_BOARDS

I assume the list is out of date, because I see the following boards configured to use dark resume:

elm
link
lulu (disabled)
oak (disabled)
parrot
samus (disabled)

The only ones it lists actually have the disable_dark_resume pref also set. :-/

The test will only work on select 3.14 and 3.18 kernels at the moment, so perhaps that's why the list seems more restrictive?

Would there be any way of updating the 'kernel_daily_regression' test suite so that tests which shouldn't be run on a particular board are skipped or automatically marked as passed when that board attempts to run the suite?

I'm not running this test as an individual test on the board, I just asked the entire suite to run on the board, and this particular test (and some others) fail the test, which makes the entire suite overall marked as FAILED.  It would be nice if, when all the *appropriate* tests for a board passes, the suite is marked as  PASSED when run on that board (i.e. as I said at the start, if a particular test isn't supposed to work on a particular board, the suite doesn't try to run it on that board).

If the target board doesn't support a test's dependencies, does the test get skipped?

That does not appear to be the case. For example, when I run the kernal_daily_regression suite on an x86-alex machine, which is a 3.8 kernel I get a FAIL on power_DarkResumeDisplay, which according to comment #11 shouldn't even try to run on that board, since it's the wrong kernel version.

The only dependency listed in power_DarkResumeDisplay's control file is on the "servo" label[1]. I'm wondering if it'd be skipped if it listed the correct dependencies. :-)

1. which is incorrect, since it apparently also depends on a particular kernel version... but maybe not on dark resume being enabled in powerd, since it looks like dark_resume_utils modifies powerd's pref files.

I suppose we could also mark the test as passed if the error_count file in question doesn't exist on the device (ie: board and/or kernel not supported). 

Ping?  What's the status of this now?  

I looked at the consistent failures of this test suite (kernel_daily_regression) on go/wmatrix for R55, for the 14 boards we test regularly.  Most of the boards are passing most of the tests (or sometimes passing a test & sometimes failing it).   There are 3 tests that appear to be major problems:

power_DarkResumeDisplay  (10 boards consistently fail test; 3 boards don't run it)
power_DarkResumeShutdownServer (6 boards consistently fail; 3 boards don't run it)
power_SuspendStress.bare (7 boards consistently fail; 1 board doesn't run it)

1 board consistently fails power_SuspendShutdown.

Which tests fail on what boards (consistently):

power_DarkResumeDisplay: falco, x86-alex, oak, squawks, terra, peppy, veyron_jaq, sentry, chell, nyan_big
power_DarkResumeShutdownServer: falco, x86-alex, sentry, chell, nyan_big, link
power_SuspendStress.bare: falco, daisy, x86-alex, oak, squawks, veyron_jaq, lumpy
power_SuspendShutdown: falco

Derek and Sameer (and Sean, in the case of power_DarkResumeDisplay): You're listed as the owners of power_DarkResumeDisplay and power_DarkResumeDisplay. I believe that we have dark resume disabled on most/all boards due to kernel issues. Are these tests expected to pass anyway, or can we disable them until the underlying issues are fixed?

It may make sense to use a different bug to track the power_SuspendStress.bare and power_SuspendShutdown failures, since those are probably due to unrelated flakiness problems in those tests. (Should we get rid of power_SuspendShutdown? powerd already has pretty good unit test coverage for that functionality.)

Copying Derek's comment from  issue 631504 : "If power_DarkResumeShutdownServer is having problems, it likely that mosys is just broken. It should just call into mosys to check the platform and return on daisy, oak, and alex."

(copying comment from similar issue)
Can we please put the onus of these tests on you instead of on the users of the suite?
that is, if these tests are flaky or need some setup to work on only a few boards, then disable the tests until you decide what to do with them instead of having the users of the suite analyze these failures every day.
I think we can all agree that flaky or constantly failing tests are BAD.

That comment was from power_DarkResumeShutdownServer, not power_DarkResumeDisplay.

This seems to be because the test raises an error when it should just silently exit.

fix to address power_DarkResumeDisplay & Derek's observation (#c22) here:

https://chromium-review.googlesource.com/#/c/383980

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/5e38889798004f055a3e9a9f901d1049edf07cb3

commit 5e38889798004f055a3e9a9f901d1049edf07cb3
Author: Todd Broch <tbroch@chromium.org>
Date: Sat Sep 10 13:11:09 2016

power_DarkResumeDisplay: raise TestNAError if unsupported.

Only some platforms support this test.  For those that dont raise
TestNAError instead of TestError accordingly.

BUG= 625281 
TEST=test_that <ip> power_DarkResumeDisplay on unsupported platform and
it passes

Change-Id: Ib8dc564281fe652dc2570e6f629272a2fc883d0d
Reviewed-on: https://chromium-review.googlesource.com/383980
Commit-Ready: Todd Broch <tbroch@chromium.org>
Tested-by: Todd Broch <tbroch@chromium.org>
Reviewed-by: Derek Basehore <dbasehore@chromium.org>

[modify] https://crrev.com/5e38889798004f055a3e9a9f901d1049edf07cb3/server/site_tests/power_DarkResumeDisplay/power_DarkResumeDisplay.py

we are still hitting an issue with this test.

from this builder:
https://uberchromegw.corp.google.com/i/chromeos/builders/amd64-gcc-toolchain/builds/1

you can see that kernel_daily_regression still fails with:

[Test-Logs]: power_DarkResumeDisplay: ERROR: /sys/kernel/debug/dri/0/i915_crtc_errors file not found.


please fix this last one. 
That is the only remaining failure that we have for the suite.

https://chromium-review.googlesource.com/#/c/385701

should take care of remaining issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/ae6129d620a528a771764c7e5973f0b327b2bec6

commit ae6129d620a528a771764c7e5973f0b327b2bec6
Author: Todd Broch <tbroch@chromium.org>
Date: Wed Sep 14 23:15:56 2016

power_DarkResumeDisplay: check kernel version.

This test will only work on 3.14 & 3.18 kernels at the moment.  Check
that and if its not the case raise TestNAError accordingly.

Also cleaned up a few pylint warnings.

BUG= chromium:625281 
TEST=power_DarkResumeDisplay

1. run on 4.4 device and raises TestNAError but fails
2. run on 3.18 device and it succeeds.

Change-Id: Ib74c909e0513ccd98b0cc23e7e90cc28f725af2d
Reviewed-on: https://chromium-review.googlesource.com/385701
Commit-Ready: Todd Broch <tbroch@chromium.org>
Tested-by: Todd Broch <tbroch@chromium.org>
Reviewed-by: Derek Basehore <dbasehore@chromium.org>

[modify] https://crrev.com/ae6129d620a528a771764c7e5973f0b327b2bec6/server/site_tests/power_DarkResumeDisplay/power_DarkResumeDisplay.py

Please re-open if #c27 didn't squash the last issue.

Reopening per llozano@ at https://bugs.chromium.org/p/chromium/issues/detail?id=631504#c20:

----

There is still a failure in ARM

from this builder:

https://uberchromegw.corp.google.com/i/chromeos/builders/arm-gcc-toolchain/builds/6

The error is

[Test-Logs]: power_DarkResumeShutdownServer: ERROR: Unhandled RemotePowerException: Failed to change outlet status for host: chromeos2-row3-rack5-host21 to state: ON.

which links to: http://cautotest/tko/retrieve_logs.cgi?job=/results/77296329-chromeos-test/

Please help. This seems to be the last one. Testing with this suite is looking much better now.

The Unhandled RemotePowerException is an old lab RPM infra  issue 517233 . 
+fdeng,jrbarnette

Thanks. I pasted #29 to  issue 517233 ; let's use it to track that failure.

adding 517233 to blocked on. 

the failure seems pretty consistent (does not seem like a flake)
Is there a short term solution?  Issue 517233  has been open for a while.

Verified in R57-9008.0.0 that tests skipped with "Test support on 3.14 | 3.18 kernels only"
https://wmatrix.googleplex.com/testrun/kernel_daily?test_ids=384337989