What steps will reproduce the problem?
(1) Add "DISABLED" without a trailing underscore to a test. For example, Rename SomeTest to SomeTest_DISABLED.
(2) Replace the body of the test with:
FAIL() << "Unexpected failure: Disabled test should not be run.";
(3) Run the test on Linux and Android (specifically, the linux_android_rel_ng bot)
What is the expected output?
Most importantly, the behavior should be the same on both platforms.
Ideally, the behavior would be the same as Google Test, which is that "DISABLED" without the trailing underscore is NOT a disabled test and thus the test should fail on both platforms.
What do you see instead?
The test does not run on Linux and thus there is no test failure, but the test runs and fails on Android.
Please use labels and text to provide additional information.
Raw Google Test, only disables tests if the test case or name *begins* with "DISABLED_", including the trailing underscore. [1] This can be verified by modifying Google Test's unit tests.
However, Chromium's test launcher disables all tests that contain "DISABLED". [2] This can be confirmed by copying and modifying Google Test's NotDISABLED_TestShouldRun test. For example:
TEST(DisabledTest, NotDISABLED_TestShouldRun) {
FAIL() << "You should see a failure because this should run.";
}
While this would fail in Google Test, the test does not run in Chromium unit tests.
Chromium should be consistent with Google Test and avoid potentially disabling tests unexpectedly. (This also seems preferred since diverging opens us up to inconsistencies like those below.) Any code analysis that looks for "DISABLED_" could be inaccurate too.
Most importantly, though, Chromium should be consistent across platforms! If we're not going to match the Google Test behavior, we need to ensure all platforms behave the same, which is not currently the case for Android at least. I didn't determine why Android doesn't follow this behavior, but I can confirm that a test of the form SomeTest_DISABLED ran (and failed) on the linux_android_rel_ng bot. The stdio contained a line that started with "Note: Google Test filter = " and contained SomeTest_DISABLED.
Assigning to phajdan.jr who has modified the test launcher the last few times.
[1] https://cs.chromium.org/chromium/src/testing/gtest/src/gtest.cc?q=DISABLED_&sq=package:chromium&l=160&dr=C
[2] https://cs.chromium.org/chromium/src/base/test/launcher/test_launcher.cc?q=DISABLED+file:%5Esrc/base/test/&sq=package:chromium&l=956&dr=C
Comment 1 by phajdan.jr@chromium.org
, Sep 23 2016Owner: ----
Status: Available (was: Assigned)