New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 638407 link

Starred by 1 user

Issue metadata

Status: Untriaged
Owner: ----
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Android , All
Pri: 3
Type: Bug



Sign in to add a comment

Handling of test names containing "DISABLED" without trailing underscore is inconsistent with GTest except on Android

Project Member Reported by ddorwin@chromium.org, Aug 16 2016

Issue description

What steps will reproduce the problem?
(1) Add "DISABLED" without a trailing underscore to a test. For example, Rename SomeTest to SomeTest_DISABLED.
(2) Replace the body of the test with:
  FAIL() << "Unexpected failure: Disabled test should not be run.";
(3) Run the test on Linux and Android (specifically, the linux_android_rel_ng bot)

What is the expected output?
Most importantly, the behavior should be the same on both platforms.

Ideally, the behavior would be the same as Google Test, which is that "DISABLED" without the trailing underscore is NOT a disabled test and thus the test should fail on both platforms.

What do you see instead?
The test does not run on Linux and thus there is no test failure, but the test runs and fails on Android.

Please use labels and text to provide additional information.
Raw Google Test, only disables tests if the test case or name *begins* with "DISABLED_", including the trailing underscore. [1] This can be verified by modifying Google Test's unit tests.

However, Chromium's test launcher disables all tests that contain "DISABLED". [2] This can be confirmed by copying and modifying Google Test's NotDISABLED_TestShouldRun test. For example:
  TEST(DisabledTest, NotDISABLED_TestShouldRun) {
    FAIL() << "You should see a failure because this should run.";
  }
While this would fail in Google Test, the test does not run in Chromium unit tests.

Chromium should be consistent with Google Test and avoid potentially disabling tests unexpectedly. (This also seems preferred since diverging opens us up to inconsistencies like those below.) Any code analysis that looks for "DISABLED_" could be inaccurate too.


Most importantly, though, Chromium should be consistent across platforms! If we're not going to match the Google Test behavior, we need to ensure all platforms behave the same, which is not currently the case for Android at least. I didn't determine why Android doesn't follow this behavior, but I can confirm that a test of the form SomeTest_DISABLED ran (and failed) on the linux_android_rel_ng bot. The stdio contained a line that started with "Note: Google Test filter = " and contained SomeTest_DISABLED.

Assigning to phajdan.jr who has modified the test launcher the last few times.

[1] https://cs.chromium.org/chromium/src/testing/gtest/src/gtest.cc?q=DISABLED_&sq=package:chromium&l=160&dr=C
[2] https://cs.chromium.org/chromium/src/base/test/launcher/test_launcher.cc?q=DISABLED+file:%5Esrc/base/test/&sq=package:chromium&l=956&dr=C
 
Cc: phajdan.jr@chromium.org
Owner: ----
Status: Available (was: Assigned)
Project Member

Comment 2 by sheriffbot@chromium.org, Sep 25 2017

Labels: Hotlist-Recharge-Cold
Status: Untriaged (was: Available)
This issue has been Available for over a year. If it's no longer important or seems unlikely to be fixed, please consider closing it out. If it is important, please re-triage the issue.

Sorry for the inconvenience if the bug really should have been left as Available. If you change it back, also remove the "Hotlist-Recharge-Cold" label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Components: Tests>Disabled
Labels: Test-Disabled

Sign in to add a comment