New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 601464 link

Starred by 3 users
Status: Fixed
Owner:
agrieve@chromium.org
Closed: May 2016
Cc:
█ kjellander@chromium.org
jbudorick@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Android
Pri: 3
Type: Bug



Sign in to add a comment

Instrumentation tests without @SmallTest, @MediumTest, etc are never run

Project Member Reported by agrieve@chromium.org, Apr 7 2016 
This is particularly a problem for webrtc's AppRTCDemoTest, which doesn't have any annotations, and so has 0 tests being run!

There are both whitelist and blacklist annotations.
Whitelist: @SmallTest, @MediumTest, @EnormousTest, etc
Blacklist: @Disabled, @FlakyTest

By default, test_runner.py runs tests that are annotated with any of the whitelisted annotation, but if it is lacking any of them, does not get run.

Possible solutions:
1. Raise an exception if any tests is found that doesn't have at least one annotation from the set of whitelist annotations
2. Always run tests that are lacking a whitelist annotation, but also show a warning for them
3. Change the default behaviour (where no explicit set of whitelist annotations is passed via the command-line), to run all tests.

1. Is probably the best outcome since it will ensure all tests are annotated
3. Is reasonable, but would result in tests that might get skipped on bots
2. Seems strictly better than right now, but #1 would be better longer term. Unsure if warnings would entice projects to annotate.

Comment 1 by jbudorick@chromium.org, Apr 7 2016 

... how would 3 result in tests that might get skipped by the bots?

Comment 2 by agrieve@chromium.org, Apr 7 2016 

Maybe this isn't the case, but if every bot we have in chromium specifies annotations, then tests without an annotation would never run. If we have bots that don't set annotations, then we'd be fine. It's still probably preferable to someway force all tests to have annotations though.
Project Member

Comment 3 by bugdroid1@chromium.org, Apr 12 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/external/webrtc.git/+/9c4fadc1997e99f1ba48408fc4ee8910f4cbcae1

commit 9c4fadc1997e99f1ba48408fc4ee8910f4cbcae1
Author: kjellander <kjellander@webrtc.org>
Date: Tue Apr 12 04:27:32 2016

Add test annotations to AppRTCDemoTest.

After rolling in https://codereview.webrtc.org/1847963004 the AppRTCDemoTest
started running 0 tests due to  https://crbug.com/601464 . Adding test annotations
makes the tests being executed again.

BUG= chromium:601464 
NOTRY=True

Review URL: https://codereview.webrtc.org/1876233002

Cr-Commit-Position: refs/heads/master@{#12325}

[modify] https://crrev.com/9c4fadc1997e99f1ba48408fc4ee8910f4cbcae1/webrtc/examples/androidtests/src/org/appspot/apprtc/test/LooperExecutorTest.java
[modify] https://crrev.com/9c4fadc1997e99f1ba48408fc4ee8910f4cbcae1/webrtc/examples/androidtests/src/org/appspot/apprtc/test/PeerConnectionClientTest.java
Project Member

Comment 4 by bugdroid1@chromium.org, Apr 30 2016 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/clank/internal/apps/+/86775cbcab40cd38de1c543e1b3c7a23379820b1

commit 86775cbcab40cd38de1c543e1b3c7a23379820b1
Author: Andrew Grieve <agrieve@chromium.org>
Date: Fri Apr 29 19:15:58 2016
Project Member

Comment 5 by bugdroid1@chromium.org, May 3 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/70d776ed260e4c48a4f4a92903030f1e578ef291

commit 70d776ed260e4c48a4f4a92903030f1e578ef291
Author: agrieve <agrieve@chromium.org>
Date: Tue May 03 17:55:04 2016

Fail if an instrumentation test is missing size annotation

Right now we just silently never run such tests.

BUG= 601464 

Review-Url: https://codereview.chromium.org/1863353002
Cr-Commit-Position: refs/heads/master@{#391288}

[modify] https://crrev.com/70d776ed260e4c48a4f4a92903030f1e578ef291/android_webview/javatests/src/org/chromium/android_webview/test/LoadUrlTest.java
[modify] https://crrev.com/70d776ed260e4c48a4f4a92903030f1e578ef291/android_webview/javatests/src/org/chromium/android_webview/test/NavigationHistoryTest.java
[modify] https://crrev.com/70d776ed260e4c48a4f4a92903030f1e578ef291/android_webview/javatests/src/org/chromium/android_webview/test/StandaloneAwQuotaManagerBridgeTest.java
[modify] https://crrev.com/70d776ed260e4c48a4f4a92903030f1e578ef291/base/android/javatests/src/org/chromium/base/AdvancedMockContextTest.java
[modify] https://crrev.com/70d776ed260e4c48a4f4a92903030f1e578ef291/build/android/pylib/instrumentation/instrumentation_test_instance.py
[modify] https://crrev.com/70d776ed260e4c48a4f4a92903030f1e578ef291/chrome/android/javatests/src/org/chromium/chrome/browser/ProcessIsolationTest.java
[modify] https://crrev.com/70d776ed260e4c48a4f4a92903030f1e578ef291/chrome/android/javatests/src/org/chromium/chrome/browser/media/ui/NotificationTitleUpdatedTest.java
[modify] https://crrev.com/70d776ed260e4c48a4f4a92903030f1e578ef291/chrome/android/javatests/src/org/chromium/chrome/browser/preferences/website/SiteSettingsPreferencesTest.java
[modify] https://crrev.com/70d776ed260e4c48a4f4a92903030f1e578ef291/content/public/android/javatests/src/org/chromium/content/browser/PhoneNumberDetectionTest.java

Comment 6 by agrieve@chromium.org, May 3 2016

Status: Fixed (was: Assigned)
Project Member

Comment 7 by bugdroid1@chromium.org, May 3 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e682757ee4379efa1b1e23fd07d7541248fd12ff

commit e682757ee4379efa1b1e23fd07d7541248fd12ff
Author: mathp <mathp@chromium.org>
Date: Tue May 03 20:25:57 2016

Revert of
Project Member

Comment 8 by bugdroid1@chromium.org, May 4 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/01c2d6a9bdc61d422d8af9589ca0a0830d853ec1

commit 01c2d6a9bdc61d422d8af9589ca0a0830d853ec1
Author: agrieve <agrieve@chromium.org>
Date: Wed May 04 02:28:24 2016

Reland of Fail if an instrumentation test is missing size annotation

Changes since last one:
SiteSettingsPreferencesTest.testBackgroundSyncPermission now
named as a helper instead of annotated as a test.

TBR=jbudorick@chromium.org,sgurun@chromium.org,yfriedman@chromium.org,mathp@chromium.org
BUG= 601464 

Review-Url: https://codereview.chromium.org/1950793002
Cr-Commit-Position: refs/heads/master@{#391421}

[modify] https://crrev.com/01c2d6a9bdc61d422d8af9589ca0a0830d853ec1/android_webview/javatests/src/org/chromium/android_webview/test/LoadUrlTest.java
[modify] https://crrev.com/01c2d6a9bdc61d422d8af9589ca0a0830d853ec1/android_webview/javatests/src/org/chromium/android_webview/test/NavigationHistoryTest.java
[modify] https://crrev.com/01c2d6a9bdc61d422d8af9589ca0a0830d853ec1/android_webview/javatests/src/org/chromium/android_webview/test/StandaloneAwQuotaManagerBridgeTest.java
[modify] https://crrev.com/01c2d6a9bdc61d422d8af9589ca0a0830d853ec1/base/android/javatests/src/org/chromium/base/AdvancedMockContextTest.java
[modify] https://crrev.com/01c2d6a9bdc61d422d8af9589ca0a0830d853ec1/build/android/pylib/instrumentation/instrumentation_test_instance.py
[modify] https://crrev.com/01c2d6a9bdc61d422d8af9589ca0a0830d853ec1/chrome/android/javatests/src/org/chromium/chrome/browser/ProcessIsolationTest.java
[modify] https://crrev.com/01c2d6a9bdc61d422d8af9589ca0a0830d853ec1/chrome/android/javatests/src/org/chromium/chrome/browser/media/ui/NotificationTitleUpdatedTest.java
[modify] https://crrev.com/01c2d6a9bdc61d422d8af9589ca0a0830d853ec1/chrome/android/javatests/src/org/chromium/chrome/browser/preferences/website/SiteSettingsPreferencesTest.java
[modify] https://crrev.com/01c2d6a9bdc61d422d8af9589ca0a0830d853ec1/content/public/android/javatests/src/org/chromium/content/browser/PhoneNumberDetectionTest.java
►

Sign in to add a comment