New issue
Advanced search Search tips

Issue 807237 link

Starred by 1 user
Status: Untriaged
Owner: ----
Cc:
tobiasjs@chromium.org
jbudorick@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Android
Pri: 2
Type: Bug



Sign in to add a comment

JUnit initialization errors confuse test runner

Project Member Reported by bauerb@chromium.org, Jan 30 2018 
Steps to reproduce:

1) Add something to an instrumentation test that results in a validation error, like making a @Rule not public.
2) Run the instrumentation test

Result:
JUnit throws an InitializationError when creating the TestRunner for the class, but the testing script on the host misinterprets that as a test name, i.e. it thinks there is one test named initializationError() in the test class (!). That test of course doesn't actually exist, so either the test filter matches nothing, or the test runner throws an error because the test is missing a size annotation. Nothing in the current output indicates the underlying InitializationError.
Project Member

Comment 1 by bugdroid1@chromium.org, Jan 30 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/adee399fe7d7084f5e040c32091cdca5ac0b98ca

commit adee399fe7d7084f5e040c32091cdca5ac0b98ca
Author: Bernhard Bauer <bauerb@chromium.org>
Date: Tue Jan 30 14:40:14 2018

Log initialization errors in JUnit test suites.

This helps with debugging, as the thrown exception will get mangled by
the test runner script.

Bug: 807237
Change-Id: I74677e491c0115521b9b79c051f5213e832545ae
Reviewed-on: https://chromium-review.googlesource.com/892863
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Bernhard Bauer <bauerb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#532862}
[modify] https://crrev.com/adee399fe7d7084f5e040c32091cdca5ac0b98ca/base/test/android/javatests/src/org/chromium/base/test/BaseJUnit4ClassRunner.java

Comment 2 by bauerb@chromium.org, Feb 15 2018

Cc: tobiasjs@chromium.org

Comment 3 by yolandyan@chromium.org, Feb 15 2018 

I have encountered this in the past. Before every test run, we use BaseChromiumAndroidJUnitRunner.listTests() to list out all the test names on the device side by fake-running them. However, if one of the test is incorrectly written (e.g. missing an annotation), the JUnit runner would raise initializationError during the fake-run and save that as the name of a class in a JSON file, meanwhile, the host side will think there is a test named "initializationError"

fake-running: https://cs.chromium.org/chromium/src/base/test/android/javatests/src/org/chromium/base/test/BaseJUnit4ClassRunner.java?l=174

save to json:
https://cs.chromium.org/chromium/src/base/test/android/javatests/src/org/chromium/base/test/BaseChromiumAndroidJUnitRunner.java?l=149

Comment 4 by bauerb@chromium.org, Mar 19 2018

Cc: jbudorick@chromium.org
Owner: bauerb@chromium.org
Status: Started (was: Untriaged)
So, I have a WIP CL that plumbs the failure information through to the test runner script on the host (https://crrev.com/c/968487), but I'm not really sure what to do with it there – my CL right now actually makes the test run succeed, because it will simply list no tests in a class that can't be instantiated, which would make it likely that no one would notice the failure. John, do you have an idea what the best way would be to surface the initialization error from the test runner script?

Comment 5 by bauerb@chromium.org, Sep 8

Owner: ----
Status: Untriaged (was: Started)
Bulk edit: Moving back into the untriaged pool, as I'm leaving the project.
►

Sign in to add a comment