The stack trace is interesting. It is complaining that the mUrlRequestContextAdapter is 0, even though we have not done anything yet. Is it possible that because the initialization of mUrlRequestContextAdapter is not protected by mLock, that it is somehow not visible when we call createRequest() ? 

Maybe we should try synchronizing the access with mLock in:

        mUrlRequestContextAdapter = nativeCreateRequestContextAdapter(
                createNativeUrlRequestContextConfig(builder.getContext(), builder));
        if (mUrlRequestContextAdapter == 0) {
            throw new NullPointerException("Context Adapter creation failed.");
        }

C  221.290s Main  [FAIL] org.chromium.net.CronetUrlRequestContextTest#testShutdown:
C  221.290s Main  java.lang.Throwable: CronetTestBase#runTest failed.
C  221.290s Main  	at org.chromium.net.CronetTestBase.runTest(CronetTestBase.java:147)
C  221.290s Main  	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:191)
C  221.290s Main  	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:176)
C  221.290s Main  	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:554)
C  221.290s Main  	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1701)
C  221.290s Main  Caused by: java.lang.IllegalStateException: Engine is shut down.
C  221.290s Main  	at org.chromium.net.CronetUrlRequestContext.checkHaveAdapter(CronetUrlRequestContext.java:375)
C  221.290s Main  	at org.chromium.net.CronetUrlRequestContext.createRequest(CronetUrlRequestContext.java:133)
C  221.290s Main  	at org.chromium.net.UrlRequest$Builder.build(UrlRequest.java:248)
C  221.290s Main  	at org.chromium.net.CronetUrlRequestContextTest.testShutdown(CronetUrlRequestContextTest.java:416)
C  221.290s Main  	at java.lang.reflect.Method.invokeNative(Native Method)
C  221.290s Main  	at org.chromium.net.CronetTestBase.runTest(CronetTestBase.java:144)
C  221.290s Main  	... 9 more

This flake just happened on a different builder as well.

Current url: https://build.chromium.org/p/chromium.android/builders/Android%20Cronet%20Builder/builds/1611/steps/Instrumentation%20test%20CronetTestInstrumentation/logs/stdio

Miriam, if you want to tackle this bug, I'd be happy to do the review.
I think we should surround the access of mUrlRequestContextAdapter with mLock, like the following:

synchronized(mLock) {
        mUrlRequestContextAdapter = nativeCreateRequestContextAdapter(
                createNativeUrlRequestContextConfig(builder.getContext(), builder));
        if (mUrlRequestContextAdapter == 0) {
            throw new NullPointerException("Context Adapter creation failed.");
        }
}

We can submit this fix, and let it run on the bots for a few days. Given how frequent this flake occurs, we should be able to verify just by just waiting a few days.

 Issue 597375  has been merged into this issue.

I will upload a CL.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/bcdf8741bf408d9c2bbbc96c22d0406d7e9fc7ec

commit bcdf8741bf408d9c2bbbc96c22d0406d7e9fc7ec
Author: xunjieli <xunjieli@chromium.org>
Date: Mon Mar 28 17:13:14 2016

Synchronize native adapter access in CronetUrlRequestContext

BUG= 596929 

Review URL: https://codereview.chromium.org/1827883004

Cr-Commit-Position: refs/heads/master@{#383509}

[modify] https://crrev.com/bcdf8741bf408d9c2bbbc96c22d0406d7e9fc7ec/components/cronet/android/java/src/org/chromium/net/CronetUrlRequestContext.java

This just showed up again: https://build.chromium.org/p/chromium.android/builders/Android%20Cronet%20Data%20Reduction%20Proxy%20Builder/builds/1629

That's too bad. I don't have any guess why the test fails then :\

Another failure: https://build.chromium.org/p/chromium.android/builders/Android%20Cronet%20Data%20Reduction%20Proxy%20Builder/builds/1725

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ad624b50e23aa5ccd7844747b2cb38bd0e761ccd

commit ad624b50e23aa5ccd7844747b2cb38bd0e761ccd
Author: xunjieli <xunjieli@chromium.org>
Date: Tue Apr 12 14:59:51 2016

Mark CronetUrlRequestContextTest#testShutdown as flaky

TBR=pauljensen@chromium.org

BUG= 596929 

Review URL: https://codereview.chromium.org/1884563002

Cr-Commit-Position: refs/heads/master@{#386686}

[modify] https://crrev.com/ad624b50e23aa5ccd7844747b2cb38bd0e761ccd/components/cronet/android/test/javatests/src/org/chromium/net/CronetUrlRequestContextTest.java

We should remove failed attempt to fix and fix it for real.

Yesterday I was working on a unit test and found weirdness when I don't put @OnlyRunNativeCronet

Should we add this annotation to this test and try narrow down the problem?

I think that theoretically it should behave the same way for native and java implementation. What kind of weirdness did you see with java engine?

Moving this nonessential bug to the next milestone.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

This doesn't seem essential. I am removing the milestone and moving it down to P3

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/772024836ca4c86c35176d5146359f67ee12077a

commit 772024836ca4c86c35176d5146359f67ee12077a
Author: kapishnikov <kapishnikov@chromium.org>
Date: Thu Aug 11 16:19:03 2016

Re-enable flaky tests

Re-enable flaky tests that were disabled in March to see if they are
still flaky and collect logs (the old logs have been deleted already).

Re-enabled tests:
CronetUrlRequestContextTest#testShutdownAfterError
CronetUrlRequestContextTest#testShutdown
CronetUrlRequestTest#testFailures

BUG= 635025 , 635026 , 596929 

Review-Url: https://codereview.chromium.org/2239743002
Cr-Commit-Position: refs/heads/master@{#411347}

[modify] https://crrev.com/772024836ca4c86c35176d5146359f67ee12077a/components/cronet/android/test/javatests/src/org/chromium/net/CronetUrlRequestContextTest.java
[modify] https://crrev.com/772024836ca4c86c35176d5146359f67ee12077a/components/cronet/android/test/javatests/src/org/chromium/net/CronetUrlRequestTest.java

There was a new test failure: https://build.chromium.org/p/chromium.android/builders/Android%20Cronet%20ARMv6%20Builder/builds/2819/

From the logcat:
4affc:  08-12 17:31:37.159  8582  8595 I UrlRequestFactory: Using network stack: Cronet/54.0.2828.0@201e95b3
4affc:  08-12 17:31:37.179  8582  8605 F chromium: [0812/173137:FATAL:url_request_context.cc(108)] Check failed: false. Leaked 1 URLRequest(s). First URL: http://127.0.0.1:54466/echo?status=200.
4affc:  08-12 17:31:37.179  8582  8605 F chromium: #00 0x75658e1f /data/app-lib/org.chromium.net-1/libcronet_tests.so+0x0004ee1f
...
4affc:  08-12 17:31:37.179  8582  8605 F libc    : Fatal signal 6 (SIGABRT) at 0x00002186 (code=-6), thread 8605 (network)
4affc:  08-12 17:31:37.279   174   174 I DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***

Here's another failure:
https://build.chromium.org/p/chromium.android/builders/Android%20Cronet%20KitKat%20Builder/builds/62/steps/cronet_test_instrumentation_apk/logs/stdio

4b976:  08-13 03:15:04.329  8382  8396 I UrlRequestFactory: Using network stack: Cronet/54.0.2828.0@bf8b2660
4b976:  08-13 03:15:04.339  8382  8406 F chromium: [0813/031504:FATAL:url_request_context.cc(108)] Check failed: false. Leaked 1 URLRequest(s). First URL: http://127.0.0.1:54499/echo?status=200.

This is showing up a lot on the bots.  It's shown up several times that I've ignored.  Here's an example in testShutdownAfterError:
https://build.chromium.org/p/chromium.android/builders/Android%20Cronet%20Lollipop%20Builder/builds/337/steps/cronet_test_instrumentation_apk/logs/stdio

41bd9:  08-13 07:43:41.354 10607 10626 I UrlRequestFactory: Using network stack: Cronet/54.0.2829.0@89331165
41bd9:  08-13 07:43:41.367 10607 10638 F chromium: [0813/074341:FATAL:url_request_context.cc(108)] Check failed: false. Leaked 1 URLRequest(s). First URL: http://mock.failed.request/-2.

While looking at the test I found a couple of issues:

1. There is a race condition that prevents the test from executing completly depending on CPU schedulling:
https://bugs.chromium.org/p/chromium/issues/detail?id=637986

2. The test is supposed to run both with the Java and Cronet native implementations. However, it runs twice with the Cronet implementation and never with the Java one:
https://bugs.chromium.org/p/chromium/issues/detail?id=637972

3. If the test ran with the Java implementation, it would fail since JavaCronetEngine does not implement all functionality that the test expects:
https://bugs.chromium.org/p/chromium/issues/detail?id=637979

Just to clarify: None of those three issues should be causing the failures we're seeing, correct?

Should we disable the test while we work on a fix? There are quite a few flaky runs now.

That is correct. The issues are probably not causing the test failure.
I will annotate the test with @OnlyRunNativeCronet and add some extra logging. After the first failure, we can disable the test.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1c83f45c207fbcba48a65a7c87a7eea741f100a4

commit 1c83f45c207fbcba48a65a7c87a7eea741f100a4
Author: kapishnikov <kapishnikov@chromium.org>
Date: Tue Aug 16 17:25:15 2016

Instrument flaky tests in CronetUrlRequestContextTest

BUG= 635025 , 596929 

Review-Url: https://codereview.chromium.org/2247933003
Cr-Commit-Position: refs/heads/master@{#412273}

[modify] https://crrev.com/1c83f45c207fbcba48a65a7c87a7eea741f100a4/components/cronet/android/test/javatests/src/org/chromium/net/CronetUrlRequestContextTest.java

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1d5bef478ace1273acb57a4342be9558e0b7daaa

commit 1d5bef478ace1273acb57a4342be9558e0b7daaa
Author: kapishnikov <kapishnikov@chromium.org>
Date: Thu Aug 18 17:46:57 2016

Fix flaky tests in CronetUrlRequestContextTest

1. Fix of  Bug 635025  & 596929. When
   CronetUrlRequestContextTest#testShutdown() is run multiple times, a
   race condition is created between the test thread and the executor
   threads. When the second run starts, the test replaces the value of
   mTestFramework with a new CronetEngine value; however,  the executor
   thread may still be executing ShutdownTestUrlRequestCallback from the
   previous run, which also references the same shared mTestFramework.
   If the test thread replaces the value of mTestFramework while
   onSucceeded() is still being executed, the callback may shut down the
   wrong engine. The same reasoning applies to testShutdownAfterError().
   The reason why the test was executed multiple times is due to
   http://crbug/637972.
2. Fix of  Bug 637986 . Block the test thread until the request callback
   finishes its work. See changes in
   ShutdownTestUrlRequestCallback#mCallbackCompletionBlock
3. At the end of the test, shut down the callback executor, to release
   the underlying thread.
4. Removed mTestFramework member variable to avoid similar issues in the
   future.

BUG= 596929 , 635025 , 637986 

Review-Url: https://codereview.chromium.org/2254043002
Cr-Commit-Position: refs/heads/master@{#412885}

[modify] https://crrev.com/1d5bef478ace1273acb57a4342be9558e0b7daaa/components/cronet/android/test/javatests/src/org/chromium/net/CronetUrlRequestContextTest.java