Running these tests on Linux, under ASAN, I just saw a similar flake:

../../components/cronet/stale_host_resolver_unittest.cc:468: Failure
Expected equality of these values:
  test_case.error
    Which is: 0
  resolve_error()
    Which is: -21
4
../../components/cronet/stale_host_resolver_unittest.cc:473: Failure
Expected equality of these values:
  expected
    Which is: "2.2.2.2"
  resolve_addresses()[0].ToStringWithoutPort()
    Which is: "3.3.3.3"
4

Are these tests relying on real-time clock values, rather than running against a controlled test clock?

This test (and presumably others in this suite) is/are inherently flaky because they do not use a mock clock, so thread scheduling delays can alter their behaviour -> bumping to P1 since this will affect CQ, try-bots, etc stability.

Specifically the test suite's CreateCacheEntry inserts a cache entry based on the requested Time-To-Live, relative to Now(), then does some more work depending on the test, then calls Resolve(). If a delay occurs between CreateCacheEntry() and Resolve() then the test may observe the wrong results; this can occur if e.g. the host running the test is under heavy load.

You can trivially repro this failure-mode by introducing a one-second Sleep() between the two calls.

Looking at net/dns/host_cache.cc it uses base::Time::Now() and base::TimeTicks::Now() directly, for example:

    base::TimeTicks expiration_time =
        base::TimeTicks::Now() -
        (base::Time::Now() - base::Time::FromInternalValue(time_internal));

IIUIC this means that we should change not only test but also underlying implementation.

Re #7: There were "seams" added to base::Time/TimeTicks/ThreadTicks::Now() (see https://cs.chromium.org/chromium/src/base/time/time_override.h) that allow time to be mocked without dependency injection. The seams apply process-wide, though, so perhaps are unsuitable.

 Issue 836022  has been merged into this issue.

Can we disable this test until the flakiness has been fixed?

Disabling test until flakiness is fixed sgtm.

Test was disabled in r553020

 Issue 836106  has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b99e7da3ec5b3adc77d2512ec0bfacca65e5a019

commit b99e7da3ec5b3adc77d2512ec0bfacca65e5a019
Author: Misha Efimov <mef@chromium.org>
Date: Wed May 30 16:59:02 2018

[Cronet] Fix flaky StaleHostResolverTest.StaleUsability test.

- Use base::SimpleTestTickClock to provide deterministic time.
- Use net::test::MockNetworkChangeNotifier to avoid notifications from real NCN.

Bug:  792173 
Cq-Include-Trybots: master.tryserver.chromium.android:android_cronet_tester;master.tryserver.chromium.mac:ios-simulator-cronet
Change-Id: I6e9b20363452a1e665b133960167551808486b48
Reviewed-on: https://chromium-review.googlesource.com/1067958
Reviewed-by: Paul Jensen <pauljensen@chromium.org>
Commit-Queue: Misha Efimov <mef@chromium.org>
Cr-Commit-Position: refs/heads/master@{#562874}
[modify] https://crrev.com/b99e7da3ec5b3adc77d2512ec0bfacca65e5a019/components/cronet/stale_host_resolver.cc
[modify] https://crrev.com/b99e7da3ec5b3adc77d2512ec0bfacca65e5a019/components/cronet/stale_host_resolver.h
[modify] https://crrev.com/b99e7da3ec5b3adc77d2512ec0bfacca65e5a019/components/cronet/stale_host_resolver_unittest.cc
[modify] https://crrev.com/b99e7da3ec5b3adc77d2512ec0bfacca65e5a019/net/dns/host_cache.cc
[modify] https://crrev.com/b99e7da3ec5b3adc77d2512ec0bfacca65e5a019/net/dns/host_cache.h
[modify] https://crrev.com/b99e7da3ec5b3adc77d2512ec0bfacca65e5a019/net/dns/host_resolver_impl.cc
[modify] https://crrev.com/b99e7da3ec5b3adc77d2512ec0bfacca65e5a019/net/dns/host_resolver_impl.h