mek@, could you take look?

These failures all happened on android_n5x_swarming_rel, which has been unhealthy recently, see  issue 672382 . So it's possible that there's nothing wrong with this particular test.

Looking at the failures, they don't seem to be related to the bot troubles after all.
I can't find any related recent changes though :(

+iclelland who also owns origin_trials.

Those tests are sensitive to the time returned by base::Time::Now. Looking at the results from ValidateValidToken, which is the simplest test, it's returning a status code saying that the token is expired (3 vs 0)

(https://build.chromium.org/p/tryserver.chromium.android/builders/android_n5x_swarming_rel/builds/81375/steps/content_unittests%20%28with%20patch%29%20on%20Android/logs/TrialTokenValidatorTest.ValidateValidToken)

The token itself has an expiry timestamp of 2000000000, which should be ~17 years in the future, unless something is seriously wrong with the system time on those devices.

I'll add an assert that Now() is < 2000000000, and see if that gets triggered in the swarm.

If that's the case, perhaps we want to mock the clock for tests -- it'll involve passing a clock to the validator constructor, but shouldn't be difficult.

That sounds reasonable, thanks!
Could you own this? If it'll take more than a day or two, we should probably disable those tests in the meantime.

We can do that -- If you're okay with it as sheriff, I'd like to commit https://codereview.chromium.org/2559973003 and see if if flakes one more time first, just to catch the error message.

SGTM, though I'll only be sheriff for another half hour or so ;)

Fair enough; I'll sync with flackr about it after that :)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1523ca526b1c0519fd5e38c6b81528164ae92033

commit 1523ca526b1c0519fd5e38c6b81528164ae92033
Author: iclelland <iclelland@chromium.org>
Date: Thu Dec 08 17:07:53 2016

Assert that the system clock is between 1e9 and 2e9 for Origin Trial token validation tests

This is investigating the cause of flaky tests on the Android swarming
bots.

BUG=672294
TBR=treib@chromium.org

Review-Url: https://codereview.chromium.org/2559973003
Cr-Commit-Position: refs/heads/master@{#437266}

[modify] https://crrev.com/1523ca526b1c0519fd5e38c6b81528164ae92033/content/common/origin_trials/trial_token_validator_unittest.cc

[ RUN      ] TrialTokenValidatorTest.ValidateRequestMultipleHeaders
I  108.429s run_tests_on_device(00b9ed8d80c89937)  ../../content/common/origin_trials/trial_token_validator_unittest.cc:167: Failure
I  108.429s run_tests_on_device(00b9ed8d80c89937)  Expected: (base::Time::Now()) < (base::Time::FromDoubleT(kFutureTimestamp)), actual: 2047-08-19 22:29:49.162 UTC vs 2033-05-18 03:33:20.000 UTC
I  108.429s run_tests_on_device(00b9ed8d80c89937)  [  FAILED  ] TrialTokenValidatorTest.ValidateRequestMultipleHeaders (0 ms)

Seems like the clock on the bot is indeed wrong and it affects the test.

Filed infra issue at  https://crbug.com/672843  for the system time. I'm going to rewrite some of the validator so that we can mock the system clock in unittests, and disable the tests in the meantime.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6f8aac8277ea14b19c533303288ca1df8d4c8bef

commit 6f8aac8277ea14b19c533303288ca1df8d4c8bef
Author: iclelland <iclelland@chromium.org>
Date: Fri Dec 09 16:08:37 2016

Disable flaky origin trial tests on Android

Several tests are flaking on Android swarming bots as they rely on the
system time being correct. Disabling them until we can remove the
dependency on the system clock.

BUG=672294, 672843 
TBR=asargent@chromium.org

Review-Url: https://codereview.chromium.org/2564873002
Cr-Commit-Position: refs/heads/master@{#437553}

[modify] https://crrev.com/6f8aac8277ea14b19c533303288ca1df8d4c8bef/content/common/origin_trials/trial_token_validator_unittest.cc

 Issue 672636  has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/334e05fb8a8035585a25b1ce419425ac64b3889a

commit 334e05fb8a8035585a25b1ce419425ac64b3889a
Author: iclelland <iclelland@chromium.org>
Date: Thu Aug 03 01:39:46 2017

Deflake origin trial token validator tests

This removes the dependency on the system clock of the origin trial
token validator tests. All static functions in the TrialTokenValidator
namespace which validate tokens now take a base::Time argument which is used to
compare against the tokens provided.

BUG=672294
R=chasej@chromium.org,nhiroki@chromium.org
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_n5x_swarming_rel

Review-Url: https://codereview.chromium.org/2555333006
Cr-Commit-Position: refs/heads/master@{#491601}

[modify] https://crrev.com/334e05fb8a8035585a25b1ce419425ac64b3889a/content/browser/service_worker/foreign_fetch_request_handler_unittest.cc
[modify] https://crrev.com/334e05fb8a8035585a25b1ce419425ac64b3889a/content/browser/service_worker/link_header_support.cc
[modify] https://crrev.com/334e05fb8a8035585a25b1ce419425ac64b3889a/content/browser/service_worker/service_worker_version.cc
[modify] https://crrev.com/334e05fb8a8035585a25b1ce419425ac64b3889a/content/browser/service_worker/service_worker_version.h
[modify] https://crrev.com/334e05fb8a8035585a25b1ce419425ac64b3889a/content/common/origin_trials/trial_token_validator.cc
[modify] https://crrev.com/334e05fb8a8035585a25b1ce419425ac64b3889a/content/common/origin_trials/trial_token_validator.h
[modify] https://crrev.com/334e05fb8a8035585a25b1ce419425ac64b3889a/content/common/origin_trials/trial_token_validator_unittest.cc
[modify] https://crrev.com/334e05fb8a8035585a25b1ce419425ac64b3889a/content/renderer/origin_trials/web_trial_token_validator_impl.cc