Hum, it seems slightly flaky without virtual/navigation-mojo-response too.
https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=webkit_layout_tests&tests=http%2Ftests%2Fserviceworker%2Fchromium%2Fmemory-cache.html

I just saw a local flake locally without virtual/navigation, so I guess it's not the fault of virtual/navigation.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/35d9f2983f317cf2cbba652bc3c1b175b6bdfeb7

commit 35d9f2983f317cf2cbba652bc3c1b175b6bdfeb7
Author: Yoshifumi Inoue <yosin@chromium.org>
Date: Tue Feb 20 06:19:52 2018

Make virtual/navigation-mojo-response/http/tests/serviceworker/chromium/memory-cache.html flaky

This patch marks "memory-cache.html" flaky.

NOTRY=true
TBR=alken@chromium.org, horo@chromium.org, shimazu@chromium.org

Bug:  813468 
Change-Id: I8c49b37caca7d328eb31fb65fa2ae6ab074a9636
Reviewed-on: https://chromium-review.googlesource.com/925871
Commit-Queue: Yoshifumi Inoue <yosin@chromium.org>
Reviewed-by: Yoshifumi Inoue <yosin@chromium.org>
Cr-Commit-Position: refs/heads/master@{#537727}
[modify] https://crrev.com/35d9f2983f317cf2cbba652bc3c1b175b6bdfeb7/third_party/WebKit/LayoutTests/TestExpectations

 Issue 815050  has been merged into this issue.

I'll try to repro on Linux.

As far as I checked, this flakiness does not (or rarely) happens on linux/win buildbot. So might be Mac specific.

https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_chromium_rel_ng/33927 shows some flakiness on Linux (I think I remember seeing some failures on linux_blink_trusty_rel last week that were affecting WPT imports too).

My bisect attempt didn't show a plausible result:
git bisect start
# bad: [fe4a1775a29fa1918d74dbaf0c5310acb2ccd85d] Expose occlusion in WebContents and WebContentsObserver.
git bisect bad fe4a1775a29fa1918d74dbaf0c5310acb2ccd85d
# good: [af3170e5f40955165ffea7a5dd0d8032a9e012f5] Move ownership of gpu texture resources to ResourcePool
git bisect good af3170e5f40955165ffea7a5dd0d8032a9e012f5
# good: [8bc20d8233df8daf60ae5fa4f5588c71bdce7fae] Add a sampling mode to out of process heap profiling.
git bisect good 8bc20d8233df8daf60ae5fa4f5588c71bdce7fae
# good: [3904dc2ec06653b0b7281580c6d1662aeb9d87e0] Rename cc animation classes to be consistent with blink classes and spec
git bisect good 3904dc2ec06653b0b7281580c6d1662aeb9d87e0
# bad: [7f56fd822d1edb7400af10da9fd66788b6e91542] Do not get TaskScheduler field trial params from command line in renderers.
git bisect bad 7f56fd822d1edb7400af10da9fd66788b6e91542
# good: [de996bafd4753a7482b4ef47df2a5fce7256309d] Simplifying perf_data_generator
git bisect good de996bafd4753a7482b4ef47df2a5fce7256309d
# good: [885dd7990ab735bf7250b3b29edde798ca04c65c] Set render_frame_id of resouce request to MSG_ROUTING_NONE in SignedExchangeCertFetcher
git bisect good 885dd7990ab735bf7250b3b29edde798ca04c65c
# bad: [debce06e67728c92cbebb2eb7b0072f4c040b6aa] Only allow multiple action buttons in Custom Tabs for trusted intents.
git bisect bad debce06e67728c92cbebb2eb7b0072f4c040b6aa
# bad: [5df5afe227fae9012ed80c214836a851a4346b5a] [tab-under] Remove comment about navigating in the background
git bisect bad 5df5afe227fae9012ed80c214836a851a4346b5a
# good: [8e6c2940db6b889e84773101d072d859d283d0dd] Update test/data/htxg/README
git bisect good 8e6c2940db6b889e84773101d072d859d283d0dd
# good: [05db906e922ecc5cc6d2c2141a324bc10465ca9b] Increase AudioArray alignment for better AVX optimization
git bisect good 05db906e922ecc5cc6d2c2141a324bc10465ca9b
# bad: [4b6c4188b42aace5b8020ee8f906a3057e7480b0] Reland "content: Enable CompositorImageAnimation by default."
git bisect bad 4b6c4188b42aace5b8020ee8f906a3057e7480b0

I tried reverting the CompositorImageAnimation thing but still see the failure.

I think the test might also just be overly aggressive, I don't know if it's really guaranteed whether something will go into the memory cache or not. What we really want to make sure for this test is that the SW-controlled load and the non-SW-controlled load don't use the same memory cache.

Actually I just forgot to compile. After reverting CompositorImageAnimation the test passes 100 times. Before reverting it would fail within the 10th try.

khushalsagar, chrishtr: this test tests whether resources come from the memory cache. The CompositorImageAnimation change seems to have made the test flaky. Can CompositorImageAnimation affect the timing of when a resource enters the memory cache, or whether it enters the cache at all?

See the revert patch at https://chromium-review.googlesource.com/c/chromium/src/+/939941, this seems to fix the test.

If so, and this is expected, I think we should just relax the test.

Could you verify if running the test with threaded compositing also makes it flaky? That change ensured that layout tests went through rendering phases in the compositor which can change timing of things and result in cc dispatching more callbacks that can affect the RendererScheduler. In threaded mode, all this should already happen.

Sure, can you tell me how to run with threaded compositing?

Just use --additional-driver-flag=--enable-threaded-compositing.

Yep, I confirm that with the revert patch in, --additional-driver-flag="--enable-threaded-compositing" causes the test to fail.

I guess this is WAI. Not sure how to fix the test. We could remove the strict memory cache checking but it could make the test meaningless for its original intended purpose about cache poisoning: if its testing before the resources go into the memory cache it will trivially be true that the cache wasn't poisoned.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/bc0fc7083e5466f5227bc868c2543957672def9a

commit bc0fc7083e5466f5227bc868c2543957672def9a
Author: Matt Falkenhagen <falken@chromium.org>
Date: Mon Mar 05 03:01:06 2018

service worker: Deflake memory-cache.html.

CompositorImageAnimation changed some timings and the resource was not
yet cached when the test was expecting it to be. I've rearranged the
test and it seems to pass reliably now on Linux: it passes 100
iterations whereas it would fail within 10 iterations previously.

R=nhiroki

Bug:  813468 
Change-Id: I5b76353f90e728981922c325486fa1700e85f73e
Reviewed-on: https://chromium-review.googlesource.com/945708
Commit-Queue: Matt Falkenhagen <falken@chromium.org>
Reviewed-by: Hiroki Nakagawa <nhiroki@chromium.org>
Cr-Commit-Position: refs/heads/master@{#540777}
[modify] https://crrev.com/bc0fc7083e5466f5227bc868c2543957672def9a/third_party/WebKit/LayoutTests/TestExpectations
[modify] https://crrev.com/bc0fc7083e5466f5227bc868c2543957672def9a/third_party/WebKit/LayoutTests/http/tests/serviceworker/chromium/memory-cache.html