Example: https://build.chromium.org/p/chromium.chromiumos/builders/Linux%20ChromiumOS%20Tests%20(1)/builds/27966/steps/browser_tests%20on%20Ubuntu-12.04/logs/PageLoadMetricsBrowserTest.NoStatePrefetchObserverCacheable

Flakiness dashboard link: http://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=browser_tests&tests=NoStatePrefetch

I can see a few flakes on the dashboard but could not reproduce locally in a couple hundreds runs.

PageLoadMetricsBrowserTest.NoStatePrefetchObserverNoStore is similarly flaky, with a similar error.

And actually most of the PageLoadMetricsBrowserTests are flaky with a similar error, so I would think the flakiness is in the metrics code rather than the prefetch code.

Can you provide the list of tests that you are seeing flakes for, and the environment in which you're seeing those flakes?

Yes.

I'm looking at the flakiness dshboard for all those tests:
http://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=browser_tests&tests=%5EPageLoadMetricsBrowserTest

I can see a few FAIL-PASS for many tests (blue rectangle), with an error related to histograms not found or not updated.

A typical failure looks like:

[ RUN      ] PageLoadMetricsBrowserTest.DocumentWriteBlock
[27862:27862:1023/183046:WARNING:chrome_browser_main_chromeos.cc(340)] Running as stub user with profile dir: test-user
[27862:27862:1023/183046:WARNING:audio_manager.cc(317)] Multiple instances of AudioManager detected
[27862:27862:1023/183046:WARNING:audio_manager.cc(278)] Multiple instances of AudioManager detected
Xlib:  extension "RANDR" missing on display ":9".
[27862:27983:1023/183046:ERROR:drive_integration_service.cc(120)] /tmp should have been created as clear.
[27862:28006:1023/183046:WARNING:local_extension_cache.cc(259)] Extensions will not be installed from update URLs until /tmp/.org.chromium.Chromium.W7UPdZ/d6LXJ7H/stub_device_local_extension_cache/.initialized exists.
[27862:27985:1023/183046:WARNING:freezer_cgroup_process_manager.cc(59)] Cgroup freezer does not exist or is not writable. Unable to freeze renderer processes.
A Parser-blocking, cross-origin script, http://localhost:8000/page_load_metrics/empty.js, is invoked via document.write. This may be blocked by the browser if the device has poor network connectivity. See https://www.chromestatus.com/feature/5718547946799104 for more details.
A Parser-blocking, cross-origin script, http://localhost:8000/page_load_metrics/empty.js, is invoked via document.write. This may be blocked by the browser if the device has poor network connectivity. See https://www.chromestatus.com/feature/5718547946799104 for more details.
[27862:27862:1023/183047:INFO:CONSOLE(7)] "A Parser-blocking, cross-origin script, http://localhost:8000/page_load_metrics/empty.js, is invoked via document.write. This may be blocked by the browser if the device has poor network connectivity. See https://www.chromestatus.com/feature/5718547946799104 for more details.", source: http://127.0.0.1:47407/page_load_metrics/document_write_script_block.html (7)
[27862:27862:1023/183047:INFO:CONSOLE(7)] "A Parser-blocking, cross-origin script, http://localhost:8000/page_load_metrics/empty.js, is invoked via document.write. This may be blocked by the browser if the device has poor network connectivity. See https://www.chromestatus.com/feature/5718547946799104 for more details.", source: http://127.0.0.1:47407/page_load_metrics/document_write_script_block.html (7)
[27862:28057:1023/183047:WARNING:embedded_test_server.cc(201)] Request not handled. Returning 404: /favicon.ico
../../base/test/histogram_tester.cc:79: Failure
Value of: 0
Expected: count
Which is: 1
Histogram "PageLoad.Clients.DocWrite.Block.PaintTiming.ParseStartToFirstContentfulPaint" does not exist.



I can see the following tests being affected:
AbortClientRedirect
AbortClose
BadXhtml
DocumentWriteBlock
DocumentWriteReload
FirstMeaningfulPaintNotRecorded
NoStatePrefetchObserverCacheable
NoStatePrefetchObserverNoStore
PreloadDocumentWrite


There are also timeouts, but the list of affected test is a bit different, and it's probably out of scope of this bug.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2ce641d6fa80fc0a9ea2fe1484f6040224e2a6a3

commit 2ce641d6fa80fc0a9ea2fe1484f6040224e2a6a3
Author: droger <droger@chromium.org>
Date: Thu Oct 27 16:12:23 2016

Fix flakiness in page_load_metrics_browser_test

This is a speculative fix for test failures that happen
because histograms are not updated.

This CL catches the case where the IPC updating the
histogram would still be in flight when the histograms
are checked.
If the histograms are missing, there is now a mechanism
to wait for the IPC to arrive.

BUG= 657022 

Review-Url: https://codereview.chromium.org/2449863002
Cr-Commit-Position: refs/heads/master@{#428039}

[modify] https://crrev.com/2ce641d6fa80fc0a9ea2fe1484f6040224e2a6a3/chrome/browser/page_load_metrics/page_load_metrics_browsertest.cc

Here is a flake from Nov 4th after this patch

https://build.chromium.org/p/chromium.win/builders/Win7%20Tests%20%28dbg%29%281%29/builds/54379

Yes, the fix did not work. There are flakes in other tests too, although they don't make the test red because they pass on retries.

We should probably revert the fix since it did not work.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/da3cfdb3e1f0b69e10aaadc30ff05811c26f46af

commit da3cfdb3e1f0b69e10aaadc30ff05811c26f46af
Author: droger <droger@chromium.org>
Date: Mon Nov 14 19:09:09 2016

Revert "Fix flakiness in page_load_metrics_browser_test"

This reverts commit 2ce641d6fa80fc0a9ea2fe1484f6040224e2a6a3.

The fix seems to have failed, because the tests are still flaky.
Remove the code, as the additional complexity is not worth it.

BUG= 657022 

Review-Url: https://codereview.chromium.org/2479293003
Cr-Commit-Position: refs/heads/master@{#431886}

[modify] https://crrev.com/da3cfdb3e1f0b69e10aaadc30ff05811c26f46af/chrome/browser/page_load_metrics/page_load_metrics_browsertest.cc

This fails flakily with the the following error:

[ RUN      ] PageLoadMetricsBrowserTest.NoStatePrefetchObserverCacheable
Xlib:  extension "RANDR" missing on display ":99".
Xlib:  extension "RANDR" missing on display ":99".
[14846:14846:0207/024146.891811:WARNING:audio_manager.cc(321)] Multiple instances of AudioManager detected
[14846:14846:0207/024146.891840:WARNING:audio_manager.cc(278)] Multiple instances of AudioManager detected
[14846:14846:0207/024146.912595:WARNING:password_store_factory.cc(247)] Using basic (unencrypted) store for password storage. See https://chromium.googlesource.com/chromium/src/+/master/docs/linux_password_storage.md for more information about password storage options.
[14846:14906:0207/024147.057041:WARNING:embedded_test_server.cc(219)] Request not handled. Returning 404: /favicon.ico
../../base/test/histogram_tester.cc:72: Failure
Value of: 0
Expected: count
Which is: 1
Histogram "Prerender.none_PrefetchTTFCP.Reference.Cacheable.Visible" does not exist.
[  FAILED  ] PageLoadMetricsBrowserTest.NoStatePrefetchObserverCacheable, where TypeParam =  and GetParam() =  (295 ms)

The failure happened on linux as well:
https://build.chromium.org/p/chromium.linux/builders/Linux%20Tests/builds/51730/steps/site_per_process_browser_tests/logs/PageLoadMetricsBrowserTest.NoStatePrefetchObserverCacheable

I see 5 flakes on Linux recently and will disable the test for Linux.

mattcary@ can you please take a look at this as well? I wonder whether your CL https://codereview.chromium.org/2423383002 from a few days ago is to blame.

When I looked at these flakes, this kind of missing histogram failures were happening across all tests of this suite. See comment #9 above for example.

I'm fine with disabling the test, but my impression is that the issue comes from the test setup in general and not the particular code that is tested here.

Sometimes the test just times out while other times it throws the error from comment 14.

Example for timeout
https://chromium-swarm.appspot.com/task?id=342d3af0200e7210&refresh=10&show_raw=1
[ RUN      ] PageLoadMetricsBrowserTest.NoStatePrefetchObserverCacheable
Xlib:  extension "RANDR" missing on display ":99".
Xlib:  extension "RANDR" missing on display ":99".
[27660:27660:0206/091441.745222:WARNING:audio_manager.cc(321)] Multiple instances of AudioManager detected
[27660:27660:0206/091441.745259:WARNING:audio_manager.cc(278)] Multiple instances of AudioManager detected
[27660:27660:0206/091441.768744:WARNING:password_store_factory.cc(247)] Using basic (unencrypted) store for password storage. See https://chromium.googlesource.com/chromium/src/+/master/docs/linux_password_storage.md for more information about password storage options.
[514/1066] PageLoadMetricsBrowserTest.NoStatePrefetchObserverCacheable (TIMED OUT)

This one seems to be very related:

 RUN      ] PageLoadMetricsBrowserTest.NoStatePrefetchObserverNoStore
Xlib:  extension "RANDR" missing on display ":99".
Xlib:  extension "RANDR" missing on display ":99".
[4553:4553:0206/081926.758601:WARNING:audio_manager.cc(321)] Multiple instances of AudioManager detected
[4553:4553:0206/081926.758649:WARNING:audio_manager.cc(278)] Multiple instances of AudioManager detected
[4553:4553:0206/081926.805946:WARNING:password_store_factory.cc(247)] Using basic (unencrypted) store for password storage. See https://chromium.googlesource.com/chromium/src/+/master/docs/linux_password_storage.md for more information about password storage options.
[4553:4687:0206/081927.083980:WARNING:embedded_test_server.cc(219)] Request not handled. Returning 404: /favicon.ico
../../base/test/histogram_tester.cc:72: Failure
Value of: 0
Expected: count
Which is: 1
Histogram "Prerender.none_PrefetchTTFCP.Reference.NoStore.Visible" does not exist.

I'll help droger@ look into it.

Thanks a lot. https://codereview.chromium.org/2682593004 is in the CQ.

https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=site_per_process_browser_tests&tests=PageLoadMetricsBrowserTest. is the link to the flakiness dashboard.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/af410cfc843ea958a070ffb864366e43129f5755

commit af410cfc843ea958a070ffb864366e43129f5755
Author: battre <battre@chromium.org>
Date: Tue Feb 07 15:20:36 2017

Disable flaky PageLoadMetricsBrowserTests on Linux

Disable PageLoadMetricsBrowserTest.NoStatePrefetchObserverNoStore and
PageLoadMetricsBrowserTest.NoStatePrefetchObserverCacheable for both timing out
and/or failing with expectations.

TBR=droger@chromium.org,bmcquade@chromium.org
BUG= 657022 

Review-Url: https://codereview.chromium.org/2682593004
Cr-Commit-Position: refs/heads/master@{#448624}

[modify] https://crrev.com/af410cfc843ea958a070ffb864366e43129f5755/chrome/browser/page_load_metrics/page_load_metrics_browsertest.cc

The flakiness is really hard to reproduce.

David: do you know if the call to NavigateToUntrackedUrl() is put in to effectively block until the FCP happens? I wonder if it's not sufficient. I can reproduce the flakiness if I remove that call  (that is, it fails one of 4 runs).

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/20af066f183963806cf7f43837bd6fdacbe6d5ec

commit 20af066f183963806cf7f43837bd6fdacbe6d5ec
Author: mattcary <mattcary@chromium.org>
Date: Mon Feb 20 11:07:58 2017

Prerender: fix flaky page load metrics test (First Contentful Paint).

Adds a waiter for the TimingUpdated IPC. This is a little subtle, as the waiter
may fire before the true PageLoadMetrics observer sees the IPC, hence the waiter
handler needs to be requeued.

BUG= 657022 

Review-Url: https://codereview.chromium.org/2698813005
Cr-Commit-Position: refs/heads/master@{#451594}

[modify] https://crrev.com/20af066f183963806cf7f43837bd6fdacbe6d5ec/chrome/browser/page_load_metrics/page_load_metrics_browsertest.cc

This appears to fix the flakes for the following:

BadXhtml
DocumentWriteBlock
DocumentWriteReload
NoStatePrefetchObserverCacheable
NoStatePrefetchObserverNoStore
PreloadDocumentWrite

Thanks for fixing this!

Ah, we spoke too soon, a couple are still flaking a bit. Looking into it...

The new flakes are all timeouts rather than the failures that seem to have been fixed by my CL. Closing for now; the timeouts are not reproducible and the affected tests are flaking less than other things in the same suite.

If the problem worsens we can open a new bug.

Unfortunately, I have found that some of PageLoadMetricsBrowserTests are still flaky on Linux under some conditions.
I created https://bugs.chromium.org/p/chromium/issues/detail?id=697761

Thanks atuchin@. I would not have expected this fix to affect the CSSTiming test.

Thank you for creating a new bug!

Yes, this fix doesn't affect to CSSTiming test.
But for me it looks like it's a similar problem with that test.
Do you think it can be fixed in a similar way?

Possibly. Let's move the conversation over to that bug.