We have tests for how chrome renders webpages, but not for how the UI is rendered. With a lot of work happening on the New Tab Page (which is a very visible part of chrome on android) including a potential large refactor of the code, tests that prevent UI regressions will be essential. For example, the work that we are doing on the New Tab Page is hidden behind a flag but touches code used by the current New Tab Page and the Incognito New Tab Page. In addition, these regressions may only show up in certain configurations (such as when using a tablet in landscape).

This is indeed relevant work. What's the status? :)

I have a CL that is part way through code review (https://codereview.chromium.org/2114963002/), I'm going to pick up work on that while I'm in America.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/324e06c25c137f097e6f001b12abb54d3e8bf268

commit 324e06c25c137f097e6f001b12abb54d3e8bf268
Author: peconn <peconn@chromium.org>
Date: Thu Aug 18 19:03:35 2016

Create Java infrastructure for UI Render Tests.

Creates the first part of the infrastructure that will allow us to test for
any regressions to the Chrome on Android UI. This includes utilities to save
Views to file, an annotation (RenderTest) to identify the instrumentation
tests that use this and two examples of their use.

The aim is to follow this up with host side code that grabs the images after
the tests have been run and performs pixel comparisons on them against golden
images stored in the repository.

BUG= 614305 

Review-Url: https://codereview.chromium.org/2114963002
Cr-Commit-Position: refs/heads/master@{#412905}

[modify] https://crrev.com/324e06c25c137f097e6f001b12abb54d3e8bf268/chrome/android/java/src/org/chromium/chrome/browser/ntp/LogoBridge.java
[modify] https://crrev.com/324e06c25c137f097e6f001b12abb54d3e8bf268/chrome/android/java/src/org/chromium/chrome/browser/ntp/cards/NewTabPageAdapter.java
[add] https://crrev.com/324e06c25c137f097e6f001b12abb54d3e8bf268/chrome/android/java/src/org/chromium/chrome/browser/ntp/snippets/FakeSuggestionsSource.java
[modify] https://crrev.com/324e06c25c137f097e6f001b12abb54d3e8bf268/chrome/android/java_sources.gni
[modify] https://crrev.com/324e06c25c137f097e6f001b12abb54d3e8bf268/chrome/android/javatests/src/org/chromium/chrome/browser/ntp/NewTabPageTest.java
[add] https://crrev.com/324e06c25c137f097e6f001b12abb54d3e8bf268/chrome/android/javatests/src/org/chromium/chrome/browser/ntp/snippets/ArticleSnippetsTest.java
[modify] https://crrev.com/324e06c25c137f097e6f001b12abb54d3e8bf268/chrome/android/junit/src/org/chromium/chrome/browser/ntp/cards/NewTabPageAdapterTest.java
[modify] https://crrev.com/324e06c25c137f097e6f001b12abb54d3e8bf268/chrome/test/android/BUILD.gn
[add] https://crrev.com/324e06c25c137f097e6f001b12abb54d3e8bf268/chrome/test/android/javatests/src/org/chromium/chrome/test/util/RenderUtils.java
[add] https://crrev.com/324e06c25c137f097e6f001b12abb54d3e8bf268/chrome/test/data/android/render_tests/ArticleSnippetsTest.long_snippet.Nexus_5X.port.png
[add] https://crrev.com/324e06c25c137f097e6f001b12abb54d3e8bf268/chrome/test/data/android/render_tests/ArticleSnippetsTest.minimal_snippet.Nexus_5X.port.png
[add] https://crrev.com/324e06c25c137f097e6f001b12abb54d3e8bf268/chrome/test/data/android/render_tests/ArticleSnippetsTest.short_snippet.Nexus_5X.port.png
[add] https://crrev.com/324e06c25c137f097e6f001b12abb54d3e8bf268/chrome/test/data/android/render_tests/ArticleSnippetsTest.snippets.Nexus_5X.port.png
[add] https://crrev.com/324e06c25c137f097e6f001b12abb54d3e8bf268/chrome/test/data/android/render_tests/NewTabPageTest.fakebox.Nexus_5X.port.png
[add] https://crrev.com/324e06c25c137f097e6f001b12abb54d3e8bf268/chrome/test/data/android/render_tests/NewTabPageTest.most_visited.Nexus_5X.port.png
[add] https://crrev.com/324e06c25c137f097e6f001b12abb54d3e8bf268/chrome/test/data/android/render_tests/NewTabPageTest.new_tab_page.Nexus_5X.port.png
[add] https://crrev.com/324e06c25c137f097e6f001b12abb54d3e8bf268/chrome/test/data/android/render_tests/NewTabPageTest.new_tab_page_scrolled.Nexus_5X.port.png

It'd be worth adding tests for https://codereview.chromium.org/2270443002/.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0d5366d038448a862755eab685548c90bbbda7d5

commit 0d5366d038448a862755eab685548c90bbbda7d5
Author: peconn <peconn@chromium.org>
Date: Mon May 15 14:53:38 2017

📷 Add toggling zoom and test title to RenderTest results.

BUG= 614305 

Review-Url: https://codereview.chromium.org/2875263002
Cr-Commit-Position: refs/heads/master@{#471761}

[modify] https://crrev.com/0d5366d038448a862755eab685548c90bbbda7d5/build/android/pylib/instrumentation/render_test.html.jinja
[modify] https://crrev.com/0d5366d038448a862755eab685548c90bbbda7d5/build/android/pylib/local/device/local_device_instrumentation_test_run.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7edc11ce0dadc7444908df164a8beff41646acb5

commit 7edc11ce0dadc7444908df164a8beff41646acb5
Author: peconn <peconn@chromium.org>
Date: Wed May 17 10:04:53 2017

📷 Update Render Test goldens for 5X.

Now that Render Tests are running on the trybots, we can start using
them for real!

BUG= 614305 

Review-Url: https://codereview.chromium.org/2888493003
Cr-Commit-Position: refs/heads/master@{#472406}

[modify] https://crrev.com/7edc11ce0dadc7444908df164a8beff41646acb5/chrome/test/data/android/render_tests/ArticleSnippetsTest.long_minimal_snippet_narrow.Nexus_5X.port.png
[modify] https://crrev.com/7edc11ce0dadc7444908df164a8beff41646acb5/chrome/test/data/android/render_tests/ArticleSnippetsTest.long_snippet.Nexus_5X.port.png
[modify] https://crrev.com/7edc11ce0dadc7444908df164a8beff41646acb5/chrome/test/data/android/render_tests/ArticleSnippetsTest.long_snippet_narrow.Nexus_5X.port.png
[modify] https://crrev.com/7edc11ce0dadc7444908df164a8beff41646acb5/chrome/test/data/android/render_tests/ArticleSnippetsTest.minimal_snippet.Nexus_5X.port.png
[modify] https://crrev.com/7edc11ce0dadc7444908df164a8beff41646acb5/chrome/test/data/android/render_tests/ArticleSnippetsTest.short_minimal_snippet_narrow.Nexus_5X.port.png
[modify] https://crrev.com/7edc11ce0dadc7444908df164a8beff41646acb5/chrome/test/data/android/render_tests/ArticleSnippetsTest.short_snippet.Nexus_5X.port.png
[modify] https://crrev.com/7edc11ce0dadc7444908df164a8beff41646acb5/chrome/test/data/android/render_tests/ArticleSnippetsTest.short_snippet_narrow.Nexus_5X.port.png
[modify] https://crrev.com/7edc11ce0dadc7444908df164a8beff41646acb5/chrome/test/data/android/render_tests/ArticleSnippetsTest.snippets.Nexus_5X.port.png
[modify] https://crrev.com/7edc11ce0dadc7444908df164a8beff41646acb5/chrome/test/data/android/render_tests/ArticleSnippetsTest.snippets_narrow.Nexus_5X.port.png
[modify] https://crrev.com/7edc11ce0dadc7444908df164a8beff41646acb5/chrome/test/data/android/render_tests/NewTabPageTest.fakebox.Nexus_5X.port.png
[modify] https://crrev.com/7edc11ce0dadc7444908df164a8beff41646acb5/chrome/test/data/android/render_tests/NewTabPageTest.most_visited.Nexus_5X.port.png
[modify] https://crrev.com/7edc11ce0dadc7444908df164a8beff41646acb5/chrome/test/data/android/render_tests/NewTabPageTest.new_tab_page.Nexus_5X.port.png
[modify] https://crrev.com/7edc11ce0dadc7444908df164a8beff41646acb5/chrome/test/data/android/render_tests/NewTabPageTest.new_tab_page_scrolled.Nexus_5X.port.png

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1c831a5af48331b017f2bf3d6a07abb40f143626

commit 1c831a5af48331b017f2bf3d6a07abb40f143626
Author: peconn <peconn@chromium.org>
Date: Fri May 26 10:25:44 2017

📷 Create RenderTestRule which fails after test method finishes.

This CL:
- Removes RenderUtils.
- Creates RenderTestRule with most of the RenderUtils functionality.
- Updates render tests to RenderTestRule.
- Removes the 'regenerate-goldens' flag. This is not needed because the
  RenderTestRule will throw an Exception once the test method has
  finished, so all image renders and diffs will be saved anyway.
- Simplifies the rendering logic.

BUG= 614305 

Review-Url: https://codereview.chromium.org/2901053003
Cr-Commit-Position: refs/heads/master@{#474978}

[modify] https://crrev.com/1c831a5af48331b017f2bf3d6a07abb40f143626/build/android/pylib/instrumentation/instrumentation_test_instance.py
[modify] https://crrev.com/1c831a5af48331b017f2bf3d6a07abb40f143626/build/android/pylib/instrumentation/instrumentation_test_instance_test.py
[modify] https://crrev.com/1c831a5af48331b017f2bf3d6a07abb40f143626/build/android/test_runner.py
[modify] https://crrev.com/1c831a5af48331b017f2bf3d6a07abb40f143626/chrome/android/javatests/src/org/chromium/chrome/browser/ntp/NewTabPageTest.java
[modify] https://crrev.com/1c831a5af48331b017f2bf3d6a07abb40f143626/chrome/android/javatests/src/org/chromium/chrome/browser/ntp/snippets/ArticleSnippetsTest.java
[modify] https://crrev.com/1c831a5af48331b017f2bf3d6a07abb40f143626/chrome/test/android/BUILD.gn
[add] https://crrev.com/1c831a5af48331b017f2bf3d6a07abb40f143626/chrome/test/android/javatests/src/org/chromium/chrome/test/util/RenderTestRule.java
[delete] https://crrev.com/424fd27db211d380ddc81f09f0f35b71661e6cdf/chrome/test/android/javatests/src/org/chromium/chrome/test/util/RenderUtils.java

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/90b0ed1030a7fdcfedb95069060a9fa67ab20c9a

commit 90b0ed1030a7fdcfedb95069060a9fa67ab20c9a
Author: Peter E Conn <peconn@chromium.org>
Date: Wed Jul 05 12:30:09 2017

📷 Update Nexus 5 goldens.

Bug:  614305 
Change-Id: I829ecdd9715e1f44332aba3cb1319c63441da476
Reviewed-on: https://chromium-review.googlesource.com/558288
Reviewed-by: Bernhard Bauer <bauerb@chromium.org>
Commit-Queue: Peter Conn <peconn@chromium.org>
Cr-Commit-Position: refs/heads/master@{#484251}
[modify] https://crrev.com/90b0ed1030a7fdcfedb95069060a9fa67ab20c9a/chrome/test/data/android/render_tests/ArticleSnippetsTest.download_snippet_placeholder.Nexus_5.port.png
[modify] https://crrev.com/90b0ed1030a7fdcfedb95069060a9fa67ab20c9a/chrome/test/data/android/render_tests/ArticleSnippetsTest.download_snippet_thumbnail.Nexus_5.port.png
[modify] https://crrev.com/90b0ed1030a7fdcfedb95069060a9fa67ab20c9a/chrome/test/data/android/render_tests/ArticleSnippetsTest.long_minimal_snippet_narrow.Nexus_5.port.png
[modify] https://crrev.com/90b0ed1030a7fdcfedb95069060a9fa67ab20c9a/chrome/test/data/android/render_tests/ArticleSnippetsTest.long_snippet.Nexus_5.port.png
[modify] https://crrev.com/90b0ed1030a7fdcfedb95069060a9fa67ab20c9a/chrome/test/data/android/render_tests/ArticleSnippetsTest.long_snippet_narrow.Nexus_5.port.png
[modify] https://crrev.com/90b0ed1030a7fdcfedb95069060a9fa67ab20c9a/chrome/test/data/android/render_tests/ArticleSnippetsTest.minimal_snippet.Nexus_5.port.png
[modify] https://crrev.com/90b0ed1030a7fdcfedb95069060a9fa67ab20c9a/chrome/test/data/android/render_tests/ArticleSnippetsTest.short_minimal_snippet_narrow.Nexus_5.port.png
[modify] https://crrev.com/90b0ed1030a7fdcfedb95069060a9fa67ab20c9a/chrome/test/data/android/render_tests/ArticleSnippetsTest.short_snippet.Nexus_5.port.png
[modify] https://crrev.com/90b0ed1030a7fdcfedb95069060a9fa67ab20c9a/chrome/test/data/android/render_tests/ArticleSnippetsTest.short_snippet_narrow.Nexus_5.port.png
[modify] https://crrev.com/90b0ed1030a7fdcfedb95069060a9fa67ab20c9a/chrome/test/data/android/render_tests/ArticleSnippetsTest.snippets.Nexus_5.port.png
[modify] https://crrev.com/90b0ed1030a7fdcfedb95069060a9fa67ab20c9a/chrome/test/data/android/render_tests/ArticleSnippetsTest.snippets_narrow.Nexus_5.port.png
[modify] https://crrev.com/90b0ed1030a7fdcfedb95069060a9fa67ab20c9a/chrome/test/data/android/render_tests/NewTabPageTest.fakebox.Nexus_5.port.png
[modify] https://crrev.com/90b0ed1030a7fdcfedb95069060a9fa67ab20c9a/chrome/test/data/android/render_tests/NewTabPageTest.most_visited.Nexus_5.port.png
[modify] https://crrev.com/90b0ed1030a7fdcfedb95069060a9fa67ab20c9a/chrome/test/data/android/render_tests/NewTabPageTest.new_tab_page.Nexus_5.port.png
[modify] https://crrev.com/90b0ed1030a7fdcfedb95069060a9fa67ab20c9a/chrome/test/data/android/render_tests/NewTabPageTest.new_tab_page_scrolled.Nexus_5.port.png