Now that goldens for Nexus 5 have been committed, see how flaky the Render Tests are.
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/9608749b27c3ac2a1e74afac53db1503f7737bea commit 9608749b27c3ac2a1e74afac53db1503f7737bea Author: Peter E Conn <peconn@chromium.org> Date: Fri Jul 21 10:52:20 2017 📷 Enable Render Tests. Make the render tests fail if the rendered bitmap does not match the golden. Bug: 641973 Change-Id: I432111296893bafab18dc65bfb62b12a8212d68e Reviewed-on: https://chromium-review.googlesource.com/565399 Commit-Queue: Peter Conn <peconn@chromium.org> Reviewed-by: John Budorick <jbudorick@chromium.org> Reviewed-by: Bernhard Bauer <bauerb@chromium.org> Cr-Commit-Position: refs/heads/master@{#488631} [modify] https://crrev.com/9608749b27c3ac2a1e74afac53db1503f7737bea/chrome/test/android/javatests/src/org/chromium/chrome/test/util/RenderTestRule.java
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/b91fa1fd64c37d20f1fe21a751f2ebf13f5fab0b commit b91fa1fd64c37d20f1fe21a751f2ebf13f5fab0b Author: Peter Conn <peconn@chromium.org> Date: Fri Jul 21 15:28:37 2017 Revert "📷 Enable Render Tests." This reverts commit 9608749b27c3ac2a1e74afac53db1503f7737bea. Reason for revert: Failures on Lollipop Phone Tester, eg: https://uberchromegw.corp.google.com/i/chromium.android/builders/Lollipop%20Phone%20Tester/builds/13630 https://uberchromegw.corp.google.com/i/chromium.android/builders/Lollipop%20Phone%20Tester/builds/13631 https://uberchromegw.corp.google.com/i/chromium.android/builders/Lollipop%20Phone%20Tester/builds/13632 https://uberchromegw.corp.google.com/i/chromium.android/builders/Lollipop%20Phone%20Tester/builds/13633 Original change's description: > 📷 Enable Render Tests. > > Make the render tests fail if the rendered bitmap does not match the > golden. > > Bug: 641973 > Change-Id: I432111296893bafab18dc65bfb62b12a8212d68e > Reviewed-on: https://chromium-review.googlesource.com/565399 > Commit-Queue: Peter Conn <peconn@chromium.org> > Reviewed-by: John Budorick <jbudorick@chromium.org> > Reviewed-by: Bernhard Bauer <bauerb@chromium.org> > Cr-Commit-Position: refs/heads/master@{#488631} TBR=bauerb@chromium.org,mikecase@chromium.org,peconn@chromium.org,jbudorick@chromium.org Change-Id: I17ba42053f6679098f176c6ec7fdfb5e14984a39 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: 641973 Reviewed-on: https://chromium-review.googlesource.com/581527 Reviewed-by: Peter Conn <peconn@chromium.org> Commit-Queue: Peter Conn <peconn@chromium.org> Cr-Commit-Position: refs/heads/master@{#488661} [modify] https://crrev.com/b91fa1fd64c37d20f1fe21a751f2ebf13f5fab0b/chrome/test/android/javatests/src/org/chromium/chrome/test/util/RenderTestRule.java
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/75d5cc55de99398763e312c2da1ac669a717606b commit 75d5cc55de99398763e312c2da1ac669a717606b Author: Peter E Conn <peconn@chromium.org> Date: Wed Jul 26 10:15:30 2017 📷 Enable Render Tests and encode Android API level in goldens. This makes goldens specific to both the device model and Android SDK version and enables render tests on the Nexus 5 Kitkat. It also removes the golden images for Nexus 5Xs since chrome_public_test_apk is not run on Nexus 5Xs on the CQ. Bug: 641973 Change-Id: I46dd428489820f814355d51d266d256fcf189c4c Reviewed-on: https://chromium-review.googlesource.com/583067 Commit-Queue: Peter Conn <peconn@chromium.org> Reviewed-by: Bernhard Bauer <bauerb@chromium.org> Reviewed-by: John Budorick <jbudorick@chromium.org> Cr-Commit-Position: refs/heads/master@{#489597} [modify] https://crrev.com/75d5cc55de99398763e312c2da1ac669a717606b/build/android/pylib/local/device/local_device_instrumentation_test_run.py [modify] https://crrev.com/75d5cc55de99398763e312c2da1ac669a717606b/chrome/android/javatests/src/org/chromium/chrome/browser/ntp/NewTabPageTest.java [modify] https://crrev.com/75d5cc55de99398763e312c2da1ac669a717606b/chrome/test/android/javatests/src/org/chromium/chrome/test/util/RenderTestRule.java [rename] https://crrev.com/75d5cc55de99398763e312c2da1ac669a717606b/chrome/test/data/android/render_tests/ArticleSnippetsTest.download_snippet_placeholder.Nexus_5-19.png [delete] https://crrev.com/1236993289b3a34ac29c6d18fbd31273eee86c73/chrome/test/data/android/render_tests/ArticleSnippetsTest.download_snippet_placeholder.Nexus_5X.port.png [rename] https://crrev.com/75d5cc55de99398763e312c2da1ac669a717606b/chrome/test/data/android/render_tests/ArticleSnippetsTest.download_snippet_thumbnail.Nexus_5-19.png [delete] https://crrev.com/1236993289b3a34ac29c6d18fbd31273eee86c73/chrome/test/data/android/render_tests/ArticleSnippetsTest.download_snippet_thumbnail.Nexus_5X.port.png [rename] https://crrev.com/75d5cc55de99398763e312c2da1ac669a717606b/chrome/test/data/android/render_tests/ArticleSnippetsTest.long_minimal_snippet_narrow.Nexus_5-19.png [delete] https://crrev.com/1236993289b3a34ac29c6d18fbd31273eee86c73/chrome/test/data/android/render_tests/ArticleSnippetsTest.long_minimal_snippet_narrow.Nexus_5X.port.png [rename] https://crrev.com/75d5cc55de99398763e312c2da1ac669a717606b/chrome/test/data/android/render_tests/ArticleSnippetsTest.long_snippet.Nexus_5-19.png [delete] https://crrev.com/1236993289b3a34ac29c6d18fbd31273eee86c73/chrome/test/data/android/render_tests/ArticleSnippetsTest.long_snippet.Nexus_5X.port.png [rename] https://crrev.com/75d5cc55de99398763e312c2da1ac669a717606b/chrome/test/data/android/render_tests/ArticleSnippetsTest.long_snippet_narrow.Nexus_5-19.png [delete] https://crrev.com/1236993289b3a34ac29c6d18fbd31273eee86c73/chrome/test/data/android/render_tests/ArticleSnippetsTest.long_snippet_narrow.Nexus_5X.port.png [rename] https://crrev.com/75d5cc55de99398763e312c2da1ac669a717606b/chrome/test/data/android/render_tests/ArticleSnippetsTest.minimal_snippet.Nexus_5-19.png [delete] https://crrev.com/1236993289b3a34ac29c6d18fbd31273eee86c73/chrome/test/data/android/render_tests/ArticleSnippetsTest.minimal_snippet.Nexus_5X.port.png [rename] https://crrev.com/75d5cc55de99398763e312c2da1ac669a717606b/chrome/test/data/android/render_tests/ArticleSnippetsTest.modern-long_minimal_snippet_narrow.Nexus_5-19.png [rename] https://crrev.com/75d5cc55de99398763e312c2da1ac669a717606b/chrome/test/data/android/render_tests/ArticleSnippetsTest.modern-long_snippet.Nexus_5-19.png [rename] https://crrev.com/75d5cc55de99398763e312c2da1ac669a717606b/chrome/test/data/android/render_tests/ArticleSnippetsTest.modern-long_snippet_narrow.Nexus_5-19.png [rename] https://crrev.com/75d5cc55de99398763e312c2da1ac669a717606b/chrome/test/data/android/render_tests/ArticleSnippetsTest.modern-minimal_snippet.Nexus_5-19.png [rename] https://crrev.com/75d5cc55de99398763e312c2da1ac669a717606b/chrome/test/data/android/render_tests/ArticleSnippetsTest.modern-short_minimal_snippet_narrow.Nexus_5-19.png [rename] https://crrev.com/75d5cc55de99398763e312c2da1ac669a717606b/chrome/test/data/android/render_tests/ArticleSnippetsTest.modern-short_snippet.Nexus_5-19.png [rename] https://crrev.com/75d5cc55de99398763e312c2da1ac669a717606b/chrome/test/data/android/render_tests/ArticleSnippetsTest.modern-short_snippet_narrow.Nexus_5-19.png [rename] https://crrev.com/75d5cc55de99398763e312c2da1ac669a717606b/chrome/test/data/android/render_tests/ArticleSnippetsTest.modern-snippets.Nexus_5-19.png [rename] https://crrev.com/75d5cc55de99398763e312c2da1ac669a717606b/chrome/test/data/android/render_tests/ArticleSnippetsTest.modern-snippets_narrow.Nexus_5-19.png [rename] https://crrev.com/75d5cc55de99398763e312c2da1ac669a717606b/chrome/test/data/android/render_tests/ArticleSnippetsTest.short_minimal_snippet_narrow.Nexus_5-19.png [delete] https://crrev.com/1236993289b3a34ac29c6d18fbd31273eee86c73/chrome/test/data/android/render_tests/ArticleSnippetsTest.short_minimal_snippet_narrow.Nexus_5X.port.png [rename] https://crrev.com/75d5cc55de99398763e312c2da1ac669a717606b/chrome/test/data/android/render_tests/ArticleSnippetsTest.short_snippet.Nexus_5-19.png [delete] https://crrev.com/1236993289b3a34ac29c6d18fbd31273eee86c73/chrome/test/data/android/render_tests/ArticleSnippetsTest.short_snippet.Nexus_5X.port.png [rename] https://crrev.com/75d5cc55de99398763e312c2da1ac669a717606b/chrome/test/data/android/render_tests/ArticleSnippetsTest.short_snippet_narrow.Nexus_5-19.png [delete] https://crrev.com/1236993289b3a34ac29c6d18fbd31273eee86c73/chrome/test/data/android/render_tests/ArticleSnippetsTest.short_snippet_narrow.Nexus_5X.port.png [rename] https://crrev.com/75d5cc55de99398763e312c2da1ac669a717606b/chrome/test/data/android/render_tests/ArticleSnippetsTest.snippets.Nexus_5-19.png [delete] https://crrev.com/1236993289b3a34ac29c6d18fbd31273eee86c73/chrome/test/data/android/render_tests/ArticleSnippetsTest.snippets.Nexus_5X.port.png [rename] https://crrev.com/75d5cc55de99398763e312c2da1ac669a717606b/chrome/test/data/android/render_tests/ArticleSnippetsTest.snippets_narrow.Nexus_5-19.png [delete] https://crrev.com/1236993289b3a34ac29c6d18fbd31273eee86c73/chrome/test/data/android/render_tests/ArticleSnippetsTest.snippets_narrow.Nexus_5X.port.png [rename] https://crrev.com/75d5cc55de99398763e312c2da1ac669a717606b/chrome/test/data/android/render_tests/NewTabPageTest.fakebox.Nexus_5-19.png [delete] https://crrev.com/1236993289b3a34ac29c6d18fbd31273eee86c73/chrome/test/data/android/render_tests/NewTabPageTest.fakebox.Nexus_5X.port.png [rename] https://crrev.com/75d5cc55de99398763e312c2da1ac669a717606b/chrome/test/data/android/render_tests/NewTabPageTest.most_visited.Nexus_5-19.png [delete] https://crrev.com/1236993289b3a34ac29c6d18fbd31273eee86c73/chrome/test/data/android/render_tests/NewTabPageTest.most_visited.Nexus_5X.port.png [rename] https://crrev.com/75d5cc55de99398763e312c2da1ac669a717606b/chrome/test/data/android/render_tests/NewTabPageTest.new_tab_page.Nexus_5-19.png [delete] https://crrev.com/1236993289b3a34ac29c6d18fbd31273eee86c73/chrome/test/data/android/render_tests/NewTabPageTest.new_tab_page.Nexus_5X.port.png [rename] https://crrev.com/75d5cc55de99398763e312c2da1ac669a717606b/chrome/test/data/android/render_tests/NewTabPageTest.new_tab_page_scrolled.Nexus_5-19.png [delete] https://crrev.com/1236993289b3a34ac29c6d18fbd31273eee86c73/chrome/test/data/android/render_tests/NewTabPageTest.new_tab_page_scrolled.Nexus_5X.port.png [delete] https://crrev.com/1236993289b3a34ac29c6d18fbd31273eee86c73/chrome/test/data/android/render_tests/VrShellTest.blank_page.Nexus_5.land.png [delete] https://crrev.com/1236993289b3a34ac29c6d18fbd31273eee86c73/chrome/test/data/android/render_tests/VrShellTest.blank_page.Nexus_6P.land.png [delete] https://crrev.com/1236993289b3a34ac29c6d18fbd31273eee86c73/chrome/test/data/android/render_tests/VrShellTest.vr_entered.Nexus_6P.land.png
Closing this since Render Tests are now in use :-)
Comment 1 by peconn@chromium.org
, Aug 29 2016