Recent regression https://stainless.corp.google.com/search?view=matrix&row=model&col=build&first_date=2018-10-17&last_date=2018-10-30&test=graphics_Idle%24&status=GOOD&status=WARN&status=FAIL&exclude_cts=false&exclude_not_run=false&exclude_non_release=true&exclude_au=true&exclude_acts=true&exclude_retried=true&exclude_non_production=false No Chrome uprev https://crosland.corp.google.com/log/11205.0.0..11209.0.0 All kind of failures https://stainless.corp.google.com/search?view=list&first_date=2018-10-17&last_date=2018-10-30&test=graphics_Idle%24&build=11210&status=FAIL&exclude_cts=false&exclude_not_run=false&exclude_non_release=true&exclude_au=true&exclude_acts=true&exclude_retried=true&exclude_non_production=false To graphics guard for investigation.
Notice that R72-11205.0.0 was the last fully clean release build.
Looks like Chrome and ARC++ are not idle, even though they did not change in this range. That said there was just before another set of failures which points at https://crosland.corp.google.com/log/11201.0.0..11202.0.0 which is pretty empty and the Chrome uprev https://chromium.googlesource.com/chromium/src/+log/72.0.3592.0..72.0.3593.0?n=10000 Details https://stainless.corp.google.com/search?view=matrix&row=builder_name_number&col=build&first_date=2018-10-17&last_date=2018-10-30&test=graphics_Idle&build=R72&status=GOOD&status=WARN&status=FAIL&exclude_cts=false&exclude_not_run=false&exclude_non_release=true&exclude_au=true&exclude_acts=true&exclude_retried=true&exclude_non_production=false
Robert, time to bisect Chrome?
'graphics_idle' test is failing on M71 beta RC (11151.17.0, 71.0.3578.27) for BVT-ARC and BVT-Inline test suite Stainless results: https://stainless.corp.google.com/search?view=list&first_date=2018-10-28&last_date=2018-10-30&test=graphics_Idle%24&build=11151.17.0&status=FAIL&exclude_cts=false&exclude_not_run=false&exclude_non_release=true&exclude_au=true&exclude_acts=true&exclude_retried=true&exclude_non_production=false
R70 also started failing with build R70-11021.63.0. https://stainless.corp.google.com/search?view=matrix&row=builder_name_number&col=build&first_date=2018-10-17&last_date=2018-10-30&test=graphics_Idle&build=R70&status=GOOD&status=WARN&status=FAIL&exclude_cts=false&exclude_not_run=false&exclude_non_release=true&exclude_au=true&exclude_acts=true&exclude_retried=true&exclude_non_production=false The plot thickens. It looks either like a timebomb or the lab is doing bad stuff to the DUTs. Unlikely a change in the DUT images.
I think we are seeing this in CQ as well: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8931213564783166960 https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8931213442121149120 https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8931213394697130592
This (or something similar) is causing PFQ failures: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8931206114855718544
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/a200f184bc483db7f494f25accccb74cad489129 commit a200f184bc483db7f494f25accccb74cad489129 Author: Ilja H. Friedel <ihf@chromium.org> Date: Wed Oct 31 02:34:59 2018 graphics_Idle: move to bvt-perbuild. It looks like the NTP page shows a very GPU intense halloween logo. This causes failures on all branches. BUG= chromium:900299 TEST=None. Change-Id: I1e96fc5892d18512c73fce080471b64aee2f5b06 Reviewed-on: https://chromium-review.googlesource.com/c/1308743 Tested-by: Ilja H. Friedel <ihf@chromium.org> Reviewed-by: Stéphane Marchesin <marcheu@chromium.org> [modify] https://crrev.com/a200f184bc483db7f494f25accccb74cad489129/client/site_tests/graphics_Idle/control.arc [modify] https://crrev.com/a200f184bc483db7f494f25accccb74cad489129/client/site_tests/graphics_Idle/control
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/8e30da4cc8a23ae708884396a226bf01fc081113 commit 8e30da4cc8a23ae708884396a226bf01fc081113 Author: Ilja H. Friedel <ihf@chromium.org> Date: Wed Oct 31 03:05:02 2018 graphics_Idle: hide NTP. The NTP contains the Google Doodle which can cause false positives in this test. BUG= chromium:900299 TEST=Ran on peppy. Change-Id: Id2141515f36f56d982db2f478ea1a69449cb2e8c Reviewed-on: https://chromium-review.googlesource.com/c/1308744 Tested-by: Ilja H. Friedel <ihf@chromium.org> Reviewed-by: Stéphane Marchesin <marcheu@chromium.org> [modify] https://crrev.com/8e30da4cc8a23ae708884396a226bf01fc081113/client/site_tests/graphics_Idle/graphics_Idle.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/0257fd827a861151f33c46d136a1e7c072641590 commit 0257fd827a861151f33c46d136a1e7c072641590 Author: Ilja H. Friedel <ihf@chromium.org> Date: Wed Oct 31 03:05:29 2018 graphics_Idle: hide NTP. The NTP contains the Google Doodle which can cause false positives in this test. BUG= chromium:900299 TEST=Ran on peppy. Change-Id: Id2141515f36f56d982db2f478ea1a69449cb2e8c Reviewed-on: https://chromium-review.googlesource.com/c/1308744 Tested-by: Ilja H. Friedel <ihf@chromium.org> Reviewed-by: Stéphane Marchesin <marcheu@chromium.org> (cherry picked from commit 8e30da4cc8a23ae708884396a226bf01fc081113) Reviewed-on: https://chromium-review.googlesource.com/c/1308745 Reviewed-by: Ilja H. Friedel <ihf@chromium.org> [modify] https://crrev.com/0257fd827a861151f33c46d136a1e7c072641590/client/site_tests/graphics_Idle/graphics_Idle.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/143417effabee56ae5573b5c87bf084d4ab4c932 commit 143417effabee56ae5573b5c87bf084d4ab4c932 Author: Ilja H. Friedel <ihf@chromium.org> Date: Wed Oct 31 03:06:10 2018 graphics_Idle: hide NTP. The NTP contains the Google Doodle which can cause false positives in this test. BUG= chromium:900299 TEST=Ran on peppy. Change-Id: Id2141515f36f56d982db2f478ea1a69449cb2e8c Reviewed-on: https://chromium-review.googlesource.com/c/1308744 Tested-by: Ilja H. Friedel <ihf@chromium.org> Reviewed-by: Stéphane Marchesin <marcheu@chromium.org> (cherry picked from commit 8e30da4cc8a23ae708884396a226bf01fc081113) Reviewed-on: https://chromium-review.googlesource.com/c/1308745 Reviewed-by: Ilja H. Friedel <ihf@chromium.org> (cherry picked from commit 0257fd827a861151f33c46d136a1e7c072641590) Reviewed-on: https://chromium-review.googlesource.com/c/1308746 [modify] https://crrev.com/143417effabee56ae5573b5c87bf084d4ab4c932/client/site_tests/graphics_Idle/graphics_Idle.py
There's still failure on graphics_Idle_P on nocturne-paladin: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8931180022493789808 Should the patches for graphics_Idle be applied to graphics_Idle_P too?
You are right, thank you for posting the patch!
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/cbd3f2023b0076a2fdecc752798d1557c0e2f259 commit cbd3f2023b0076a2fdecc752798d1557c0e2f259 Author: Peter Shih <pihsun@chromium.org> Date: Wed Oct 31 20:16:03 2018 graphics_Idle_P: hide NTP. The NTP contains the Google Doodle which can cause false positives in this test. BUG= chromium:900299 TEST=Ran on soraka. Change-Id: Ia61dbbdf481e8f4fcb2859bcde4032ef477f8b52 Reviewed-on: https://chromium-review.googlesource.com/c/1309277 Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Ilja H. Friedel <ihf@chromium.org> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> [modify] https://crrev.com/cbd3f2023b0076a2fdecc752798d1557c0e2f259/client/site_tests/graphics_Idle_P/graphics_Idle_P.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/3ac1a1ae8b652d6503852a66a43f23eb40399f8e commit 3ac1a1ae8b652d6503852a66a43f23eb40399f8e Author: Peter Shih <pihsun@chromium.org> Date: Wed Oct 31 20:17:03 2018 graphics_Idle_P: hide NTP. The NTP contains the Google Doodle which can cause false positives in this test. BUG= chromium:900299 TEST=Ran on soraka. Change-Id: Ia61dbbdf481e8f4fcb2859bcde4032ef477f8b52 Reviewed-on: https://chromium-review.googlesource.com/c/1309277 Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Ilja H. Friedel <ihf@chromium.org> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> (cherry picked from commit cbd3f2023b0076a2fdecc752798d1557c0e2f259) Reviewed-on: https://chromium-review.googlesource.com/c/1311215 [modify] https://crrev.com/3ac1a1ae8b652d6503852a66a43f23eb40399f8e/client/site_tests/graphics_Idle_P/graphics_Idle_P.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/80f0559dd933b51ddb46583fe51e0a37c0f77126 commit 80f0559dd933b51ddb46583fe51e0a37c0f77126 Author: Peter Shih <pihsun@chromium.org> Date: Wed Oct 31 20:18:27 2018 graphics_Idle_P: hide NTP. The NTP contains the Google Doodle which can cause false positives in this test. BUG= chromium:900299 TEST=Ran on soraka. Change-Id: Ia61dbbdf481e8f4fcb2859bcde4032ef477f8b52 Reviewed-on: https://chromium-review.googlesource.com/c/1309277 Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Ilja H. Friedel <ihf@chromium.org> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> (cherry picked from commit cbd3f2023b0076a2fdecc752798d1557c0e2f259) Reviewed-on: https://chromium-review.googlesource.com/c/1311215 (cherry picked from commit 3ac1a1ae8b652d6503852a66a43f23eb40399f8e) Reviewed-on: https://chromium-review.googlesource.com/c/1311216 [modify] https://crrev.com/80f0559dd933b51ddb46583fe51e0a37c0f77126/client/site_tests/graphics_Idle_P/graphics_Idle_P.py
There is another new failure: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8931120309572236592 on nocturne-paladin Test-Logs]: graphics_Idle_P: retry_count: 2, FAIL: Failed: Did not reach 0 gem actives.
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/164ad3d325df7b29942648529b6ee0d73915c958 commit 164ad3d325df7b29942648529b6ee0d73915c958 Author: Ilja H. Friedel <ihf@chromium.org> Date: Wed Nov 07 19:40:57 2018 Revert "graphics_Idle: move to bvt-perbuild." This reverts commit a200f184bc483db7f494f25accccb74cad489129. The test got fixed, so can go back to blocking state. BUG= chromium:900299 TEST=check stainless https://stainless.corp.google.com/search?view=list&row=model&col=build&first_date=2018-11-04&last_date=2018-11-06&test=graphics_Idle&status=GOOD&status=WARN&status=FAIL&status=ERROR&exclude_cts=false&exclude_not_run=false&exclude_non_release=true&exclude_au=true&exclude_acts=true&exclude_retried=true&exclude_non_production=false Original change's description: > graphics_Idle: move to bvt-perbuild. > > It looks like the NTP page shows a very GPU intense halloween logo. > This causes failures on all branches. > > BUG= chromium:900299 > TEST=None. > > Change-Id: I1e96fc5892d18512c73fce080471b64aee2f5b06 > Reviewed-on: https://chromium-review.googlesource.com/c/1308743 > Tested-by: Ilja H. Friedel <ihf@chromium.org> > Reviewed-by: Stphane Marchesin <marcheu@chromium.org> Bug: chromium:900299 Change-Id: Ie517833fd2bc87d51405ddf78ac840790b0d3711 Reviewed-on: https://chromium-review.googlesource.com/1321290 Commit-Ready: Ilja H. Friedel <ihf@chromium.org> Tested-by: Ilja H. Friedel <ihf@chromium.org> Reviewed-by: Stéphane Marchesin <marcheu@chromium.org> [modify] https://crrev.com/164ad3d325df7b29942648529b6ee0d73915c958/client/site_tests/graphics_Idle/control.arc [modify] https://crrev.com/164ad3d325df7b29942648529b6ee0d73915c958/client/site_tests/graphics_Idle/control
Comment 1 by ihf@chromium.org
, Oct 30