Issue metadata
Sign in to add a comment
|
Linux ChromiumOS Ozone Builder and trybot failing on "isolate tests" step. |
||||||||||||||||||||||
Issue descriptionThe https://build.chromium.org/p/chromium.chromiumos/builders/Linux%20ChromiumOS%20Ozone%20Builder?numbuilds=100 and the trybot https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_ozone_rel_ng?numbuilds=200 is having severe problems with "isolate tests", causing the bots to hang and timeout. This is severely affecting the CQ which was noticed in https://codereview.chromium.org/2504143002/ that we've been trying to land for many hours now. I think this is possibly P0, but I set to P1 since it seems to not affect all CQ jobs (but maybe all the green ones are cached entries or jobs that don't trigger isolate steps due to analyze). The bot's problem also doesn't show up under neither Chromium or Chrome OS sheriff-o-matic actions, which may be why it hasn't been fixed. It seems to start failing pretty clearly with https://build.chromium.org/p/chromium.chromiumos/builders/Linux%20ChromiumOS%20Ozone%20Builder/builds/41293 at least.
,
Nov 16 2016
xdisplaycheck must NOT be referenced by ozone build. ozone is X-less by definition. It's a regression in the BUILD.gn files. Andrii, any suggestion how isolate could better handle this failure mode?
,
Nov 16 2016
In alst 200 builds, I see 16 failing due to isolate tests, 4 i checked had the same error with xdisplaycheck. I wonder if this CL could be the reason https://codereview.chromium.org/2493063003 for that file being missing. Henrik confirmed that it's in the blamelist of https://build.chromium.org/p/chromium.chromiumos/builders/Linux%20ChromiumOS%20Ozone%20Builder/builds/41293 . So revert in https://codereview.chromium.org/2506013002/
,
Nov 16 2016
If that helps, we still have 2 things to worry about: * why CQ let https://codereview.chromium.org/2493063003 in -> perhaps analyze decided not to build the target that needs xdisplaycheck * a bug in isolate that makes it hang instead of terminating with an error.
,
Nov 16 2016
> * why CQ let https://codereview.chromium.org/2493063003 in > -> perhaps analyze decided not to build the target that needs xdisplaycheck No, not because of this. I see passing builds with isolate tests working on unit_tests target, which is what's failing in others because of missing xdisplaycheck. Can someone with better GN/mb knowledge look in this?
,
Nov 16 2016
,
Nov 16 2016
Sorry about the failures! Actually, with regards to #2 we want to test Ozone X11 on the Chrome CQ trybots and need xvfb and xdisplaycheck for that. We have been building Ozone X11 on linux_chromium_chromeos_ozone_rel_ng for the past 6 months or so. I just flipped on running the first test with Ozone X11 this week which I think worked fine for the Chrome CQ trybots. If I understand correctly the "Linux ChromiumOS Tests (1)" is a Chrome OS trybot? Or at least, it's not part of the Chrome CQ. It looks like I need to make sure that linux_chromium_chromeos_ozone_rel_ng gets xdisplaycheck but that doesn't happen automatically for all trybots where use_ozone=true.
,
Nov 16 2016
First, it's not yet clear that your CL has caused this, as revert is still in progress. You are correct that linux_chromium_chromeos_ozone_rel_ng isn't in default chromium CQ bots. I don't understand GN/build configs, so can't confirm/deny/comment on the rest of your comment.
,
Nov 16 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/2939f78b519bc6cd0909a9615a666205ad6aecc2 commit 2939f78b519bc6cd0909a9615a666205ad6aecc2 Author: tandrii <tandrii@chromium.org> Date: Wed Nov 16 16:33:31 2016 Revert of Run gl_unittests with Ozone X11. (patchset #3 id:40001 of https://codereview.chromium.org/2493063003/ ) Reason for revert: Likely caused error in isolate steps in http://crbug.com/665895 : /b/c/b/linux_chromeos/src/out/Release/xdisplaycheck: no such file or directory Original issue's description: > Run gl_unittests with Ozone X11. > > Add a new test target gl_unittests_ozonex that runs with > --ozone-platform=x11. The Ozone CrOS bot will now run gl_unittests with > both Ozone headless and Ozone X11. > > The test requires an X server to be running on the bot. The mb.py script > is modified to look for GN arg use_ozone=true and runtime flag > --ozone-platform=x11. In this case, it runs the test using xvfb. > > BUG= 664570 > > Committed: https://crrev.com/7895a178293de777611dd91255b9b90934f1a5df > Cr-Commit-Position: refs/heads/master@{#432038} TBR=dpranke@chromium.org,sadrul@chromium.org,kylechar@chromium.org # tandrii@ decided it's enough waiting for tests. # so not (Not skipping CQ checks because original CL landed more than 1 days ago.) NOTRY=True BUG= 664570 , 665895 Review-Url: https://codereview.chromium.org/2506013002 Cr-Commit-Position: refs/heads/master@{#432513} [modify] https://crrev.com/2939f78b519bc6cd0909a9615a666205ad6aecc2/BUILD.gn [modify] https://crrev.com/2939f78b519bc6cd0909a9615a666205ad6aecc2/testing/buildbot/chromium.chromiumos.json [modify] https://crrev.com/2939f78b519bc6cd0909a9615a666205ad6aecc2/testing/buildbot/gn_isolate_map.pyl [modify] https://crrev.com/2939f78b519bc6cd0909a9615a666205ad6aecc2/tools/mb/mb.py [modify] https://crrev.com/2939f78b519bc6cd0909a9615a666205ad6aecc2/ui/gl/BUILD.gn
,
Nov 16 2016
OK, let's see if this waterfall build succeeds to through isolate tests: https://build.chromium.org/p/chromium.chromiumos/builders/Linux%20ChromiumOS%20Ozone%20Builder/builds/41324
,
Nov 16 2016
And success. So revert most likely have helped.
,
Nov 16 2016
The bot looks reliably green: https://build.chromium.org/p/chromium.chromiumos/builders/Linux%20ChromiumOS%20Ozone%20Builder Anything else for a trooper to do here?
,
Nov 16 2016
A trooper, no. But there is still a bug in isolate.
,
Nov 16 2016
Andrii, can you file a new bug for isolate specifically so that tansell@ or djd@ can pick it up? This issue should then be closed.
,
Nov 16 2016
Re #14: By coincidence, I think Edward just filed such a bug after noticing this in WebRTC as well: bug 666047 .
,
Nov 16 2016
Thanks, Henrik! I'm closing this one.
,
Mar 4 2017
,
Apr 17 2017
,
May 30 2017
,
Aug 1 2017
,
Oct 14 2017
|
|||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||
Comment 1 by tandrii@chromium.org
, Nov 16 2016