New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 665895 link

Starred by 1 user

Issue metadata

Status: Archived
Owner:
Closed: Nov 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug-Regression



Sign in to add a comment

Linux ChromiumOS Ozone Builder and trybot failing on "isolate tests" step.

Project Member Reported by kjellander@chromium.org, Nov 16 2016

Issue description

The https://build.chromium.org/p/chromium.chromiumos/builders/Linux%20ChromiumOS%20Ozone%20Builder?numbuilds=100 and the trybot https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_ozone_rel_ng?numbuilds=200 is having severe problems with "isolate tests", causing the bots to hang and timeout.

This is severely affecting the CQ which was noticed in https://codereview.chromium.org/2504143002/ that we've been trying to land for many hours now.

I think this is possibly P0, but I set to P1 since it seems to not affect all CQ jobs (but maybe all the green ones are cached entries or jobs that don't trigger isolate steps due to analyze).

The bot's problem also doesn't show up under neither Chromium or Chrome OS sheriff-o-matic actions, which may be why it hasn't been fixed. It seems to start failing pretty clearly with https://build.chromium.org/p/chromium.chromiumos/builders/Linux%20ChromiumOS%20Ozone%20Builder/builds/41293 at least. 
 
Cc: djd@chromium.org mar...@chromium.org tansell@chromium.org
So, isolate test hits this error:


unit_tests  lstat /b/c/b/linux_chromeos/src/out/Release/xdisplaycheck: no such file or directory


So, while something likely got broken on in the GN config for this builder, isolate isn't gracefully handling the failure.

Comment 2 by mar...@chromium.org, Nov 16 2016

xdisplaycheck must NOT be referenced by ozone build. ozone is X-less by definition. It's a regression in the BUILD.gn files.

Andrii, any suggestion how isolate could better handle this failure mode?
In alst 200 builds, I see 16 failing due to isolate tests, 4 i checked had the same error with xdisplaycheck.

I wonder if this CL could be the reason https://codereview.chromium.org/2493063003 for that file being missing. Henrik confirmed that it's in the blamelist of https://build.chromium.org/p/chromium.chromiumos/builders/Linux%20ChromiumOS%20Ozone%20Builder/builds/41293 . So revert in https://codereview.chromium.org/2506013002/
If that helps, we still have 2 things to worry about:
 * why CQ let https://codereview.chromium.org/2493063003 in -> perhaps analyze decided not to build the target that needs xdisplaycheck
 * a bug in isolate that makes it hang instead of terminating with an error.
> * why CQ let https://codereview.chromium.org/2493063003 in
>  -> perhaps analyze decided not to build the target that needs xdisplaycheck
No, not because of this. I see passing builds with isolate tests working on unit_tests target, which is what's failing in others because of missing xdisplaycheck. Can someone with better GN/mb knowledge look in this?
Cc: kylec...@chromium.org
+kylechar@chromium.org as owner of reverted CL
Sorry about the failures! Actually, with regards to #2 we want to test Ozone X11 on the Chrome CQ trybots and need xvfb and xdisplaycheck for that.

We have been building Ozone X11 on linux_chromium_chromeos_ozone_rel_ng for the past 6 months or so. I just flipped on running the first test with Ozone X11 this week which I think worked fine for the Chrome CQ trybots.

If I understand correctly the "Linux ChromiumOS Tests (1)" is a Chrome OS trybot? Or at least, it's not part of the Chrome CQ. It looks like I need to make sure that linux_chromium_chromeos_ozone_rel_ng gets xdisplaycheck but that doesn't happen automatically for all trybots where use_ozone=true.
First, it's not yet clear that your CL has caused this, as revert is still in progress.

You are correct that linux_chromium_chromeos_ozone_rel_ng isn't in default chromium CQ bots. I don't understand GN/build configs, so can't confirm/deny/comment on the rest of your comment.
Project Member

Comment 9 by bugdroid1@chromium.org, Nov 16 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2939f78b519bc6cd0909a9615a666205ad6aecc2

commit 2939f78b519bc6cd0909a9615a666205ad6aecc2
Author: tandrii <tandrii@chromium.org>
Date: Wed Nov 16 16:33:31 2016

Revert of Run gl_unittests with Ozone X11. (patchset #3 id:40001 of https://codereview.chromium.org/2493063003/ )

Reason for revert:
Likely caused error in isolate steps in  http://crbug.com/665895 :

/b/c/b/linux_chromeos/src/out/Release/xdisplaycheck: no such file or directory

Original issue's description:
> Run gl_unittests with Ozone X11.
>
> Add a new test target gl_unittests_ozonex that runs with
> --ozone-platform=x11. The Ozone CrOS bot will now run gl_unittests with
> both Ozone headless and Ozone X11.
>
> The test requires an X server to be running on the bot. The mb.py script
> is modified to look for GN arg use_ozone=true and runtime flag
> --ozone-platform=x11. In this case, it runs the test using xvfb.
>
> BUG= 664570 
>
> Committed: https://crrev.com/7895a178293de777611dd91255b9b90934f1a5df
> Cr-Commit-Position: refs/heads/master@{#432038}

TBR=dpranke@chromium.org,sadrul@chromium.org,kylechar@chromium.org
# tandrii@ decided it's enough waiting for tests.
# so not (Not skipping CQ checks because original CL landed more than 1 days ago.)
NOTRY=True
BUG= 664570 , 665895 

Review-Url: https://codereview.chromium.org/2506013002
Cr-Commit-Position: refs/heads/master@{#432513}

[modify] https://crrev.com/2939f78b519bc6cd0909a9615a666205ad6aecc2/BUILD.gn
[modify] https://crrev.com/2939f78b519bc6cd0909a9615a666205ad6aecc2/testing/buildbot/chromium.chromiumos.json
[modify] https://crrev.com/2939f78b519bc6cd0909a9615a666205ad6aecc2/testing/buildbot/gn_isolate_map.pyl
[modify] https://crrev.com/2939f78b519bc6cd0909a9615a666205ad6aecc2/tools/mb/mb.py
[modify] https://crrev.com/2939f78b519bc6cd0909a9615a666205ad6aecc2/ui/gl/BUILD.gn

OK, let's see if this waterfall build succeeds to through isolate tests:

https://build.chromium.org/p/chromium.chromiumos/builders/Linux%20ChromiumOS%20Ozone%20Builder/builds/41324
And success. So revert most likely have helped.
The bot looks reliably green: https://build.chromium.org/p/chromium.chromiumos/builders/Linux%20ChromiumOS%20Ozone%20Builder

Anything else for a trooper to do here?
A trooper, no. But there is still a bug in isolate.
Andrii, can you file a new bug for isolate specifically so that tansell@ or djd@ can pick it up?

This issue should then be closed.
Cc: ehmaldonado@chromium.org
Re #14: By coincidence, I think Edward just filed such a bug after noticing this in WebRTC as well:  bug 666047 .
Owner: tandrii@chromium.org
Status: Fixed (was: Untriaged)
Thanks, Henrik! I'm closing this one.

Comment 17 by dchan@google.com, Mar 4 2017

Labels: VerifyIn-58

Comment 18 by dchan@google.com, Apr 17 2017

Labels: VerifyIn-59

Comment 19 by dchan@google.com, May 30 2017

Labels: VerifyIn-60
Labels: VerifyIn-61

Comment 21 by dchan@chromium.org, Oct 14 2017

Status: Archived (was: Fixed)

Sign in to add a comment