New issue
Advanced search Search tips

Issue 666867 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Nov 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

Swarming task executes successfully locally but fails remotely.

Project Member Reported by ehmaldonado@chromium.org, Nov 18 2016

Issue description

The following task fails
https://chromium-swarm.appspot.com/task?id=3291b3ba8fa6d610&refresh=10&show_raw=1
but when I execute
python tools\swarming_client\swarming.py reproduce -S https://chromium-swarm.appspot.com 3291b3ba8fa6d610
it succeeds.

The problem seems to be that it can't find ..\..\third_party\gtest-parallel\gtest-parallel, but it appears in the isolated inputs and also in the files downloaded when executed locally.

It also fails when triggered from the buildbots.
 

Comment 1 by mar...@chromium.org, Nov 18 2016

Cc: mar...@chromium.org
Owner: ehmaldonado@chromium.org
I have a few questions first:

https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=9cfd4e362b4d32f79001168a7e406e13b86a4a52

> gtest_parallel_path = os.path.normpath(os.path.split(sys.argv[0])[0])
Instead, do the following:
gtest_parallel_path = os.path.dirname(os.path.abspath(__file__))
Don't assume much about sys.argv[0].

> subprocess.Popen(command, env=test_env, cwd=os.getcwd())
Is this just for testing or is that the normal code? Because Swarming can't know when the task completed because of this. :/
Use instead:
sys.exit(subprocess.call(command, env=test_env, cwd=os.getcwd()))


What does print "Additional test environment:" ? "CHROME_DEVEL_SANDBOX=/opt/chromium/chrome_sandbox" is not printed by your wrapper. Do you know where this comes from?
OK, I'll do that instead, thanks :). And this is just a proof of concept right now.

That comes from test_env.py I assume.
The command I'm running is

    "../../testing/test_env.py",
    "python",
    "../../third_party/gtest-parallel/gtest-parallel-wrapper",
    ".\\audio_decoder_unittests.exe",
    "--",
    "--asan=0",
    "--msan=0",
    "--tsan=0"
I'm not sure why the output looks like that. I'd think it'd be printed first and then the output from gtest-parallel-wrapper

Comment 4 by mar...@chromium.org, Nov 18 2016

Ok so at least I found one bug in the test framework. CL sent for this.

"python" is likely getting in the way, try removing it. But that would require you to rename the wrapper to end with .py and make it executable on POSIX. Do both.
How do I make it executable on POSIX?

Comment 6 by mar...@chromium.org, Nov 18 2016

git mv gtest-parallel-wrapper gtest-parallel-wrapper.py
chmod +x gtest-parallel-wrapper.py
git commit -a -m "+x"
Huh, I didn't know permissions were stored too.
Good to know :)
I'll do it in about half an hour. I don't see how it would be getting in the way, but it's worth a try.
Status: Fixed (was: Untriaged)
It fixed it, thanks.
It'd be nice to know why it didn't fail locally, tho.
Hypothesis: you may have python.exe in your PATH. The bots don't.
But you can see that the command executed by test_env.py is
e:\b\depot_tools\python276_bin\python.exe ../../third_party/gtest-parallel/gtest-parallel-wrapper .\audio_decoder_unittests.exe --
And the command executed by gtest-parallel-wrapper is
e:\b\depot_tools\python276_bin\python.exe ..\..\third_party\gtest-parallel\gtest-parallel --shard_count 1 --shard_index 0 .\audio_decoder_unittests.exe --

So having python shouldn't be necessary, since you're executing the command via the absolute path.
And it looks like (at least some of) the bots do have python in PATH. (search for 'PATH': 'e:\\b\\depot_tools\\python276_bin;)
https://chromium-swarm.appspot.com/task?id=3292555964652410&refresh=10&show_raw=1
Project Member

Comment 12 by bugdroid1@chromium.org, Nov 19 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f00125f82a0e0ce948644ff239941e6f727984bf

commit f00125f82a0e0ce948644ff239941e6f727984bf
Author: maruel <maruel@chromium.org>
Date: Sat Nov 19 00:01:14 2016

Flush stdout before shelling out.

Otherwise the output can be intermingled.

R=dpranke@chromium.org
BUG= 666867 

Review-Url: https://codereview.chromium.org/2515643003
Cr-Commit-Position: refs/heads/master@{#433340}

[modify] https://crrev.com/f00125f82a0e0ce948644ff239941e6f727984bf/testing/test_env.py

Sign in to add a comment