Project: chromium Issues People Development process History Sign in
New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 3 users
Status: Available
Owner: ----
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Feature

Blocked on:
issue angleproject:1810



Sign in to add a comment
Chromium recipe should support running gtest isolates locally
Project Member Reported by kbr@chromium.org, Jan 28 2016 Back to list
It would be ideal if the Chromium recipe supported running the isolate for a given gtest locally, in the case where the builder doesn't support swarming.

The use case is as follows: for a few one-off pieces of GPU hardware, we will not add machines into the swarming pool; instead, we'll run the tests directly on that hardware.

Currently, this uses the package_build/extract_build steps on the builder and tester, respectively, to transmit the binaries. The gtest is run out of the out/Release or out/Debug directories from the extracted build. This is a quite different mechanism than running the isolate for this gtest, since the isolate contains command line arguments and potentially other data dependencies.

It would be much better if generate_gtest (recipe_modules/chromium_tests/steps.py) supported a flag indicating to require the test to be run via an isolate, even if the tester doesn't support swarming.

If all of the gtests built by a particular builder ran in this isolated mode, and there were no script_tests (note that isolated_script_tests always run via isolate), then the package_build step could be skipped. Similarly, the extract_build step could be skipped for testers that don't support swarming, if they run only forced-isolated gtests and no script_tests. It could be trivially skipped on testers that support swarming if all tests are swarmed.

 
Comment 1 by sergiyb@google.com, Jan 28 2016
Labels: Pri-2
Status: Available
Cc: phajdan@google.com
It may already be possible with ScriptTest.

Please let me know if you'd like to get some help with implementing this.
Comment 3 by kbr@chromium.org, Jan 28 2016
Yes, thank you. I'd really appreciate any help making it possible to run gtest_tests (as specified in the src-side JSON files) via isolates locally. I won't be able to work on this in the near future. (script_tests are not relevant for our group)

Turning off the package_build/extract_build steps would be super awesome, too, even if it had to be done manually per builder and tester.

Comment 4 by kbr@chromium.org, Jan 29 2016
Note: I found while trying to use GCE slaves to trigger and collect the Swarming jobs on all platforms, that the extract_build step requires the OS on the builder and tester to match. Here's one example build:

https://build.chromium.org/p/chromium.gpu.fyi/builders/Win7%20NextGen%20Release%20%28NVIDIA%29/builds/1043

and the full stdout from the build is attached.

It seems impractical to push to support disabling the extract_build step on waterfall builders, because then the tryservers (like win_chromium_rel_ng) which are mirroring these waterfall testers would have a slightly different configuration. (The waterfall tester would disable the extract_build step, but the tryserver would presumably still run it.) For this reason I'm abandoning this approach and using VMs which match the builders' platforms.

stdout.txt
1.8 MB View Download
Comment 5 by aga...@chromium.org, Apr 26 2016
Components: Infra>CQ
Labels: -Infra-CommitQueue
Cc: -tandrii@chromium.org -phajdan@google.com
Components: -Infra>CQ Infra>Client>Chrome
Comment 7 by kbr@chromium.org, Jan 31 2017
Blockedon: angleproject:1810
Cc: dpranke@chromium.org andyb...@chromium.org zmo@chromium.org jmad...@chromium.org
Labels: -Pri-2 Pri-1
This issue has come up a few more times, and is currently a hot issue on one of our new bots: https://bugs.chromium.org/p/angleproject/issues/detail?id=1810 .

The issue is that zip_build.py doesn't know anything about the data_deps from the GN files. We'll have to hack either the script or its invocation to pick up some key files from the top-level gen/ directory.

All of our tests are already isolated. We would really like to run them via isolate, even if they aren't swarmed.

Could we please prioritize this bug? I would be happy to contribute but am going to be mostly in meetings for the next two weeks. jmadill@ is most interested in having this work.

Thanks for your help.

Comment 8 by mar...@chromium.org, Jan 31 2017
Cc: iannucci@chromium.org martiniss@chromium.org
Adding potentially relevant folks.
Cc: -andyb...@chromium.org
I don't think I'm following the logic here.

How does running everything via isolates eliminate problems w/ zip_build and package_build/extract_build ? Don't you still need to convey the files from one machine to another? How would you convey what the list of isolates to use is?

From looking at the Angle bug, it looks like the problem is that you're trying to depend on files in the gen/ directory, which we generally don't want you to do. I think if you just fix that, the immediate problem goes away, right?


Cc: -serg...@chromium.org
Comment 12 by kbr@chromium.org, Feb 8 2017
Labels: -Pri-1 Pri-2
The Chromium recipe already determines which targets a given builder needs to compile, even when the builder and tester are separate machines.

All that needs to happen is for the isolates to be uploaded by the builder, and for the tester to trigger the tests via run_isolated.py. Then the zip_build/extract_build steps can be eliminated on the builder/tester.

I agree that this wouldn't really have helped solve https://bugs.chromium.org/p/angleproject/issues/detail?id=1810 , so downgrading to P2 again.

Our group keeps running into weird differences between running tests locally vs. via Swarming. Changing the local case to run via isolate would help eliminate these differences. However it wouldn't necessarily solve the problem that e.g. SwarmingIsolatedScriptTest has a lot more intelligence than LocalIsolatedScriptTest, especially in processing the outputs of the test. That'd still be an issue.

Cc: smut@chromium.org
I think this is more-or-less what smut@ is working on in bug 685882, so I think we probably agree with you (though there's probably more work to do beyond that to make it work for non-iOS cases).
Sign in to add a comment