chanli@

Where did you get these three tests from? They don't seem to exist...

Just to make sure --gtest_filter work correctly with .any.js tests, I tried another example:

run_web_tests --gtest_filter=external/wpt/fetch/api/idl.any.html:external/wpt/fetch/api/idl.any.worker.html:external/wpt/fetch/api/idl.any.sharedworker.html

All 3 tests ran as expected.

This is the link to the Findit analysis: https://findit-for-me.appspot.com/waterfall/build-failure?url=https://luci-milo.appspot.com/buildbot/chromium.webkit/WebKit%20Linux%20Trusty%20(dbg)/13365

the failed build is: https://luci-milo.appspot.com/buildbot/chromium.webkit/WebKit%20Linux%20Trusty%20(dbg)/13365

The test was introduced at 1ff37758f07612906228581dbc808df4b282e250 (and later reverted).

The FindIt job page says "Got Revision	f58a0adad674d61c38e66932273cfa03043b3525", which was before the test was introduced. The test didn't exist at that point.

Oh, that makes sense. Thank you for your help. I'll make changes at Findit side to cover this case properly.

robertma@ To double check: if a swarming task for webkit_layout_tests was to ask to run some non-existing tests, it will end with no output at all?

Re #7:

If there's no test to run, the runner won't produce any artifacts (including the JSON); it will say "No tests to run." in the log. Furthermore, if the "--zero-tests-executed-ok" flag isn't given (the flag is used by default on trybots), the runner will exit with code 253.

Closing as wontfix, but feel free to reopen if you have any other question.

robertma@: For gtests, if there is no test to run, it'll still keep an empty result (see https://chromium-swarm.appspot.com/task?id=3e38fca744f72610&refresh=10&show_raw=1).

Is it possible to have something like
{"tests":{},"skipped":0,"num_regressions":0,"build_number":"DUMMY_BUILD_NUMBER","interrupted":false,"chromium_revision":"","random_order_seed":4,"layout_tests_dir":"/b/s/w/ir/third_party/WebKit/LayoutTests","path_delimiter":"/","version":3,"builder_name":"","num_passes":0,"pixel_tests_enabled":true,"seconds_since_epoch":1529429950,"fixable":0,"num_flaky":0,"num_failures_by_type":{"SKIP":0,"CRASH":1,"MISSING":0,"TEXT":0,"IMAGE":0,"PASS":1,"REBASELINE":0,"LEAK":0,"SLOW":0,"TIMEOUT":0,"IMAGE+TEXT":0,"FAIL":0,"WONTFIX":0,"AUDIO":0,"NEEDSMANUALREBASELINE":0},"flag_name":null} if there's no test to run for webkit_layout_tests?

Without this, Findit thinks the task failed in an unexpected way, rather than realizes the test doesn't exist.

Gentle ping.

Sorry I somehow missed this.

I gave it a try and had something close to what you described in #9. The major difference is that "tests" won't be empty, but will contain a list of WONTFIX tests (due to an implementation detail of how they are added). It should be possible to change that, but it's probably not necessary.

https://chromium-review.googlesource.com/c/chromium/src/+/1117623

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3aecd1ec86298a2f6ef3750bf5d12c763219a700

commit 3aecd1ec86298a2f6ef3750bf5d12c763219a700
Author: Robert Ma <robertma@chromium.org>
Date: Thu Jun 28 15:10:58 2018

[blinkpy] Produce valid results for --zero-tests-executed-ok

When this flag is specified (e.g. in tryjob retry without patch &
FindIt), the test runner doesn't fail if no tests are run (e.g. because
of an empty test list). Previously, the runner would still exit early
and produce no artifacts in such case. This CL changes the behaviour to
keep executing and produce valid empty (*) test result JSONs.

* Caveat: the JSONs aren't really empty; they still contain a list of
WONTFIX (and hence skipped) tests because of implementation details.

Bug:  851639 
Change-Id: Idfc990263e130bc02dc441bc0d375a149e8b44d3
Reviewed-on: https://chromium-review.googlesource.com/1117623
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Robert Ma <robertma@chromium.org>
Cr-Commit-Position: refs/heads/master@{#571127}
[modify] https://crrev.com/3aecd1ec86298a2f6ef3750bf5d12c763219a700/third_party/blink/tools/blinkpy/web_tests/controllers/manager.py

chanli@: this should be fixed now. Please try again and verify. Thanks!

 Issue 857210  has been merged into this issue.

robertma: Here is a task I ran this morning: https://chromium-swarm.appspot.com/task?id=3e6052a68a48db10&refresh=10&show_raw=1

I still don't see isolated outputs?

chanli@ what's the Chromium revision of this task?

Unfortunately we can't change the history, so if you try to run a non-existent test before 3aecd1ec86298a2f6ef3750bf5d12c763219a700 there will still be no output.

I indeed reran an old task. I'll try using a revision newer than that. 

robertma@: Yes task on a newer revision has isolated outputs now: https://chromium-swarm.appspot.com/task?id=3e60f797b03b0c10&refresh=10&show_raw=1

Thank you!

No problem :)