For reference, here's what we have to do now: https://chromium-review.googlesource.com/c/chromium/tools/build/+/804399/5/scripts/slave/recipe_modules/ios/api.py#557

It would be much more robust if we can list these names programmatically.

I think a better way would be to programatically inspect the executable (for XCTests) or the shared library (for XCUITests), for two reasons:

1. this is how XCode and XCTests/XCUITests framework work
2. this does not require running anything on the bots

A quick and dirty way to do this is to parse the output of "otool -ov" that dump the content of the Objective-C section from the object file. From that, it is easy to find test methods, either by using a single regexp, or by doing a more robust parsing of the output. It is also possible to write code to parse the Mach-O format and extract the information directly (but this may be overengineering).

For example, I just build ios_web_shell_egtests, and running otool -ov, I get the following output (abbreviated since output is large):

$ otool -ov out/Debug-iphonesimulator/ios_web_shell_egtests.app/ios_web_shell_egtests
...
0000000102565150 0x10257cb80 _OBJC_CLASS_$_ContextMenuTestCase
           isa 0x10257cb58 _OBJC_METACLASS_$_ContextMenuTestCase
    superclass 0x10257c7c0 _OBJC_CLASS_$_WebShellTestCase
         cache 0x0
        vtable 0x0
          data 0x1025698e8 (struct class_ro_t *)
                    flags 0x90
            instanceStart 8
             instanceSize 8
                 reserved 0x0
               ivarLayout 0x0
                     name 0x10234be7e ContextMenuTestCase
              baseMethods 0x1025698b0 (struct method_list_t *)
                   entsize 24
                     count 2
                      name 0x102278c0d testContextMenu
                     types 0x10234cb79 v16@0:8
                       imp -[ContextMenuTestCase testContextMenu]
                      name 0x102278c1d testContextMenuWebkitTouchCalloutOverride
                     types 0x10234cb79 v16@0:8
                       imp -[ContextMenuTestCase testContextMenuWebkitTouchCalloutOverride]
            baseProtocols 0x0
                    ivars 0x0
           weakIvarLayout 0x0
           baseProperties 0x0
...

From that we can see that a class named ContextMenuTestCase exists, has WebShellTestCase for super-class and has two methods testContextMenu and testContextMenuWebkitTouchCalloutOverride. It looks like all method follow the pattern "imp -\[[A-Za-z_][A-Za-z0-9_]*TestCase test[A-Za-z0-9_]*\]". In fact, the following command output all the test that are part of ios_web_shell_egtests:

$ otool -ov out/Debug-iphonesimulator/ios_web_shell_egtests.app/ios_web_shell_egtests | \
  grep 'imp -\[[A-Za-z_][A-Za-z0-9_]*TestCase test[A-Za-z0-9_]*\]'
		       imp -[ContextMenuTestCase testContextMenu]
		       imp -[ContextMenuTestCase testContextMenuWebkitTouchCalloutOverride]
		       imp -[MetaTagsTestCase testMetaRefresh0Seconds]
		       imp -[MetaTagsTestCase testMetaRefresh3Seconds]
		       imp -[NavigationTestCase testNavigationLinkToAboutBlank]
		       imp -[NavigationTestCase testNavigationBackAndForward]
		       imp -[NavigationTestCase testNavigationBackAndForwardAfterFragmentLink]
		       imp -[NavigationTestCase testNavigationLinkPreventDefaultOverridesHref]
		       imp -[NavigationTestCase testNavigationUnsupportedSchema]
		       imp -[PageStateTestCase testScrollPositionRestoring]
		       imp -[PageStateTestCase testZeroContentOffsetAfterLoad]
		       imp -[PDFTestCase testMIMEType]
		       imp -[PluginPlaceholderTestCase testPluginPlaceholderAppletFallback]
		       imp -[PluginPlaceholderTestCase testPluginPlaceholderAppletOnly]
		       imp -[PluginPlaceholderTestCase testPluginPlaceholderObjectFlashEmbedFallback]
		       imp -[PluginPlaceholderTestCase testPluginPlaceholderObjectUndefinedEmbedFallback]
		       imp -[PluginPlaceholderTestCase testPluginPlaceholderObjectFallback]
		       imp -[PluginPlaceholderTestCase testPluginPlaceholderObjectOnly]
		       imp -[PluginPlaceholderTestCase testPluginPlaceholderPNGObject]
		       imp -[PluginPlaceholderTestCase testPluginPlaceholderSmallFlash]
		       imp -[RedirectTestCase testMultipleRedirects]
		       imp -[RedirectTestCase testRedirection301]
		       imp -[RedirectTestCase testRedirection302]
		       imp -[ServiceManagerTestCase testConnectionToAllUsersEmbeddedService]
		       imp -[ServiceManagerTestCase testConnectionToPerUserEmbeddedService]
		       imp -[ServiceManagerTestCase testConnectionToWebStateInterface]
		       imp -[ServiceManagerTestCase testConnectionToNetworkServiceTest]

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/96d2150bdd863707a4fa129ef0b064dbe890712b

commit 96d2150bdd863707a4fa129ef0b064dbe890712b
Author: Menglu Huang <huangml@chromium.org>
Date: Wed Feb 21 22:50:59 2018

Refactor iOS test listing when isolating

Bug:791807
Change-Id: Icf9f5b2304d50f69646aa3ad47d40a7a0a103c4c
Reviewed-on: https://chromium-review.googlesource.com/929785
Commit-Queue: Menglu Huang <huangml@chromium.org>
Reviewed-by: smut <smut@google.com>

[modify] https://crrev.com/96d2150bdd863707a4fa129ef0b064dbe890712b/scripts/slave/README.recipes.md
[modify] https://crrev.com/96d2150bdd863707a4fa129ef0b064dbe890712b/scripts/slave/recipe_modules/ios/api.py
[modify] https://crrev.com/96d2150bdd863707a4fa129ef0b064dbe890712b/scripts/slave/recipe_modules/ios/examples/full.expected/basic.json
[modify] https://crrev.com/96d2150bdd863707a4fa129ef0b064dbe890712b/scripts/slave/recipe_modules/ios/examples/full.py