Are you using the system version of Python? If not, it looks like the MacOS firewall by default requires confirmation for unsigned binaries to accept incoming connections:
https://superuser.com/questions/912656/how-do-i-stop-my-mac-from-asking-to-accept-incoming-network-connections

I don't see an obvious way to get around this for run-webkit-tests.

I recently got a new machine (new macbook pro) and, hopefully, have a pretty vanilla setup. "which python" shows /usr/bin/python which version 2.7.10. I just ran the tests with "--debug-rwt-logging --verbose" and I see the following:
---------------8<---------------
18:05:53.248 23785 Starting wptserve server, cmd="['/Users/pdr/.vpython/ee44b3/bin/python', '-u', '/Users/pdr/Desktop/chromium/src/third_party/WebKit/Tools/Scripts/webkitpy/thirdparty/wpt/wpt/serve', '--config', '/Users/pdr/Desktop/chromium/src/third_party/WebKit/Tools/Scripts/webkitpy/thirdparty/wpt/wpt.config.json', '--doc_root', '/Users/pdr/Desktop/chromium/src/third_party/WebKit/LayoutTests/external/wpt', '--ws_doc_root', '/Users/pdr/Desktop/chromium/src/third_party/WebKit/LayoutTests/external/wpt/websockets/handlers']"
18:05:53.255 23785 Checking whether pid 23795 is alive.
18:05:53.257 23785 Server NOT running on 8001: [Errno 61] Connection refused
18:05:53.257 23785 Waiting for action: <bound method WPTServe._is_server_running_on_all_ports of <webkitpy.layout_tests.servers.wptserve.WPTServe object at 0x10d6a92d0>>
---------------8<---------------

I don't know much about python but it looks like I might be using vpython (/Users/pdr/.vpython/ee44b3/bin/python)? Could my non-default python be from the infra vpython efforts (https://docs.google.com/document/d/1GKN7rtW6-006RbG_Ryh6vr4TpUxSxJlPSCKFLq2nOgA)?

Hm 🤔 that makes sense to explain the popup window, but I'm not familiar with the vpython efforts. Adding dnj@ who might have more context.

Since it looks like the vpython Python binary is a moving target, I'm not sure what a long-term solution would look like. It looks like you can force sign a binary to be accepted with the following:

sudo codesign --force --sign - /Users/pdr/.vpython/ee44b3/bin/python

(via https://forums.plex.tv/discussion/184717/python-app-blocked-in-mac-firewall)

"vpython" is a wrapper around VirtualEnv + Python. It looks like the wrapper part is working, since you're running the VirtualEnv's Python.

While "vpython" is the source of the current VirutalEnv installation that you're using, any VirtualEnv installation is expected to operate in the same manner.

Unfortunately, it looks like the only approach to resolving this is to sign the Python binary. This isn't really ideal :(

It looks like VirtualEnv modifies the base Python binary when it constructs an environment. Good news is that Python instances in different VirutalEnvs seem to all share the same hash, which suggests that a "sign once, run forever" approach may be possible. I'll see if I can reproduce and update.

In the meantime, please just sign the specific VirtualEnv binary as suggested in #3 to unblock.

More info: The modification is to make the binary relocatable, and seems unavoidable. https://github.com/pypa/virtualenv/blob/master/virtualenv.py#L1327

So looking into this deeper, there are lots of options we can take to sign VirutalEnv Pythons and/or have a tool like vpython help with this. None of them are really great from a security or implementation perspective.

HOWEVER, I have observed that the Mac firewall doesn't seem to care about listening sockets that bound to localhost. The Mac firewall only complains when a socket attempts to listen on a port bound to a non-localhost address. Where is the code that is listening on this socket? Is that code candidate to bind to localhost instead of a public address? If so, the solution here is to just fix that, for this, security, and port contention reasons.

WDYT?

+jeffcarp@, any idea what the code in question is?

It looks like Apache listens on 127.0.0.1:
https://cs.chromium.org/chromium/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/servers/apache_http.py?q=servers/apache_http.py&sq=package:chromium&l=121

But that wouldn't explain why MacOS is asking about 'python'. I know that WPTServe is a Python server. And it looks like it does listen on 127.0.0.1 as well:
https://cs.chromium.org/chromium/src/third_party/WebKit/Tools/Scripts/webkitpy/thirdparty/wpt/wpt/tools/serve/serve.py?q=tools/serve/serve.py&sq=package:chromium&dr&l=284

Possibly: https://cs.chromium.org/chromium/src/third_party/WebKit/Tools/Scripts/webkitpy/thirdparty/wpt/wpt/tools/serve/serve.py?q=src/third_party/webkit/.*%5C.py$+WebSocketServer+package:%5Echromium$&dr=C&l=513 ?

Only other instance of "bind" I could find in WebKitPy.

Do you think there could be a difference in the way MacOS security treats 127.0.0.1 vs. localhost?

That's entirely possible. Seems unlikely, since "127.0.0.1" is specced to be a loopback address in IPv4, so it's odd that it might try and bind something that would hit a firewall.

If the script were accessing "localhost", I could see someone's "/etc/hosts" or something override it, but all of the scripts are hardcoding a loopback address :/

It's also possible that my local test wasn't actually reproducing things. I'll try running the actual command in #0 in a few and see if I can get the firewall to complain.

This issue has been Available for over a year. If it's no longer important or seems unlikely to be fixed, please consider closing it out. If it is important, please re-triage the issue.

Sorry for the inconvenience if the bug really should have been left as Available.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot