Hi Dan, could you please attach the log to issue tracker ? Somehow the file is misssing on the server.
"The requested URL /results/hosts/chromeos1-row5-rack1-host2/54298164-cleanup/debug/autoserv.DEBUG was not found on this server."
I checked the code and it seems that is already a 30 seconds timeout on the call.
https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/server/hosts/chameleon_host.py?type=cs&q=_wait_for_connection_established+package:%5Echromeos_public$&l=124

So... it would be interesting to find out why that 30 seconds timeout did not work.
Thanks!

The full test logs are here:
https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/hosts/chromeos1-row5-rack1-host2/54298164-cleanup/debug/

I forgot that link is on drone.

I checked that it set the tunnel up before the connection. In the lab, the tunnel should not be triggered. Probably some config changed to force the tunneling.

The code has a timeout 30sec and keeps reconnecting to Chameleon. However, according to the log, it was interrupted, because of some expected exception, like tunnel error.

    def _wait_for_connection_established(self):
        """Wait for ChameleonConnection through tunnel being established."""
        def _create_connection():
            """Create ChameleonConnection.
            @returns: True if success. False otherwise.
            """
            try:
                self._chameleon_connection = chameleon.ChameleonConnection(
                        'localhost', self._local_port)
            except chameleon.ChameleonConnectionError:
                logging.debug('Connection is not ready yet ...')
                return False
            logging.debug('Connection is up')
            return True

        success = utils.wait_for_value(
            _create_connection, expected_value=True, timeout_sec=30)
        if not success:
            raise ChameleonHostError('Can not connect to Chameleon')

Assign to xixuan@ to investigate it. Better to turn the tunneling config off first to avoid this issue happening again.

oh, the current config is, 

if the host in lab && we don't open the ssh_config, tunnel won't be triggered.

Since we open the ssh_to_chameleon config, any connection to chameleon host will go through tunnel.

if self._is_in_lab and not ENABLE_SSH_TUNNEL_FOR_CHAMELEON:
      self._chameleon_connection = chameleon.ChameleonConnection(
             self.hostname, chameleon_port)
else:
      self._create_connection_through_tunnel()

The tunneling was used in the dev network, not verified in the lab environment. I don't why it was enabled in the lab and this issue happened. It is not a Chameleon issue, probably related to the lab network.

Xixuan, please investigate the cause, or simply disable the tunneling in the lab before this issue gets fixed.

The issue can't be reproduced so it's hard to see whether it gets fixed or not. At this point, we have to set tunneling for the new ACL https://buganizer.corp.google.com/u/0/issues/25934953.

The current chameleon tunnel code path is different from the code path of servo tunnel. I will first let chameleon use the same code path of servo, and then add some debugging logs if possible, to report why it's stucked. 

 Issue 588806  has been merged into this issue.

 Issue 600007  has been merged into this issue.

 Issue 609356  has been merged into this issue.

https://chromium-review.googlesource.com/#/c/340544/

Closing. please reopen if its not fixed.