ALL of the HW Lab machines belonging to the toolchain team seem to be in a bad state or have some new kind of permissions problem.  This would include the following machines:

chromeos2-row1-rack4-host7.cros
chromeos2-row1-rack4-host8.cros
chromeos2-row1-rack4-host9.cros
chromeos2-row1-rack7-host1.cros
chromeos2-row1-rack7-host2.cros
chromeos2-row3-rack10-host8.cros
chromeos2-row3-rack3-host1.cros
chromeos2-row3-rack3-host2.cros
chromeos2-row3-rack3-host3.cros
chromeos2-row3-rack2-host14.cros
chromeos2-row3-rack2-host15.cros
chromeos2-row3-rack2-host16.cros
chromeos2-row1-rack6-host4.cros
chromeos2-row1-rack6-host5.cros
chromeos2-row1-rack6-host3.cros
chromeos2-row1-rack5-host3.cros
chromeos2-row1-rack5-host4.cros
chromeos2-row1-rack5-host5.cros
chromeos2-row1-rack5-host6.cros
chromeos2-row1-rack6-host1.cros
chromeos2-row1-rack6-host2.cros
chromeos2-row1-rack6-host6.cros
chromeos2-row1-rack5-host1.cros
chromeos2-row1-rack5-host2.cros

I can ping the machines and I can ssh into the machines.  However I cannot flash an image onto the machines; 'cros flash' consistently fails with something like the following:

13:49:26: NOTICE: Preparing to update the remote device chromeos2-row1-rack5-host3.cros
13:52:08: ERROR: Devserver responded with an error!
13:52:08: WARNING: --- Start output from /tmp/devserver_wrapperS4chIc/dev_server.log ---[25/Jul/2016:13:50:10] ENGINE (wait_for_free_port) No cached port to wait for.
[25/Jul/2016:13:50:10] ENGINE (wait_for_occupied_port) Waiting for actual port 44396.
[25/Jul/2016:13:50:10] ENGINE Port 44396 written to '/tmp/devserver_wrapperS4chIc/dev_server.port'.
[25/Jul/2016:13:50:10] ENGINE Serving on :::0
[25/Jul/2016:13:50:10] ENGINE Bus STARTED
::ffff:127.0.0.1 - - [25/Jul/2016:13:50:15] "GET /check_health HTTP/1.1" 200 327 "" "Python-urllib/2.7"
[25/Jul/2016:13:50:15] XBUDDY Cannot find /mnt/host/source/src/build/images; skip syncing image registry.
[25/Jul/2016:13:50:15] XBUDDY Path is local_imagem0YEx0/link/test, location suffix is -release
[25/Jul/2016:13:50:15] XBUDDY Get artifact 'test' with board local_imagem0YEx0 and version link'. Locally? True
[25/Jul/2016:13:50:15] XBUDDY Returning path to payload: local_imagem0YEx0/link/chromiumos_test_image.bin
[25/Jul/2016:13:50:15] XBUDDY Downloading ['full_payload', 'stateful'] from gs://chromeos-image-archive/local_imagem0YEx0/link
[25/Jul/2016:13:50:15] GOOGLE_STORAGE_DOWNLOADER Downloading artifacts *_full_*->/mnt/host/source/devserver/static/local_imagem0YEx0/link stateful.tgz->/mnt/host/source/devserver/static/local_imagem0YEx0/link.
[25/Jul/2016:13:50:17] XBUDDY Cannot find /mnt/host/source/src/build/images; skip syncing image registry.
[25/Jul/2016:13:50:17] XBUDDY Path is local_imagem0YEx0/link/test, location suffix is -release
[25/Jul/2016:13:50:17] XBUDDY Get artifact 'test' with board local_imagem0YEx0 and version link'. Locally? True
[25/Jul/2016:13:50:17] XBUDDY Updating timestamp for local_imagem0YEx0/link
[25/Jul/2016:13:50:17] XBUDDY Returning path to payload: local_imagem0YEx0/link/chromiumos_test_image.bin
[25/Jul/2016:13:50:17] DEVSERVER Payload generation triggered by request
[25/Jul/2016:13:50:17] UPDATE Update label/file: local_imagem0YEx0/link/chromiumos_test_image.bin
[25/Jul/2016:13:50:17] UPDATE Generating update for src  image /mnt/host/source/devserver/static/local_imagem0YEx0/link/chromiumos_test_image.bin
[25/Jul/2016:13:50:22] UPDATE Caching in sub_dir "cache/3d3c488b7aab407f819634266a1a7623"
[25/Jul/2016:13:50:22] UPDATE Generating update for image /mnt/host/source/devserver/static/local_imagem0YEx0/link/chromiumos_test_image.bin
[25/Jul/2016:13:50:22] UPDATE Generating update image /mnt/host/source/devserver/static/cache/3d3c488b7aab407f819634266a1a7623/update.gz
[25/Jul/2016:13:50:22] UPDATE Running cros_generate_update_payload --image /mnt/host/source/devserver/static/local_imagem0YEx0/link/chromiumos_test_image.bin --out_metadata_hash_file /mnt/host/source/devserver/static/cache/3d3c488b7aab407f819634266a1a7623/metadata_hash --output /mnt/host/source/devserver/static/cache/3d3c488b7aab407f819634266a1a7623/update.gz
[25/Jul/2016:13:51:40] UPDATE Running cros_generate_stateful_update_payload --image /mnt/host/source/devserver/static/local_imagem0YEx0/link/chromiumos_test_image.bin --output_dir /mnt/host/source/devserver/static/cache/3d3c488b7aab407f819634266a1a7623
[25/Jul/2016:13:52:08] HTTP Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/cherrypy/_cprequest.py", line 656, in respond
    response.body = self.handler()
  File "/usr/lib64/python2.7/site-packages/cherrypy/lib/encoding.py", line 188, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/lib/devserver/devserver.py", line 1076, in xbuddy
    image_name=file_name)
  File "/usr/lib64/devserver/autoupdate.py", line 487, in GetUpdateForLabel
    rel_path = self.GenerateUpdateImageWithCache(static_image_path)
  File "/usr/lib64/devserver/autoupdate.py", line 420, in GenerateUpdateImageWithCache
    self.GenerateUpdateImage(image_path, cache_dir)
  File "/usr/lib64/devserver/autoupdate.py", line 386, in GenerateUpdateImage
    raise AutoupdateError('Failed to generate update in %s' % output_dir)
AutoupdateError: Failed to generate update in /mnt/host/source/devserver/static/cache/3d3c488b7aab407f819634266a1a7623

[25/Jul/2016:13:52:08] HTTP 
Request Headers:
  HOST: 172.17.222.59:44396
  CONNECTION: close
  Remote-Addr: ::ffff:172.17.222.59
  USER-AGENT: Python-urllib/2.7
  ACCEPT-ENCODING: identity
::ffff:172.17.222.59 - - [25/Jul/2016:13:52:08] "GET /xbuddy/local_imagem0YEx0/link/test?for_update=true&return_dir=true HTTP/1.1" 500 1890 "" "Python-urllib/2.7"
--- End output from /tmp/devserver_wrapperS4chIc/dev_server.log ---
13:52:08: ERROR: Device update failed.
13:52:09: ERROR: Error connecting to device chromeos2-row1-rack5-host3.cros
13:52:09: ERROR: cros flash failed before completing.
13:52:09: ERROR: Warning: Identity file /tmp/ssh-tmpscJhzG/testing_rsa not accessible: No such file or directory.
Warning: Permanently added 'chromeos2-row1-rack5-host3.cros,172.17.40.51' (RSA) to the list of known hosts.
Permission denied (publickey,keyboard-interactive).

(cr) ((bd6dc0b...)) mobiletc-prebuild@chrotomation2 ~/trunk/src/scripts $ cros flash --board=peach_pit --clobber-stateful chromeos2-row1-rack5-host4.cros /tmp/peach_pit-release/R54-8631.0.0/chromiumos_test_image.bin
13:55:43: NOTICE: Preparing to update the remote device chromeos2-row1-rack5-host4.cros
13:57:44: ERROR: Devserver responded with an error!
13:57:44: WARNING: --- Start output from /tmp/devserver_wrapperwllJUG/dev_server.log ---[25/Jul/2016:13:55:48] ENGINE (wait_for_free_port) No cached port to wait for.
[25/Jul/2016:13:55:48] ENGINE (wait_for_occupied_port) Waiting for actual port 33434.
[25/Jul/2016:13:55:48] ENGINE Port 33434 written to '/tmp/devserver_wrapperwllJUG/dev_server.port'.
[25/Jul/2016:13:55:48] ENGINE Serving on :::0
[25/Jul/2016:13:55:48] ENGINE Bus STARTED
::ffff:127.0.0.1 - - [25/Jul/2016:13:55:52] "GET /check_health HTTP/1.1" 200 327 "" "Python-urllib/2.7"
[25/Jul/2016:13:55:52] XBUDDY Cannot find /mnt/host/source/src/build/images; skip syncing image registry.
[25/Jul/2016:13:55:52] XBUDDY Path is local_imagebDnm90/link/test, location suffix is -release
[25/Jul/2016:13:55:52] XBUDDY Get artifact 'test' with board local_imagebDnm90 and version link'. Locally? True
[25/Jul/2016:13:55:52] XBUDDY Returning path to payload: local_imagebDnm90/link/chromiumos_test_image.bin
[25/Jul/2016:13:55:52] XBUDDY Downloading ['full_payload', 'stateful'] from gs://chromeos-image-archive/local_imagebDnm90/link
[25/Jul/2016:13:55:52] GOOGLE_STORAGE_DOWNLOADER Downloading artifacts *_full_*->/mnt/host/source/devserver/static/local_imagebDnm90/link stateful.tgz->/mnt/host/source/devserver/static/local_imagebDnm90/link.
[25/Jul/2016:13:55:54] XBUDDY Cannot find /mnt/host/source/src/build/images; skip syncing image registry.
[25/Jul/2016:13:55:54] XBUDDY Path is local_imagebDnm90/link/test, location suffix is -release
[25/Jul/2016:13:55:54] XBUDDY Get artifact 'test' with board local_imagebDnm90 and version link'. Locally? True
[25/Jul/2016:13:55:54] XBUDDY Updating timestamp for local_imagebDnm90/link
[25/Jul/2016:13:55:54] XBUDDY Returning path to payload: local_imagebDnm90/link/chromiumos_test_image.bin
[25/Jul/2016:13:55:54] DEVSERVER Payload generation triggered by request
[25/Jul/2016:13:55:54] UPDATE Update label/file: local_imagebDnm90/link/chromiumos_test_image.bin
[25/Jul/2016:13:55:54] UPDATE Generating update for src  image /mnt/host/source/devserver/static/local_imagebDnm90/link/chromiumos_test_image.bin
[25/Jul/2016:13:56:01] UPDATE Caching in sub_dir "cache/3d3c488b7aab407f819634266a1a7623"
[25/Jul/2016:13:56:01] UPDATE Generating update for image /mnt/host/source/devserver/static/local_imagebDnm90/link/chromiumos_test_image.bin
[25/Jul/2016:13:56:01] UPDATE Generating update image /mnt/host/source/devserver/static/cache/3d3c488b7aab407f819634266a1a7623/update.gz
[25/Jul/2016:13:56:01] UPDATE Running cros_generate_update_payload --image /mnt/host/source/devserver/static/local_imagebDnm90/link/chromiumos_test_image.bin --out_metadata_hash_file /mnt/host/source/devserver/static/cache/3d3c488b7aab407f819634266a1a7623/metadata_hash --output /mnt/host/source/devserver/static/cache/3d3c488b7aab407f819634266a1a7623/update.gz
[25/Jul/2016:13:57:17] UPDATE Running cros_generate_stateful_update_payload --image /mnt/host/source/devserver/static/local_imagebDnm90/link/chromiumos_test_image.bin --output_dir /mnt/host/source/devserver/static/cache/3d3c488b7aab407f819634266a1a7623
[25/Jul/2016:13:57:44] HTTP Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/cherrypy/_cprequest.py", line 656, in respond
    response.body = self.handler()
  File "/usr/lib64/python2.7/site-packages/cherrypy/lib/encoding.py", line 188, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/lib/devserver/devserver.py", line 1076, in xbuddy
    image_name=file_name)
  File "/usr/lib64/devserver/autoupdate.py", line 487, in GetUpdateForLabel
    rel_path = self.GenerateUpdateImageWithCache(static_image_path)
  File "/usr/lib64/devserver/autoupdate.py", line 420, in GenerateUpdateImageWithCache
    self.GenerateUpdateImage(image_path, cache_dir)
  File "/usr/lib64/devserver/autoupdate.py", line 386, in GenerateUpdateImage
    raise AutoupdateError('Failed to generate update in %s' % output_dir)
AutoupdateError: Failed to generate update in /mnt/host/source/devserver/static/cache/3d3c488b7aab407f819634266a1a7623

[25/Jul/2016:13:57:44] HTTP 
Request Headers:
  HOST: 172.17.222.59:33434
  CONNECTION: close
  Remote-Addr: ::ffff:172.17.222.59
  USER-AGENT: Python-urllib/2.7
  ACCEPT-ENCODING: identity
::ffff:172.17.222.59 - - [25/Jul/2016:13:57:44] "GET /xbuddy/local_imagebDnm90/link/test?for_update=true&return_dir=true HTTP/1.1" 500 1890 "" "Python-urllib/2.7"
--- End output from /tmp/devserver_wrapperwllJUG/dev_server.log ---
13:57:44: ERROR: Device update failed.
13:57:45: ERROR: Error connecting to device chromeos2-row1-rack5-host4.cros
13:57:45: ERROR: cros flash failed before completing.
13:57:45: ERROR: Warning: Identity file /tmp/ssh-tmpb0DDBz/testing_rsa not accessible: No such file or directory.
Warning: Permanently added 'chromeos2-row1-rack5-host4.cros,172.17.40.52' (RSA) to the list of known hosts.
Permission denied (publickey,keyboard-interactive).


I have actively tested 8 of the machines.  I tried 'repairing' them via the Repair button on the AFE website; the Repair job appeared to work but 'cros flash' still failed with the same message. These failures started late last week.  What changed then?  Please fix this ASAP!!