cros flash fails to complete through ethernet. The problem appears to be cros_generate_stateful_update_payload.

cros flash 100.127.1.198 chromeos_10303.0.0_veyron-minnie_recovery_canary-channel_minnie-mp-v4.bin
11:14:31: NOTICE: Preparing to update the remote device 100.127.1.198
11:15:30: ERROR: Devserver responded with HTTP error (HTTP Error 500: Internal Server Error)
11:15:30: WARNING: --- Start output from /tmp/devserver_wrapperyoTCpC/dev_server.log ---[22/Jan/2018:11:14:34] ENGINE Serving on :::0
[22/Jan/2018:11:14:34] ENGINE Bus STARTED
::ffff:127.0.0.1 - - [22/Jan/2018:11:14:38] "GET /check_health HTTP/1.1" 200 370 "" "Python-urllib/2.7"
[22/Jan/2018:11:14:39] XBUDDY Linking to /mnt/host/source/devserver/static/veyron_minnie/R65-10314.0.2018_01_16_1814-a1 from /mnt/host/source/src/build/images/veyron_minnie/R65-10314.0.2018_01_16_1814-a1
[22/Jan/2018:11:14:39] XBUDDY Linking to /mnt/host/source/devserver/static/veyron_minnie/R65-10317.0.2018_01_17_1151-a1 from /mnt/host/source/src/build/images/veyron_minnie/R65-10317.0.2018_01_17_1151-a1
[22/Jan/2018:11:14:39] XBUDDY Path is local_imagehZenvU/link/test, location suffix is -release
[22/Jan/2018:11:14:39] XBUDDY Get artifact 'test' with board local_imagehZenvU and version link'. Locally? True
[22/Jan/2018:11:14:39] XBUDDY Returning path to payload: local_imagehZenvU/link/chromiumos_test_image.bin
[22/Jan/2018:11:14:39] XBUDDY Downloading ['full_payload', 'stateful'] from gs://chromeos-image-archive/local_imagehZenvU/link
[22/Jan/2018:11:14:39] GOOGLE_STORAGE_DOWNLOADER Downloading artifacts *_full_*->/mnt/host/source/devserver/static/local_imagehZenvU/link stateful.tgz->/mnt/host/source/devserver/static/local_imagehZenvU/link.
[22/Jan/2018:11:14:39] A_U_TEST_PAYLOAD No marker file, *_full_*->/mnt/host/source/devserver/static/local_imagehZenvU/link is not staged.
[22/Jan/2018:11:14:39] XBUDDY Linking to /mnt/host/source/devserver/static/veyron_minnie/R65-10314.0.2018_01_16_1814-a1 from /mnt/host/source/src/build/images/veyron_minnie/R65-10314.0.2018_01_16_1814-a1
[22/Jan/2018:11:14:39] XBUDDY Linking to /mnt/host/source/devserver/static/veyron_minnie/R65-10317.0.2018_01_17_1151-a1 from /mnt/host/source/src/build/images/veyron_minnie/R65-10317.0.2018_01_17_1151-a1
[22/Jan/2018:11:14:39] XBUDDY Path is local_imagehZenvU/link/test, location suffix is -release
[22/Jan/2018:11:14:39] XBUDDY Get artifact 'test' with board local_imagehZenvU and version link'. Locally? True
[22/Jan/2018:11:14:39] XBUDDY Updating timestamp for local_imagehZenvU/link
[22/Jan/2018:11:14:39] XBUDDY Returning path to payload: local_imagehZenvU/link/chromiumos_test_image.bin
[22/Jan/2018:11:14:39] DEVSERVER Payload generation triggered by request
[22/Jan/2018:11:14:39] UPDATE Update label/file: local_imagehZenvU/link/chromiumos_test_image.bin
[22/Jan/2018:11:14:39] UPDATE Generating update for src  image /mnt/host/source/devserver/static/local_imagehZenvU/link/chromiumos_test_image.bin
[22/Jan/2018:11:14:44] UPDATE Caching in sub_dir "cache/3b48fa862626cd95db633477f0bd46a4"
[22/Jan/2018:11:14:44] UPDATE Generating update for image /mnt/host/source/devserver/static/local_imagehZenvU/link/chromiumos_test_image.bin
[22/Jan/2018:11:14:44] UPDATE Generating update image /mnt/host/source/devserver/static/cache/3b48fa862626cd95db633477f0bd46a4/update.gz
[22/Jan/2018:11:14:44] UPDATE Running cros_generate_update_payload --image /mnt/host/source/devserver/static/local_imagehZenvU/link/chromiumos_test_image.bin --out_metadata_hash_file /mnt/host/source/devserver/static/cache/3b48fa862626cd95db633477f0bd46a4/metadata_hash --output /mnt/host/source/devserver/static/cache/3b48fa862626cd95db633477f0bd46a4/update.gz
[22/Jan/2018:11:15:28] UPDATE Running cros_generate_stateful_update_payload --image /mnt/host/source/devserver/static/local_imagehZenvU/link/chromiumos_test_image.bin --output_dir /mnt/host/source/devserver/static/cache/3b48fa862626cd95db633477f0bd46a4
[22/Jan/2018:11:15:30] HTTP Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/cherrypy/_cprequest.py", line 656, in respond
    response.body = self.handler()
  File "/usr/lib64/python2.7/site-packages/cherrypy/lib/encoding.py", line 188, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/lib/devserver/devserver.py", line 1514, in xbuddy
    image_name=file_name)
  File "/usr/lib64/devserver/autoupdate.py", line 500, in GetUpdateForLabel
    rel_path = self.GenerateUpdateImageWithCache(static_image_path)
  File "/usr/lib64/devserver/autoupdate.py", line 433, in GenerateUpdateImageWithCache
    self.GenerateUpdateImage(image_path, cache_dir)
  File "/usr/lib64/devserver/autoupdate.py", line 399, in GenerateUpdateImage
    raise AutoupdateError('Failed to generate update in %s' % output_dir)
AutoupdateError: Failed to generate update in /mnt/host/source/devserver/static/cache/3b48fa862626cd95db633477f0bd46a4

[22/Jan/2018:11:15:30] HTTP 
Request Headers:
  HOST: 172.22.112.223:34031
  CONNECTION: close
  Remote-Addr: ::ffff:172.22.112.223
  USER-AGENT: Python-urllib/2.7
  ACCEPT-ENCODING: identity
::ffff:172.22.112.223 - - [22/Jan/2018:11:15:30] "GET /xbuddy/local_imagehZenvU/link/test?for_update=true&return_dir=true HTTP/1.1" 500 1890 "" "Python-urllib/2.7"
--- End output from /tmp/devserver_wrapperyoTCpC/dev_server.log ---
11:15:31: ERROR: Unable to get payloads from local path: /tmp/cros-flash37FwI3
11:15:31: ERROR: Device update failed.
11:15:32: ERROR: cros flash failed before completing.
11:15:32: ERROR: HTTP Error 500: Internal Server Error


When I run the cros_generate_statefull_update_payload, I get the following error:

ros_generate_stateful_update_payload --image chromeos_10303.0.0_veyron-minnie_recovery_canary-channel_minnie-mp-v4.bin --output_dir mnt/
INFO:cros_generate_stateful_update_payload:Generating stateful update file.
mount: special device /tmp/tmpz8V9YHstateful/var_overlay does not exist
ERROR:cros_generate_stateful_update_payload:Failed to create stateful update file
INFO    : Unmounting image from /tmp/tmpz8V9YHstateful and /tmp/tmpx5Dk2qrootfs
INFO    : Unmounting temporary rootfs /tmp/tmpx5Dk2qrootfs//build/rootfs.
WARNING : Umount called without passing the image. Some filesystems can't be unmounted in this way.
Traceback (most recent call last):
  File "/usr/bin/cros_generate_stateful_update_payload", line 109, in <module>
    main()
  File "/usr/bin/cros_generate_stateful_update_payload", line 105, in main
    options.output_dir, logger)
  File "/usr/bin/cros_generate_stateful_update_payload", line 59, in GenerateStatefulPayload
    '--stateful_mountpt=%s' % stateful_dir,
  File "/usr/lib64/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/mnt/host/source/src/scripts/mount_gpt_image.sh', '--from=/mnt/host/source/squashfs-test', '--image=chromeos_10303.0.0_veyron-minnie_recovery_canary-channel_minnie-mp-v4.bin', '--read_only', '--rootfs_mountpt=/tmp/tmpx5Dk2qrootfs', '--stateful_mountpt=/tmp/tmpz8V9YHstateful']' returned non-zero exit status 32