We are seeing increasing number of scheduler crash from exception raised from copy_file_or_directory, for example:

04/05 05:43:55.655 ERROR|     email_manager:0054| monitor_db exception
EXCEPTION: Uncaught exception; terminating monitor_db
Traceback (most recent call last):
  File "/usr/local/autotest/scheduler/monitor_db.py", line 179, in main_without_exception_handling
    dispatcher.tick()
  File "/usr/local/autotest/site-packages/statsd/timer.py", line 95, in _decorator
    return function(*args, **kwargs)
  File "/usr/local/autotest/scheduler/site_monitor_db.py", line 106, in tick
    super(SiteDispatcher, self).tick()
  File "/usr/local/autotest/scheduler/monitor_db.py", line 374, in tick
    _drone_manager.execute_actions()
  File "/usr/local/autotest/site-packages/statsd/timer.py", line 95, in _decorator
    return function(*args, **kwargs)
  File "/usr/local/autotest/scheduler/site_drone_manager.py", line 89, in execute_actions
    super(SiteDroneManager, self).execute_actions()
  File "/usr/local/autotest/scheduler/drone_manager.py", line 563, in execute_actions
    self._drones.values())
  File "/usr/local/autotest/scheduler/thread_lib.py", line 202, in execute
    return self.get_results() if wait else None
  File "/usr/local/autotest/scheduler/thread_lib.py", line 148, in get_results
    self.wait_on_drones()
  File "/usr/local/autotest/scheduler/thread_lib.py", line 136, in wait_on_drones
    raise drone_task_queue.DroneTaskQueueException(exception_msg)
DroneTaskQueueException: Drone chromeos-server2.mtv.corp.google.com raised Exception command execution error
* Command:
    /usr/bin/ssh -a -x  -o ControlPath=/tmp/_autotmp_ujfMzbssh-master/socket
    -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/tmpCRVcuc -o
    BatchMode=yes -o ConnectTimeout=300 -o ServerAliveInterval=300 -l
    chromeos-test -p 22 chromeos-server2.mtv.corp.google.com " python
    /usr/local/autotest/scheduler/drone_utility.py --call_time 1459860230.4"
Exit status: 1
Duration: 1.23830795288

stderr:
Traceback (most recent call last):
  File "/usr/local/autotest/scheduler/drone_utility.py", line 618, in <module>
    main()
  File "/usr/local/autotest/scheduler/drone_utility.py", line 612, in main
    return_value = drone_utility.execute_calls(calls)
  File "/usr/local/autotest/scheduler/drone_utility.py", line 540, in execute_calls
    results.append(method_call.execute_on(self))
  File "/usr/local/autotest/scheduler/drone_utility.py", line 54, in execute_on
    return method(*self._args, **self._kwargs)
  File "/usr/local/autotest/scheduler/drone_utility.py", line 409, in copy_file_or_directory
    os.path.join(destination_path, filename))
  File "/usr/local/autotest/scheduler/drone_utility.py", line 411, in copy_file_or_directory
    shutil.copytree(source_path, destination_path, symlinks=True)
  File "/usr/lib/python2.7/shutil.py", line 208, in copytree
    raise Error, errors

shutil.Error: [(u'/usr/local/autotest/results/hosts/chromeos4-row5-rack7-host11/53466533-provision/provision_FirmwareUpdate/sysinfo/reboot_current/mnt/stateful_partition/unencrypted/preserve/log/messages', .... ]

The missing file does not exist in GS either:
https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/hosts/chromeos4-row5-rack7-host11/53466533-provision/provision_FirmwareUpdate/

We may want to handle such error gracefully without crashing scheduler, since it's not fatal errors that leads to db corruption.