The following paladin build failed because the bot died. 
https://uberchromegw.corp.google.com/i/chromeos/builders/lumpy-paladin/builds/25386/steps/HWTest%20%5Bbvt-inline%5D/logs/stdio

12:43:38: INFO: RunCommand: /b/cbuild/internal_master/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /tmp/cbuildbot-tmpD1KspD/tmp71Nyov/temp_summary.json --raw-cmd --task-name lumpy-paladin/R53-8497.0.0-rc1-bvt-inline --dimension os Ubuntu-14.04 --dimension pool default --print-status-updates --timeout 9000 --io-timeout 9000 --hard-timeout 9000 --expiration 1200 -- /usr/local/autotest/site_utils/run_suite.py --build lumpy-paladin/R53-8497.0.0-rc1 --board lumpy --suite_name bvt-inline --pool cq --num 6 --file_bugs False --priority CQ --timeout_mins 90 --retry True --max_retries 10 --minimum_duts 4 --offload_failures_only True -c 12:43:47: ERROR: Encountered swarming internal error: stdout: Priority was reset to 100 29881 2016-06-24 19:43:46.996 E: Internal error! Triggered task: lumpy-paladin/R53-8497.0.0-rc1-bvt-inline chromeos-server31-185: 2f9cc1f231af6c10 None

summary json content:
{u'shards': [{u'outputs_ref': None, u'server_versions': [u'2140-67813a2'], u'abandoned_ts': u'2016-06-24T19:43:46.029400', u'exit_codes': [], u'id': u'2f9cc1f231af6c10', u'started_ts': u'2016-06-24T19:43:39.414470', u'cost_saved_usd': None, u'internal_failure': True, u'completed_ts': None, u'state': 80, u'etag': u'"P-Uzw_7eqDwcsWG90Qq25N39ybs/Am-9d5Vj4CwDlzRY1uLxRPVZvW8"', u'bot_version': u'5198c46aaef40c75107723d5e36915da1e3f47e4', u'tags': [u'os:Ubuntu-14.04', u'pool:default', u'priority:100', u'user:'], u'outputs': [], u'durations': [], u'kind': u'swarming#taskItem', u'deduped_from': None, u'failure': False, u'modified_ts': u'2016-06-24T19:43:46.029400', u'user': u'', u'isolated_out': None, u'children_task_ids': [], u'created_ts': u'2016-06-24T19:43:39.056490', u'name': u'lumpy-paladin/R53-8497.0.0-rc1-bvt-inline', u'bot_dimensions': {u'cpu': [u'x86', u'x86-64'], u'machine_type': [u'n1-highmem-4'], u'gpu': [u'none'], u'cores': [u'6'], u'os': [u'Linux', u'Ubuntu', u'Ubuntu-14.04'], u'id': [u'chromeos-server31-185'], u'pool': [u'default']}, u'try_number': 1, u'costs_usd': [0.0], u'properties_hash': None, u'bot_id': u'chromeos-server31-185'}]}

swarming task:
https://chromeos-proxy.appspot.com/user/task/2f9cc1f231af6c10

Expected behavior:

the bot should retry on this error (internal_failure==True and state==80)
https://cs.corp.google.com/chromeos_public/chromite/cbuildbot/swarming_lib.py?rcl=5b4d2470839b24c2d53687c4080c0fb2f3113012&l=96

However, it was not retried in this case, need to figure out why cbuildbot did not retry on the error.