All CT builders are down:
https://chrome-swarming.appspot.com/botlist?c=id&c=os&c=task&c=status&f=status%3Aquarantined&f=pool%3ACT&l=100&q=ct&s=id%3Aasc

They were created as n-highmem-32 but on GCE console they are now showing up as "Machine type custom (32 vCPUs, 68.75 GB memory)". This seems to be confusing swarming:


Failed to call hook get_state(): u'custom-32-70400'
Traceback (most recent call last):
File "swarming_bot.1.zip/bot_code/bot_main.py", line 238, in _call_hook_safe
return _call_hook(chained, botobj, name, *args)
File "swarming_bot.1.zip/bot_code/bot_main.py", line 130, in hook
return func(chained, botobj, name, *args, **kwargs)
File "swarming_bot.1.zip/bot_code/bot_main.py", line 212, in _call_hook
return hook(botobj, *args, **kwargs)
File "swarming_bot.1.zip/config/bot_config.py", line 100, in hook
return func(*args, **kwargs)
File "swarming_bot.1.zip/config/bot_config.py", line 922, in get_state
state = os_utilities.get_state_all_devices_android(devices)
File "swarming_bot.1.zip/api/os_utilities.py", line 773, in get_state_all_devices_android
state = get_state()
File "swarming_bot.1.zip/api/os_utilities.py", line 915, in get_state
u'cost_usd_hour': get_cost_hour(),
File "swarming_bot.1.zip/utils/tools.py", line 205, in wrapper
v = func(*args)
File "swarming_bot.1.zip/api/os_utilities.py", line 415, in get_cost_hour
machine_cost = GCE_MACHINE_COST_HOUR_US[machine_type]
KeyError: u'custom-32-70400'
Calling stack:
0 swarming_bot.1.zip/api/bot.py:207:post_error()
1 swarming_bot.1.zip/bot_code/bot_main.py:243:_call_hook_safe()
2 swarming_bot.1.zip/bot_code/bot_main.py:298:_get_state()
3 swarming_bot.1.zip/bot_code/bot_main.py:778:_run_bot_inner()
4 swarming_bot.1.zip/bot_code/bot_main.py:667:_run_bot()
5 swarming_bot.1.zip/bot_code/bot_main.py:1220:main()
6 swarming_bot.1.zip/__main__.py:164:CMDstart_bot()
7 swarming_bot.1.zip/__main__.py:247:main()
8 swarming_bot.1.zip/__main__.py:259:<module>()
9 runpy.py:72:_run_code()
10 runpy.py:162:_run_module_as_main()