run failed because of the kitchen error
INFO:remote_run:Command ['/b/.remote_run_cipd/kitchen', '-log-level', 'info', 'cook', '-mode', 'buildbot', '-output-result-json', '/b/rr/tmp5DN4Pn/kitchen_result.json', '-properties-file', '/b/rr/tmp5DN4Pn/remote_run_properties.json', '-recipe', 'chromium_trybot', '-repository', 'https://chromium.googlesource.com/chromium/tools/build.git', '-cache-dir', '/b/c', '-temp-dir', '/b/rr/tmp5DN4Pn/t', '-checkout-dir', '/b/rr/tmp5DN4Pn/rw', '-workdir', '/b/rr/tmp5DN4Pn/w', '-revision', 'refs/heads/master', '-logdog-only', '-logdog-annotation-url', 'logdog://luci-logdog.appspot.com/chromium/bb/tryserver.chromium.linux/linux_chromium_headless_rel/157747/+/recipes/annotations', '-logdog-tag', 'buildbot.master=tryserver.chromium.linux', '-logdog-tag', 'buildbot.builder=linux_chromium_headless_rel', '-logdog-tag', 'buildbot.buildnumber=157747', '-logdog-tag', u'logdog.viewer_url=https://luci-milo.appspot.com/buildbot/tryserver.chromium.linux/linux_chromium_headless_rel/157747'] finished with exit code 1.
INFO:remote_run:Moving file to cleanup directory '/b/rr/tmp5DN4Pn/k' => '/b/build/slave/remote_run/build.dead/knDCveW/k'
INFO:remote_run:Command ['/usr/bin/python', '../../../scripts/slave/remote_run.py', '--repository', 'https://chromium.googlesource.com/chromium/tools/build.git', '--revision', 'refs/heads/master', '--recipe', 'chromium_trybot', '--build-properties-gz=eNqdVE1v2zAM/SuDzk1kp/6Ifeqw6y4bsFNTGIrMOVpky5Eod1nR/z7ait0WW9thOQQi+Ug+PdJ6YAIR2h4rh8JihY6VcRolmzjNijiafldsr0ULWjlk5S371ZobebCmVb5dG9uwOwJY0ckDKxmjs1e63hv89vUzOQ6Ifcn55Fw/T+M9R3t2YAewTwGtOv+TL1W8PAI1ZQ+74Nix8sN0Hv2jsWOtcEgVXqu1Y1cEkhYEQl3tzyHJE7ZMozzOrotsk65i3ffDHuUxbomaGmTmTkntdX7UfpO7+qaGAbTpqXwztlEShJTGd7iWpn3ZA13oEWTMiyIjDeM8CihVh+i2yIo8yrabLN0myeY6SrMA0CAcVEe4MI0pv0jjIktCGEUz1b+9KEL3mO5ZzdeuDiBqDc5VFnTImYAOsOwFygNvwFqFfE5YWRgU3K8bYxrKM95KGG/FsyRJig2PL8yD0OU/6T3qW4kGOizlacfuHh/nkYLtaJdopG+wnrGdb/dgx4XM8yS/YpL0bYw9U7Y8EShQIesdTgv00voN3KRQFRSqvNWXDXa0wu/ptWQr5zy1CfLNzt6aHyDHXZ7rcGflkmPhO4Xo31Ef0TXgOEm/TOAZrjdOIYnwCrs/x/jXfm78rOJoMakijYvqhauPuIUwGbTaznR0/vRlsmYS/0PAwskDTaP+iPNbs43y9CkwTpTyWoUrekZevjUjalBOTWTIcloMcBnrdI6jPFnJhEL3xh5rNRbj+/D+8AnBLbQGobK+Y4+/AePlt8w=', '--factory-properties-gz=eNo9jEEOwyAMBP/icwV3PoMS1wGrECPbHKKqfy89NLfVzGjfMDavGeU8uECCFztWOuEBSsiDFsKq0nn27Hrt4rfKSkOMXfRaVXUflmL816GIlEYmU5ECSr9NdJFmcZ/cnqHw73Aa5bV49ZCOrRl9vuy5OOM='] finished with exit code 1.
program finished with exit code 1
elapsedTime=732.076540

Seeing lots of cases. The other mention of kitchen error in the log is:

kitchen_error: <
  text: "could not parse recipe result: unknown field \"recipePackage\" in recipe_engine.Result"
  call_stack: "original error: unknown field \"recipePackage\" in recipe_engine.Result"
  call_stack: ""
  call_stack: "goroutine 1:"
  call_stack: "#0 kitchen/cook.go:382 - main.(*cookRun).ensureAndRunRecipe()"
  call_stack: "  reason: \"could not parse recipe result\""
  call_stack: ""
  call_stack: "#1 kitchen/cook.go:619 - main.(*cookRun).run()"
  call_stack: "#2 kitchen/cook.go:519 - main.(*cookRun).Run()"
  call_stack: "#3 github.com/maruel/subcommands/subcommands.go:360 - subcommands.Run()"
  call_stack: "#4 kitchen/kitchen.go:52 - main.main()"
  call_stack: "#5 runtime/proc.go:185 - runtime.main()"
  call_stack: "#6 runtime/asm_amd64.s:2197 - runtime.goexit()"
>

Robbie, any insights?

This seems to be blocking all CLs from being submitted.

win also: https://build.chromium.org/p/chromium.fyi/builders/CrWinClang/builds/20973/steps/steps/logs/stdio

Looks like a bad recipe change (or, put differently, overly old kitchen binaries). We're working on reverting things now.

https://chromium.googlesource.com/infra/luci/recipes-py/+/2bb02540a2b089dae2fa7c9146322fa4caed092f was the CL in question ...

The revert helped, builds are green(ish) again.

permanent fix: https://chromium-review.googlesource.com/c/chromium/tools/build/+/647026

Looks green now:
https://luci-milo.appspot.com/buildbot/tryserver.chromium.linux/linux_chromium_headless_rel/

Will downgrade to P1 and remove troopers.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/1749502eb59cecb4e364c63b922a07d87d6acb59

commit 1749502eb59cecb4e364c63b922a07d87d6acb59
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Sep 01 20:55:02 2017

remote_run: canary latest kitchen

Kitchen wasn't rolled in remote_run since May. Pick changes:
https://chromium.googlesource.com/infra/infra/+log/626f893e5822aa2fbea113c0c2e023f2a601c463..fa75314ad412fbc989aec72993623316c8db9bc7/go/src/infra/tools/kitchen

This will fix  crbug.com/761184  because the roll includes
https://chromium-review.googlesource.com/c/infra/infra/+/503515

kitchen needs a system service account for logdog and events. Reuse logdog's
service account as kitchen's system service account.

ACLs were updated to grant GCE default service account and logdog-publisher to
publish BQ events.
https://console.cloud.google.com/iam-admin/iam/project?project=chrome-infra-events

R=dnj@chromium.org

Bug:  761184 
Change-Id: I50c48fd7186e9363f1eb6852d1ebd9d90256af62
Reviewed-on: https://chromium-review.googlesource.com/647026
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/1749502eb59cecb4e364c63b922a07d87d6acb59/scripts/slave/remote_run.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/30479f453824d5b2a86de9f51c48725d957590b9

commit 30479f453824d5b2a86de9f51c48725d957590b9
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Sep 01 21:21:22 2017

[remote_run] fix system account flag

Fix a typo in the flag name

TBR=dnj@chromium.org
Bug:  761184 
Change-Id: If2c2cea8ab6d26406cd6b43fed7b49bd4588fc48
Reviewed-on: https://chromium-review.googlesource.com/648355
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/30479f453824d5b2a86de9f51c48725d957590b9/scripts/slave/remote_run.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/c9c1865b81113f02fd618259624170f59e2c832e

commit c9c1865b81113f02fd618259624170f59e2c832e
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Sep 01 21:55:09 2017

[kitchen] Use AutoSelect auth method

Use AutoSelect auth method so that the auth library uses GCE method is
ServiceAccountJSONPath is ":gce"

R=dnj@chromium.org
BUG= 761184 

Change-Id: Ic346ecc5006ef1e5b4e637d318b679fcd21a70c1
Reviewed-on: https://chromium-review.googlesource.com/648462
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/c9c1865b81113f02fd618259624170f59e2c832e/go/src/infra/tools/kitchen/cook.go

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/e3f5d1572855a8ab4350d4e8485f8a5c3ea0414b

commit e3f5d1572855a8ab4350d4e8485f8a5c3ea0414b
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Sep 01 22:37:32 2017

[remote_run]: roll kitchen

Pick up https://chromium-review.googlesource.com/c/infra/infra/+/648462

R=vadimsh@chromium.org, dnj@chromium.org
Bug:  761184 
Change-Id: I0dd00c8c816feea4d16e17044d20aa56f5672886
Reviewed-on: https://chromium-review.googlesource.com/648473
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/e3f5d1572855a8ab4350d4e8485f8a5c3ea0414b/scripts/slave/remote_run.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/d248b51a0f1eebff046c0e46319b04fe5b1f08e5

commit d248b51a0f1eebff046c0e46319b04fe5b1f08e5
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Sep 05 19:11:54 2017

[remote_run]: roll prod kitchen to c9c1865

Change prod kitchen git_revision:626f893e5822aa2fbea113c0c2e023f2a601c463 ->
git_revision:c9c1865b81113f02fd618259624170f59e2c832e

Kitchen change log:
https://chromium.googlesource.com/infra/infra/+log/626f893e5822aa2fbea113c0c2e023f2a601c463..c9c1865b81113f02fd618259624170f59e2c832e/go/src/infra/tools/kitchen

R=dnj@chromium.org
Bug:  761184 
Change-Id: I8463e618ad81f2fa31c0eb0dbe39060d408bb5f7
Reviewed-on: https://chromium-review.googlesource.com/651066
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/d248b51a0f1eebff046c0e46319b04fe5b1f08e5/scripts/slave/remote_run.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/66d40103d6b7f844f1ae5b4501ed1f8deff209ba

commit 66d40103d6b7f844f1ae5b4501ed1f8deff209ba
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Sep 05 19:21:07 2017

[remote_run] pass -luci-system-account-json unconditionally

TBR=dnj@chromium.org
Bug:  761184 
Change-Id: I4bc796b0bd827aaf5773c334e0b80562c94885ff
No-Presubmit: true
Reviewed-on: https://chromium-review.googlesource.com/651068
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/66d40103d6b7f844f1ae5b4501ed1f8deff209ba/scripts/slave/remote_run.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/7d90084a213c231f5b104319192b3143c8862e70

commit 7d90084a213c231f5b104319192b3143c8862e70
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Sep 05 19:46:15 2017

[remote_run] fix test

An emergency fix https://chromium-review.googlesource.com/c/chromium/tools/build/+/651068
fix the problem in prod, but left a test broken. Fix the test.

R=dnj@chromium.org
Bug:  761184 
Change-Id: I2398641fbb5a9379eb7e15b6eefcba44e7b37633
NOTREECHECKS=true
Reviewed-on: https://chromium-review.googlesource.com/651266
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>

[modify] https://crrev.com/7d90084a213c231f5b104319192b3143c8862e70/scripts/slave/remote_run.py
[modify] https://crrev.com/7d90084a213c231f5b104319192b3143c8862e70/scripts/slave/unittests/remote_run_test.py