New issue
Advanced search Search tips

Issue 721468 link

Starred by 1 user

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Feature

Blocking:
issue 694887



Sign in to add a comment

recipe engine: emit result JSON even on internal errors

Project Member Reported by no...@chromium.org, May 11 2017

Issue description

if kitchen cannot read recipe engine result file (e.g. it does not exist), it crashes with an error message saying it cannot do that

(env omitted)
----------------------------------- RESULTS ------------------------------------
kitchen_error: <
  text: "could not read recipe result file at \"e:\\\\b\\\\swarm_slave\\\\w\\\\ir\\\\tmp\\\\recipe-result.json\": open e:\\b\\swarm_slave\\w\\ir\\tmp\\recipe-result.json: The system cannot find the file specified."
  call_stack: "original error: open e:\\b\\swarm_slave\\w\\ir\\tmp\\recipe-result.json: The system cannot find the file specified."
  call_stack: ""
  call_stack: "goroutine 1:"
  call_stack: "#0 kitchen/cook.go:360 - main.(*cookRun).ensureAndRunRecipe()"
  call_stack: "  reason: \"could not read recipe result file at \\\"e:\\\\\\\\b\\\\\\\\swarm_slave\\\\\\\\w\\\\\\\\ir\\\\\\\\tmp\\\\\\\\recipe-result.json\\\"\""
  call_stack: "  \"path\" = \"e:\\\\b\\\\swarm_slave\\\\w\\\\ir\\\\tmp\\\\recipe-result.json\""
  call_stack: ""
  call_stack: "#1 kitchen/cook.go:619 - main.(*cookRun).run()"
  call_stack: "#2 kitchen/cook.go:519 - main.(*cookRun).Run()"
  call_stack: "#3 github.com/maruel/subcommands/subcommands.go:360 - subcommands.Run()"
  call_stack: "#4 kitchen/kitchen.go:52 - main.main()"
  call_stack: "#5 runtime/proc.go:185 - runtime.main()"
  call_stack: "#6 runtime/asm_amd64.s:2197 - runtime.goexit()"
>
recipe_exit_code: <
  value: 1
>
annotations: <
  command: <
    command_line: "python"
    command_line: "kitchen-checkout\\recipes\\recipes.py"
    command_line: "--operational-args-path"
    command_line: "e:\\b\\swarm_slave\\w\\ir\\tmp\\rr\\op_args.json"
    command_line: "run"
    command_line: "--properties-file"
    command_line: "e:\\b\\swarm_slave\\w\\ir\\tmp\\rr\\properties.json"
    command_line: "--workdir"
    command_line: "e:\\b\\swarm_slave\\w\\ir\\kitchen-workdir"
    command_line: "--output-result-json"
    command_line: "e:\\b\\swarm_slave\\w\\ir\\tmp\\recipe-result.json"
    command_line: "infra_continuous_luci"
    cwd: "e:\\b\\swarm_slave\\w\\ir"
  >
  status: SUCCESS
  stderr_stream: <
    name: "stderr"
  >
  started: <
    seconds: 1494525160
    nanos: 46038500
  >
>
annotation_url: "logdog://luci-logdog-dev.appspot.com/infra/buildbucket/cr-buildbucket-dev.appspot.com/8979870614528752144/+/annotations"
--------------------------------------------------------------------------------


for someone unfamiliar with kitchen, this is hard to debug. The error is hidden in the recipe engine stderr:

$ logdog cat 'logdog://luci-logdog-dev.appspot.com/infra/buildbucket/cr-buildbucket-dev.appspot.com/8979870614528752144/+/stderr'
Traceback (most recent call last):
  File "e:\b\swarm_slave\w\ir\kitchen-checkout\recipes\.recipe_deps\recipe_engine\recipes.py", line 147, in <module>
    ret = main()
  File "e:\b\swarm_slave\w\ir\kitchen-checkout\recipes\.recipe_deps\recipe_engine\recipes.py", line 133, in main
    context, args.package, overrides=args.project_override)
  File "e:\b\swarm_slave\w\ir\kitchen-checkout\recipes\.recipe_deps\recipe_engine\recipe_engine\package.py", line 443, in create
    context, RootRepoSpec(package_file))
  File "e:\b\swarm_slave\w\ir\kitchen-checkout\recipes\.recipe_deps\recipe_engine\recipe_engine\package.py", line 469, in _create_package
    deps[dep] = self._create_package(context, dep_repo)
  File "e:\b\swarm_slave\w\ir\kitchen-checkout\recipes\.recipe_deps\recipe_engine\recipe_engine\package.py", line 449, in _create_package
    context, repo_spec.spec_pb())
  File "e:\b\swarm_slave\w\ir\kitchen-checkout\recipes\.recipe_deps\recipe_engine\recipe_engine\package.py", line 182, in spec_pb
    return self.current().spec
  File "e:\b\swarm_slave\w\ir\kitchen-checkout\recipes\.recipe_deps\recipe_engine\recipe_engine\package.py", line 201, in current
    return self.backend.commit_metadata(self.revision)
  File "e:\b\swarm_slave\w\ir\kitchen-checkout\recipes\.recipe_deps\recipe_engine\recipe_engine\fetch.py", line 109, in commit_metadata
    cache[revision] = self._commit_metadata_impl(revision)
  File "e:\b\swarm_slave\w\ir\kitchen-checkout\recipes\.recipe_deps\recipe_engine\recipe_engine\fetch.py", line 369, in _commit_metadata_impl
    'diff-tree', '-r', '--no-commit-id', '--name-only', '%s^!' % revision)
  File "e:\b\swarm_slave\w\ir\kitchen-checkout\recipes\.recipe_deps\recipe_engine\recipe_engine\fetch.py", line 245, in _git
    raise GitError('Git "%s" failed: %s' % (subcommand, e.message,))
GitError: Git "diff-tree" failed:
Uncaught exception (GitError): Git "diff-tree" failed:

------------


We can solve problem by retaining recipe engine's stdio in kitchen, or making recipe engine always dump a result file, even on internal errors. In this case, the failure should specify "Uncaught exception (GitError): Git "diff-tree" failed:" or something like that
 
Status: Available (was: Untriaged)
Blocking: 694887

Comment 3 by efoo@chromium.org, May 23 2017

Labels: LUCI-M1-S2 REQBY-LUCI-M2-Dev2 LUCI-M1-Dev1

Comment 4 by efoo@chromium.org, May 23 2017

Owner: iannucci@chromium.org
Status: Assigned (was: Available)

Comment 5 by no...@chromium.org, May 31 2017

it is happening again due to git quota errors when recipe is fetching deps: https://chromium-swarm.appspot.com/task?id=3678ba67966af710&refresh=10&show_raw=1&wide_logs=true

Comment 6 by efoo@chromium.org, Jun 9 2017

Labels: LUCI-M1-S3

Comment 7 by efoo@chromium.org, Jun 20 2017

Labels: LUCI-M2-S4 LUCI-M2-Dev2

Comment 8 by efoo@chromium.org, Nov 8 2017

Labels: -REQBY-LUCI-M2-Dev2 LUCI-M0-Backlog REQBY-LUCI-M5-GASupport

Comment 9 by efoo@chromium.org, Jan 30 2018

Labels: LUCI-Backlog

Comment 10 by efoo@chromium.org, Jan 30 2018

Labels: -LUCI-M0-Backlog -LUCI-M1-Dev1 -LUCI-M2-Dev2 -REQBY-LUCI-M5-GASupport LUCI-Blocker-GASupport LUCI-Dev1 LUCI-Dev2
Owner: iannu...@google.com
Cc: -iannucci@chromium.org iannu...@google.com

Sign in to add a comment