New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 835309 link

Starred by 2 users

Issue metadata

Status: WontFix
Owner: ----
Closed: Apr 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux
Pri: 1
Type: Bug



Sign in to add a comment

logdog butler not setting env variables on our linux-perf bots

Project Member Reported by eyaich@chromium.org, Apr 20 2018

Issue description

linux-perf is failing on the chromium.perf waterfall due to it not being able to find some logdog environment variables. 

See most recent failing build: https://uberchromegw.corp.google.com/i/chromium.perf/builders/linux-perf/builds/0

and logs from one of the tests: 
https://uberchromegw.corp.google.com/i/chromium.perf/builders/linux-perf/builds/0/steps/load_library_perf_tests%20on%20NVIDIA%20GPU%20on%20Linux/logs/stdio

Two interesting things you can see here are that

1) it doesn't contain the environment variables we need like the linux bots on our fyi waterfall. 
LOGDOG_COORDINATOR_HOST: logs.chromium.org
 LOGDOG_STREAM_PREFIX: bb/chromium.perf.fyi/One_Buildbot_Step_Test_Builder/5088
 LOGDOG_STREAM_PROJECT: chrome
 LOGDOG_STREAM_SERVER_PATH: unix:/b/build/rr/tmppaEcas/butler.sock

2) the stack trace points to this with a bootstrapping error
ERROR:root:Ignore this exception. crbug.com/675666
Traceback (most recent call last):
  File "/b/c/b/linux_perf/src/build/android/pylib/utils/decorators.py", line 32, in wrapper
    return f(*args, **kwargs)
  File "/b/c/b/linux_perf/src/build/android/pylib/utils/logdog_helper.py", line 51, in open_text
    return get_logdog_client().open_text(name)
  File "/b/c/b/linux_perf/src/build/android/pylib/utils/decorators.py", line 16, in wrapper
    memoize_dict[key] = f(*args, **kwargs)
  File "/b/c/b/linux_perf/src/build/android/pylib/utils/logdog_helper.py", line 92, in get_logdog_client
    return bootstrap.ButlerBootstrap.probe().stream_client()
  File "/b/c/b/linux_perf/src/tools/swarming_client/libs/logdog/bootstrap.py", line 48, in probe
    raise NotBootstrappedError('Missing project [%s]' % (cls._ENV_PROJECT,))
NotBootstrappedError: Missing project [LOGDOG_STREAM_PROJECT]

The bug implies this might be a red herring, but our code does fail further down when it tries to utilize the output stream (output_json_file is the variable name).  

Any ideas why the configuration of this bot is different?


 

Comment 1 by no...@chromium.org, Apr 20 2018

it *might* have something to do with the fact that this is the first build (number 0) in the builder. I wonder if there is an assumption that build number is a truish value. Could you check this theory by somehow forcing the next build?

Comment 2 by eyaich@chromium.org, Apr 20 2018

Given that this is running and uploading data to the perf dashboard I would prefer not to kill this build.  The step will show red, but we should see the next build sometime in the next hour or so.  The new perf recipe runs in ~3.5-4 hours.


Comment 3 by d...@chromium.org, Apr 20 2018

nodir@ is probably correct. You can see the bootstrapping error here:

Manifest:
infra/recipes-py git_revision:6eaacf24833ebd2565177157d368da33780fced9
INFO:remote_run:Recipe command line: ['/usr/bin/python', '/b/.remote_run_cipd/recipes.py', '--operational-args-path', '/b/rr/tmpYalMsN/engine_flags.json', '--verbose', 'remote', '--repository', 'https://chromium.googlesource.com/chromium/tools/build.git', '--workdir', '/b/rr/tmpYalMsN/rw', '--revision', 'refs/heads/master', '--', '--operational-args-path', '/b/rr/tmpYalMsN/engine_flags.json', '--verbose', 'run', '--properties-file', '/b/rr/tmpYalMsN/remote_run_properties.json', '--workdir', '/b/rr/tmpYalMsN/w', '--output-result-json', '/b/rr/tmpYalMsN/recipe_result.json', u'chromium']
WARNING:logdog_bootstrap:Missing mastername/buildername/buildnumber properties.
INFO:remote_run:Not using LogDog. Invoking `recipes.py` directly.


Since there isn't anything special about mastername/buildername, it would stand to reason that the build number check is regarding "0" as a missing value, likely using "not".
Status: WontFix (was: Untriaged)
This issue actually did resolve with build #1!

Comment 5 by d...@chromium.org, Apr 23 2018

Nice! FYI filed this bug for follow-up: https://bugs.chromium.org/p/chromium/issues/detail?id=835816

Sign in to add a comment