logdog butler not setting env variables on our linux-perf bots |
||
Issue descriptionlinux-perf is failing on the chromium.perf waterfall due to it not being able to find some logdog environment variables. See most recent failing build: https://uberchromegw.corp.google.com/i/chromium.perf/builders/linux-perf/builds/0 and logs from one of the tests: https://uberchromegw.corp.google.com/i/chromium.perf/builders/linux-perf/builds/0/steps/load_library_perf_tests%20on%20NVIDIA%20GPU%20on%20Linux/logs/stdio Two interesting things you can see here are that 1) it doesn't contain the environment variables we need like the linux bots on our fyi waterfall. LOGDOG_COORDINATOR_HOST: logs.chromium.org LOGDOG_STREAM_PREFIX: bb/chromium.perf.fyi/One_Buildbot_Step_Test_Builder/5088 LOGDOG_STREAM_PROJECT: chrome LOGDOG_STREAM_SERVER_PATH: unix:/b/build/rr/tmppaEcas/butler.sock 2) the stack trace points to this with a bootstrapping error ERROR:root:Ignore this exception. crbug.com/675666 Traceback (most recent call last): File "/b/c/b/linux_perf/src/build/android/pylib/utils/decorators.py", line 32, in wrapper return f(*args, **kwargs) File "/b/c/b/linux_perf/src/build/android/pylib/utils/logdog_helper.py", line 51, in open_text return get_logdog_client().open_text(name) File "/b/c/b/linux_perf/src/build/android/pylib/utils/decorators.py", line 16, in wrapper memoize_dict[key] = f(*args, **kwargs) File "/b/c/b/linux_perf/src/build/android/pylib/utils/logdog_helper.py", line 92, in get_logdog_client return bootstrap.ButlerBootstrap.probe().stream_client() File "/b/c/b/linux_perf/src/tools/swarming_client/libs/logdog/bootstrap.py", line 48, in probe raise NotBootstrappedError('Missing project [%s]' % (cls._ENV_PROJECT,)) NotBootstrappedError: Missing project [LOGDOG_STREAM_PROJECT] The bug implies this might be a red herring, but our code does fail further down when it tries to utilize the output stream (output_json_file is the variable name). Any ideas why the configuration of this bot is different?
,
Apr 20 2018
Given that this is running and uploading data to the perf dashboard I would prefer not to kill this build. The step will show red, but we should see the next build sometime in the next hour or so. The new perf recipe runs in ~3.5-4 hours.
,
Apr 20 2018
nodir@ is probably correct. You can see the bootstrapping error here: Manifest: infra/recipes-py git_revision:6eaacf24833ebd2565177157d368da33780fced9 INFO:remote_run:Recipe command line: ['/usr/bin/python', '/b/.remote_run_cipd/recipes.py', '--operational-args-path', '/b/rr/tmpYalMsN/engine_flags.json', '--verbose', 'remote', '--repository', 'https://chromium.googlesource.com/chromium/tools/build.git', '--workdir', '/b/rr/tmpYalMsN/rw', '--revision', 'refs/heads/master', '--', '--operational-args-path', '/b/rr/tmpYalMsN/engine_flags.json', '--verbose', 'run', '--properties-file', '/b/rr/tmpYalMsN/remote_run_properties.json', '--workdir', '/b/rr/tmpYalMsN/w', '--output-result-json', '/b/rr/tmpYalMsN/recipe_result.json', u'chromium'] WARNING:logdog_bootstrap:Missing mastername/buildername/buildnumber properties. INFO:remote_run:Not using LogDog. Invoking `recipes.py` directly. Since there isn't anything special about mastername/buildername, it would stand to reason that the build number check is regarding "0" as a missing value, likely using "not".
,
Apr 23 2018
This issue actually did resolve with build #1!
,
Apr 23 2018
Nice! FYI filed this bug for follow-up: https://bugs.chromium.org/p/chromium/issues/detail?id=835816 |
||
►
Sign in to add a comment |
||
Comment 1 by no...@chromium.org
, Apr 20 2018