Looks like the instructions in https://chromium.googlesource.com/chromium/src/+/master/docs/speed/perf_trybots.md only show how to start Telemetry benchmarks.

logs from a pinpoint run https://pinpoint-dot-chromeperf.appspot.com/job/15fe670c140000:

Running ['c:\\infra-system\\bin\\python.exe', '../../tools/perf/run_benchmark', 'performance_browser_tests', '--pageset-repeat', '1', '--browser', 'release_x64', '-v', '--upload-results', '--results-label', 'chromium@bb012ab', '--output-format=histograms', '--output-dir', 'c:\\b\\s\\w\\itztncu4\\tmpnth5vztelemetry', '--output-format=json-test-results'] in None (env: {'TMP': 'c:\\b\\s\\w\\itztncu4', 'LC_NUMERIC': 'English_United States.UTF-8', 'COMPUTERNAME': 'BUILD283-A7', 'USERDOMAIN': 'LABS', 'LC_CTYPE': 'English_United States.UTF-8', 'PSMODULEPATH': 'C:\\Program Files\\WindowsPowerShell\\Modules;C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\Modules', 'COMMONPROGRAMFILES': 'C:\\Program Files\\Common Files', 'PROCESSOR_IDENTIFIER': 'Intel64 Family 6 Model 158 Stepping 9, GenuineIntel', 'IPG_DIR': 'C:\\Program Files\\Intel\\Power Gadget 3.5', 'PROCESSOR_REVISION': '9e09', 'HOME': 'c:\\Users\\chrome-bot', 'BOTO_CONFIG': 'c:\\Users\\chrome-bot\\.boto', 'SWARMING_BOT_ID': 'build283-a7', 'PROGRAMFILES(X86)': 'C:\\Program Files (x86)', 'LANG': 'en_US.UTF-8', 'CIPD_PROTOCOL': 'v2', 'SWARMING_SERVER': 'https://chrome-swarming.appspot.com', 'TEMP': 'c:\\b\\s\\w\\itztncu4', 'SWARMING_TASK_ID': '416b4296cd222e11', 'LC_MONETARY': 'English_United States.UTF-8', 'CHROME_DEVEL_SANDBOX': '/opt/chromium/chrome_sandbox', 'COMMONPROGRAMFILES(X86)': 'C:\\Program Files (x86)\\Common Files', 'PROCESSOR_ARCHITECTURE': 'AMD64', 'SWARMING_HEADLESS': '1', 'ALLUSERSPROFILE': 'C:\\ProgramData', 'LOCALAPPDATA': 'C:\\Users\\chrome-bot\\AppData\\Local', 'IPG_VER': '3.0', 'HOMEPATH': '\\Users\\chrome-bot', 'LUCI_CONTEXT': 'c:\\b\\s\\w\\itztncu4\\luci_ctx.pinxfw.json', 'PROGRAMW6432': 'C:\\Program Files', 'USERNAME': 'chrome-bot', 'LC_ALL': 'English_United States.UTF-8', 'LOGONSERVER': '\\\\AD1-B', 'SYSTEMROOT': 'C:\\Windows', 'COMSPEC': 'C:\\Windows\\system32\\cmd.exe', 'PROGRAMDATA': 'C:\\ProgramData', 'USERDOMAIN_ROAMINGPROFILE': 'LABS', 'ONEDRIVE': 'C:\\Users\\chrome-bot\\OneDrive', 'PATH': 'C:\\Windows\\system32;C:\\Windows;C:\\Windows\\System32\\Wbem;C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\;c:\\Tools;C:\\CMake\\bin;C:\\Program Files\\Puppet Labs\\Puppet\\bin;C:\\Users\\chrome-bot\\AppData\\Local\\Microsoft\\WindowsApps', 'PROMPT': '$P$G', 'USERDNSDOMAIN': 'LABS.CHROMIUM.ORG', 'NO_GCE_CHECK': 'False', 'CHROME_HEADLESS': '1', 'PATHEXT': '.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC', 'TMPDIR': 'c:\\b\\s\\w\\itztncu4', 'SESSIONNAME': 'Console', 'WINDIR': 'C:\\Windows', 'APPDATA': 'C:\\Users\\chrome-bot\\AppData\\Roaming', 'HOMEDRIVE': 'C:', 'PROGRAMFILES': 'C:\\Program Files', 'SYSTEMDRIVE': 'C:', 'PUBLIC': 'C:\\Users\\Public', 'NUMBER_OF_PROCESSORS': '8', 'PROCESSOR_LEVEL': '6', 'LC_TIME': 'English_United States.UTF-8', 'COMMONPROGRAMW6432': 'C:\\Program Files\\Common Files', 'OS': 'Windows_NT', 'LC_COLLATE': 'English_United States.UTF-8', 'USERPROFILE': 'C:\\Users\\chrome-bot'})
WARNING:root:Unable to import numpy due to: No module named numpy

WARNING:root:Unable to import numpy due to: No module named numpy

WARNING:root:Failed to import devil.android.forwarder: No module named fcntl

No benchmark named "performance_browser_tests".


This blocks anyone who is trying to fix a bug in a benchmark that they can't reproduce locally. Yuri is blocked by this.

Is there so other way to try a change before it is landed?

Assigning to simonhatch@ per Hangouts discussion.

I got a very similar error when running a pinpoint job for base_perftests:

https://pinpoint-dot-chromeperf.appspot.com/job/12b0dc58540000

Traceback (most recent call last):
  File "/base/data/home/apps/s~chromeperf/pinpoint:dev-simonhatch-8e3fbf03.415445858561736965/dashboard/pinpoint/models/job.py", line 369, in Run
    work_left = self.state.ScheduleWork()
  File "/base/data/home/apps/s~chromeperf/pinpoint:dev-simonhatch-8e3fbf03.415445858561736965/dashboard/pinpoint/models/job_state.py", line 139, in ScheduleWork
    self._RaiseErrorIfAllAttemptsFailed()
  File "/base/data/home/apps/s~chromeperf/pinpoint:dev-simonhatch-8e3fbf03.415445858561736965/dashboard/pinpoint/models/job_state.py", line 159, in _RaiseErrorIfAllAttemptsFailed
    'was:\n%s' % (exception_count, attempt_count, exception))
Exception: All of the runs failed. The most common error (1/20 runs) was:
IOError: [Errno 2] No such file or directory: '/b/s/w/itIVOHDH/tmpor4neltelemetry/histograms.json'

These non-telemetry tests don't produce histogram results AFAIK, so failing is somewhat expected. I can look into how much work it would be to add support.

So there's a few spots that need to be updated and this looks like non-telemetry tests should work.

We need to provide the right isolate target when starting a job. The dashboard does this, but pinpoint's ui doesn't. That seems like enough to get the job running, although results for try jobs won't show up. Here's a version where I got base_perftests running: https://pinpoint-dot-chromeperf.appspot.com/job/14cc2ee2540000

We could probably read out the results and convert them to histograms when generating the results2 files afterwards, which would give you the results2 table and comparison features.

I think this is the same as issue 904423. Should be merged?