For me locally these benchmarks work fine. I was using windows 10 Pro N machine and ran it on canary today. These benchmarks run a little slow especially cnn it could take 3-4 minutes to finish this story. How long did you wait before killing this process? I think this timeout may be different from the one on bots.

I'm guessing the warnings in my logs have something to do with it? I already did follow [1] and install the appropriate pywin32, anything else I need..?

[1] https://github.com/catapult-project/catapult/blob/master/telemetry/docs/run_benchmarks_locally.md

Explicitly CC mythria@ ^^^

I do get those warnings as well, but the test still finishes execution. I think that warning is also a problem but does not cause the test to fail :). 
This is how my trace looks: 

[ RUN      ] browse:news:nytimes@{'case': 'browse', 'group': 'news'}
(ERROR) 2018-02-06 15:37:13,644 tracing_controller_backend.CollectAgentTraceData:314  Error when deleting c:\users\chromium\appdata\local\temp\tmpc3_ycs, will try again at exit.
Traceback (most recent call last):
  File "D:\mythria\chromium\src\third_party\catapult\telemetry\telemetry\internal\platform\tracing_controller_backend.py", line 310, in CollectAgentTraceData
    os.remove(self._trace_log)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\users\\chromium\\appdata\\local\\temp\\tmpc3_ycs'
(WARNING) 2018-02-06 15:40:19,368 timeline_based_measurement._ComputeTimelineBasedMetrics:311  Processing resulting traces took 156.968 seconds
[       OK ] browse:news:nytimes@{'case': 'browse', 'group': 'news'} (268075 ms)

But do you get:

"""
WARNING:root:Unable to import numpy due to: No module named numpy
WARNING:root:Unable to import numpy due to: No module named numpy
WARNING:root:Failed to import devil.android.forwarder: No module named fcntl
(WARNING) 2018-02-01 16:00:03,395 desktop_browser_finder.FindAllAvailableBrowsers:193  Chrome build location for win_AMD64 not found. Browser will be run without Flash.
(WARNING) 2018-02-01 16:00:03,405 desktop_browser_finder.FindAllAvailableBrowsers:193  Chrome build location for win_AMD64 not found. Browser will be run without Flash.
"""

? The test doesn't even begin executing after that for me, exiting with

"""
v8.browsing_desktop is disabled on the selected browser
Try --also-run-disabled-tests to force the benchmark to run.
"""

Yes, I do get those errors. Since the benchmark is disabled on windows it does not run if we dont pass --also-run-disabled-tests. This is the command I used. 

python tools\perf\run_benchmark v8.browsing_desktop --browser=canary --pageset-repeat=4 --also-run-disabled-tests --story-filter-exclude=flipboard

It should also work without pageset-repeat. I excluded flipboard since it is crashing.


A more detailed trace from my run:
D:\mythria\chromium\src> python tools\perf\run_benchmark v8.browsing_desktop --browser=canary --pageset-repeat=4 --also-run-disabled-tests --story-filter-exclude=flipboard
WARNING:root:Unable to import numpy due to: No module named numpy
WARNING:root:Unable to import numpy due to: No module named numpy
WARNING:root:Failed to import devil.android.forwarder: No module named fcntl
(WARNING) 2018-02-06 15:52:04,757 desktop_browser_finder.FindAllAvailableBrowsers:235  Chrome build location for win_AMD64 not found. Browser will be run without Flash.
(WARNING) 2018-02-06 15:52:04,773 desktop_browser_finder.FindAllAvailableBrowsers:235  Chrome build location for win_AMD64 not found. Browser will be run without Flash.
v8.browsing_desktop is disabled on the selected browser
Running benchmark anyway due to: --also-run-disabled-tests
(WARNING) 2018-02-06 15:52:04,894 desktop_browser_finder.FindAllAvailableBrowsers:235  Chrome build location for win_AMD64 not found. Browser will be run without Flash.
[ RUN      ] browse:news:cnn@{'case': 'browse', 'group': 'news'}
(ERROR) 2018-02-06 15:53:43,144 tracing_controller_backend.CollectAgentTraceData:314  Error when deleting c:\users\chromium\appdata\local\temp\tmpz0xf2s, will try again at exit.
Traceback (most recent call last):
  File "D:\mythria\chromium\src\third_party\catapult\telemetry\telemetry\internal\platform\tracing_controller_backend.py", line 310, in CollectAgentTraceData
    os.remove(self._trace_log)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\users\\chromium\\appdata\\local\\temp\\tmpz0xf2s'
(WARNING) 2018-02-06 15:56:50,944 timeline_based_measurement._ComputeTimelineBasedMetrics:311  Processing resulting traces took 164.214 seconds
[       OK ] browse:news:cnn@{'case': 'browse', 'group': 'news'} (287020 ms)
[ RUN      ] browse:tech:discourse_infinite_scroll@{'case': 'browse', 'group': 'tech'}
(ERROR) 2018-02-06 15:57:57,792 tracing_controller_backend.CollectAgentTraceData:314  Error when deleting c:\users\chromium\appdata\local\temp\tmprac0h7, will try again at exit.
Traceback (most recent call last):
  File "D:\mythria\chromium\src\third_party\catapult\telemetry\telemetry\internal\platform\tracing_controller_backend.py", line 310, in CollectAgentTraceData
    os.remove(self._trace_log)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\users\\chromium\\appdata\\local\\temp\\tmprac0h7'
(WARNING) 2018-02-06 15:59:36,003 timeline_based_measurement._ComputeTimelineBasedMetrics:311  Processing resulting traces took 79.863 seconds
[       OK ] browse:tech:discourse_infinite_scroll@{'case': 'browse', 'group': 'tech'} (164679 ms)