Why are you interested in running this benchmark?

So as storage.indexeddb_endure:

[ RUN      ] indexeddb-endure-testCreateAndDeleteDatabases
/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/internal/browser/browser.py:144: UserWarning: Function GetMemoryStats is deprecated. It will no lon
ger be supported on November 04, 2017. Please remove it or switch to an alternative before that time. Clients should use tracing and memory-infra in new Teleme
try benchmarks. See for context:  https://crbug.com/632021 

  'Browser': dict(pid_stats_function(browser_pid), **{'ProcessCount': 1}),
/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/internal/browser/browser.py:153: UserWarning: Function GetMemoryStats is deprecated. It will no lon
ger be supported on November 04, 2017. Please remove it or switch to an alternative before that time. Clients should use tracing and memory-infra in new Teleme
try benchmarks. See for context:  https://crbug.com/632021 

  child_stats = pid_stats_function(child_pid)
WARNING:root:Device not on battery power during power monitoring. Results may be incorrect.
WARNING:root:Exception was raised in the with statement block, the end of interaction record is not marked.
Traceback (most recent call last):
  File "/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 88, in _RunStoryAndProcessErrorIfNeeded
    state.RunStory(results)
  File "/ssd2/chromium/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 299, in RunStory
    self._current_page.Run(self)
  File "/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/page/__init__.py", line 112, in Run
    self.RunPageInteractions(action_runner)
  File "/ssd2/chromium/src/tools/perf/page_sets/indexeddb_endure_page.py", line 21, in RunPageInteractions
    action_runner.ExecuteJavaScript('window.test();')
  File "/ssd2/chromium/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py", line 214, in ExecuteJavaScript
    return self._tab.ExecuteJavaScript(*args, **kwargs)
  File "/ssd2/chromium/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 144, in ExecuteJavaScript
    return self._inspector_backend.ExecuteJavaScript(*args, **kwargs)
  File "/ssd2/chromium/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 37, in inner
    return func(inspector_backend, *args, **kwargs)
  File "/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 216, in ExecuteJavaScript
    self._runtime.Execute(statement, context_id, timeout)
  File "/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_runtime.py", line 21, in Execute
    self.Evaluate(expr + '; 0;', context_id, timeout)
  File "/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_runtime.py", line 54, in Evaluate
    description=details.get('exception', {}).get('description'))
EvaluateException: UncaughtTypeError:
TypeError: window.test is not a function
    at <anonymous>:1:8

[  FAILED  ] indexeddb-endure-testCreateAndDeleteDatabases (18534 ms)

Hi Ned, it seems that these two are the benchmarks that exercise Linux kernel most.

BTW, while they are running, the browser shows 'This site cannot be reached (connection refused)' on 127.0.0.1:[some port]/perf_test.html

#3: how do you measure that they exercise Linux kernel most? That benchmark is meant to test the blob API of the web platform, so it may not be the general coverage for Linux kernel that you need to care about.

By perf[1]. We are trying some optimization on kernel. According to the profiles collected on all telemetry benchmarks, these two spent most time in kernel, among other tests. I think this is a result of IO operations.

[1] https://perf.wiki.kernel.org/index.php/Main_Page

I'm a little concerned about this type of usage of telemetry benchmarks. The benchmark here was intended to specifically measure blob_storage, may not work quite well for that purpose, and is likely to be turned down soon. But now you have a completely different use case for it, that we had no idea we were supporting. Wouldn't it be better to write a benchmark directly to exercise linux kernel?

We are trying to see what this kernel improvement can bring us and those benchmarks looks interesting. In fact, we try not to build a benchmark around the change that we made; It's easy to build a benchmark that exaggerate the benefits but we would have to justify that it is representative.

On the other hand, shouldn't kernel be considered a part of blob_storage's performance?

blob_storage benchmark is not representative at all, so it may be a benchmark around the change you made coincidentally. 

Alright, if you say so.

Ned, why do you say that this benchmark is not representative for how the kernel is used? What benchmarks are representative of how the kernel is exercised on user devices?

For reference, go/cwp has real life data collected from user devices. Here are the top kernel functions by CPU usage for peppy boards, when the devices are busy.
https://uma.googleplex.com/p/chrome/chromeos_wide_profiling/?boards_regexp=%5B%22peppy%22%5D&versions_regexp=%5B%5D&trigger_select=%5B%5D&process_container_select=%5B%5D&samples_min_bucket=1750&dso_regexp=kernel.kallsyms&date_select=2017-03-17&range_select=3&release_select=R56&channel_select=stable&kernel_select=Any&sampling_select=1&fact_select=cycles.flat&arch_select=x86_64

The aes_encrypt function is at the top. Its callgraph (https://graphviz.corp.google.com/image?graph_id=b72d56b8f8e623ff4942be33cb720ce3#src=gwp) shows that it's primarily called from ecryptfs_encrypt_extent. We are definitely no experts in CrOS or Linux kernels, but is it possible that that function is handling some filesystem operation?

We want to test the impact of applying AutoFDO to the CrOS kernel, starting with the peppy board because we have more data for it.
Unfortunately, we couldn't find many Telemetry benchmarks that reflect actual user behavior as captured by CWP data. blob_storage was one benchmark that showed some overlap with CWP. It is unfortunate that this benchmark is not supported anymore.

#11: how do you define representative? For chrome benchmark, we define representativeness as how close the test simulation is to the average user browsing on the web. 

The benchmark that's representative for Chrome browser is system_health.common_desktop, though I am not a kernel expert so I can't tell whether this is representative of how the kernel is exercised on user devices.

blob_storage benchmark is not representative in our definition because it's basically a micro benchmark that stresses test blob API at different configuration. See https://cs.chromium.org/chromium/src/tools/perf/page_sets/blob_workshop.py?rcl=53e2a754725e7bdba370806c9fd30a2893439a2d&l=77

I would define representative if it resembles CWP data for CPU usage by function, since that data is aggregated from all the users that opted in to send data to Google. Moreover, we can filter the data by how busy the devices were at data collection time. It's probably not worth optimizing for the idle loop.

CWP data is more focused on CPU usage. It doesn't capture time when threads are blocked, so it's not useful for latency studies, but we are working on that.

Our data shows that a good amount of the time spent in the kernel is to handle encryption/decryption and (de)compression, so probably moving data between disk and RAM. And the average device spends about 25-28% of its CPU time in the kernel.

Do you have some other metrics that capture how the average user browses the web?

According to https://graphviz.corp.google.com/image?graph_id=66dd76808f372b6c552d17fc826f087f#src=gwp, only 10% of aes_encrypt call is due to Chrome browser? 

It also worths noting that Telemetry on Chrome OS may  be using "--ignore-certificate-error" flag to run HTTPS page recorded by WPR, so the coverage of encryption code on Chrome benchmark may not be very realistic at the moment. 

We also are not able to play encrypted video in our benchmark.