dmurph@: Can you please get some traces and try to figure out where the bottleneck is?

I'm surprised by the slowdown. I assumed that our bottleneck is the IPC overhead, in which case operation types wouldn't matter too much.

I got the results below on Canary (M73) with macOS 10.14. Reading after writing is quite a bit faster than reading while writing.

Read while write histogram
{
  "min": 55,
  "max": 1996,
  "sum": 11155,
  "variance": 167851.48726467334,
  "mean": 259.4186046511628,
  "stddev": 409.6968235960261,
  "count": 43,
  "median": 83,
  "p75": 168,
  "p95": 1257.3999999999996,
  "p99": 1996,
  "p999": 1996
}

Read after write histogram
{
  "min": 24,
  "max": 230,
  "sum": 17253,
  "variance": 145.95053887605846,
  "mean": 39.84526558891455,
  "stddev": 12.080999084349706,
  "count": 433,
  "median": 39,
  "p75": 41,
  "p95": 52,
  "p99": 85.27999999999975,
  "p999": 230
}

The test does too much stuff to be captured in tracing (fills up the tracing buffer right away).

I might have to fork and modify the test a bit to reduce the amount of stuff it does to get any data.

wild guess - the reading-after-writing (is this in a separate transaction? Same transaction? Same data that was writen? lots of questions here) takes advantage of cursor prefetch, whereas the other does not.

I was still confused by the benchmark, so I explored it and typed up a summary:
https://docs.google.com/document/d/1vdrx0gs4TlDwrPYx1LdxdLMjTwmPPAH59p36zi8he50/edit?usp=sharing

Still need to grab traces. But now I understand more about how it's working, and how to do less operations (and end them) to make tracing easier.