any updates?

I'm still looking into this so I have no definitive answers yet, but I'm using btrace and trying to figure out the differences so I should have more information soon. I have run page cycle tests using both ecryptfs: https://ubercautotest.corp.google.com/afe/#tab_id=view_job&object_id=103047792 and ext4: https://ubercautotest.corp.google.com/afe/#tab_id=view_job&object_id=103201051 but ext4 does not appear to be consistently worse than ecryptfs so I'm not sure how much additional information that reveals.

Running more tests and looking at the ftrace has provided a little more information about what is happening here. The original graphs correspond to the fio test using the surfing script. Running the test with different scripts shows slightly varied behavior. The read slat for ext4 is consistently higher than it is for ecryptfs, but the large difference in the original graph is only seen with the surfing script, the other scripts show much smaller differences. The write slat, however, is consistently higher for ecryptfs and the difference here is large across the different scripts. For read clat, ext4 has a higher clat at the highest percentiles, but surfing is the only script that shows ext4 with a higher clat across most percentiles. Other scripts show similar values for both or show ecryptfs with a higher clat until after the 99th percentile.  Write clat graphs show similar things, with ext4 only having significantly higher values at the highest percentiles. The exception to this is with scripts that involve only writes. When the script has writes only, the clat for writes is significantly higher for ecryptfs across most of the percentiles, including the highest. Since the slat for writes is consistently lower for ext4 and the clat for writes is consistently lower for ext4 when the scripts have only writes, it seems that ext4 has worse performance when reads are involved. Also, the surfing script seems to produce the worst performance. Even though the other scripts do show some increased latency for ext4, the differences do not seem as alarming. 
The tracing of ext4 and ecryptfs while running fio shows different patterns for the two, with ext4 having request completions that correspond to the previous request issued and ecryptfs having more complicated behavior, seeming to queue more requests. It looks like ext4 generally gets, issues, and completes one request at a time whereas ecryptfs gets multiple requests and queues them.

That being said, running the page cycle test showed no significant difference between the two, so I don't know how concerning the differences seen in these graphs should be.

The graphs are created with the output of the platform_CryptohomeFio autotest. I edited the fio_util.py file to include the data points of interest in the results-chart.json file and wrote a script to take the json results for both ext4 and ecryptfs and produce the desired graphs. I have attached multiple graphs of slat and clat (showing the results from different scripts), my script to produce the graphs, the edited fio_util.py, as well as the trace from running the test with the surfing script for both ext4 and ecryptfs.

Deleted: performance_graphs.zip 1.1 MB

performance_graphs.zip 1.1 MB Download