nvme storage performance |
||||||||||||||||||||||||||||||||||
Issue descriptionNvme performance should be better than emmc, but nothing has shown very significant improvements. Arc++ apps haven't shown significant differences in loading times between emmc and nvme. I measured the time it takes to compile the kernel on eve, using both crouton and a container. Nvme is between 3 and 6% faster than emmc, but that is not as significant as we would expect. Here are sample times: Nvme container: real 6m39.067s user 23m55.171s sys 1m59.797s Nvme crouton: real 6m46.810s user 23m47.136s sys 1m43.087s emmc container: real 7m0.848s user 25m15.338s sys 2m5.452s emmc crouton: real 6m41.608s user 23m40.113s sys 1m46.217s I used blktrace and btt to produce graphs of the io behavior while compiling the kernel, which I have attached. The graphs show a more significant difference in latencies, emmc having 5 times larger latencies, but these differences don't seem to be reflected in the overall times.
,
Oct 17 2017
I have run the cheets_CTS_N.x86.CtsFileSystemTestCases on nvme and emmc, and we do see an improvement with nvme. The test took 2h 5m 27s on nvme compared to 3h 35m 48s on emmc, about 42% faster. Looking at the iostat output (attached) we can see that this test maintains high utilization of the storage device with lower cpu utilization, which is different from the other tests I have tried. This shows that we do see the benefits of nvme when the storage device is heavily used.
,
Oct 17 2017
To summarize the tests I have tried (with associated iostat outputs attached): I started by looking at loading time for android apps, specifically looking at cheets_CandyCrushTest. While we do see utilization of the storage device here, it is not very high utilization and only over a short period of time, so we don’t seem to get the benefit of the nvme device and the loading times are not significantly different. I tried some other arc++ tests as well as having tabs open while running the cheets_CandyCrushTest, and still did not see very significant differences like we would hope (I have attached pdfs of my spreadsheets with some of the times. Note that some of these numbers come from tests run on a fizz, meaning they are comparing nvme and sata devices rather than emmc). I ran fio tests with the 4k_read script on a fizz and graphed the iops as part of this bug: crbug.com/731497 and comparing nvme and sata devices, we do see more iops with nvme. (I have attached two graphs to show this). I then tried to look at the time it takes to compile the kernel, both in crouton and inside a container. As stated above, we don’t see a significant improvement with the nvme device like we would expect, but looking at the iostat we can see that the storage device is not heavily utilized, and we seem to be cpu bound. I ran cts tests in android studio, running in crouton, but none of the tests showed sustained utilization of the storage devices, the highest being around 11% and never for longer than 10 seconds at a time, so I wasn’t able to gain much from this. As I said in the comment above, running cheets_CTS_N.x86.CtsFileSystemTestCases did show an improvement with nvme. We see sustained storage utilization of at or close to 100% with %iowait between 20 and 50. The test takes around 2 hours with nvme compared to around 3.5 hours on emmc. This shows that with stress on the storage device we are seeing the benefits of nvme. It seems that most of these use cases just don’t utilize the storage device heavily enough for us to see the effect of the nvme, or are being constrained elsewhere.
,
Feb 3 2018
As an additional use case for comparing nvme and emmc performance, I have measured the compile time for vlc for android in crouton with android studio, following these instructions: https://wiki.videolan.org/AndroidCompile/#Troubleshooting_2 and measuring the time for ./compile to complete. Overall, the compile times were not significantly different. Looking at iostat while compiling, I saw heavier utilization of the storage device with emmc, which briefly went as high as about 80% and occasionally hit around 50%, but mostly stayed somewhere between 10 and 30%. The highest that the utilization of the nvme reached was between 10 and 20%, spending most of the time below 2%. We don't see sustained high utilization of the storage device in either case, but we do see large periods of time with 95-99% cpu utilization, which suggests that we are cpu bound. Considering the low storage utilization, I tried to increase the storage pressure by compiling the app while simultaneously transferring a large file (25G). With this concurrent scp, the emmc utilization hits 100%, but the nvme utilization doesn't get higher than about 35%. Despite the heavy utilization of the emmc, we do not see significantly higher compile times. Some of the times for nvme are lower with this setup, but considering the variation that we see with the compile times in general, I don't think that this is significant. I also measured the build time in android studio, cleaning and running the project, but I did not see an improvement with nvme here either. I am attaching a spreadsheet with the compile times I measured.
,
Mar 9 2018
,
Apr 4 2018
Is there anything actionable here for the ARC++ team? This seems really more like a platform benchmarking issue, that may or may not have anything we can do?
,
May 2 2018
chatted with Alexis offline - she'll own this for now Also moved out of ARC++ component and into the best related one I could find -- please feel free to reroute if I'm mistaken. Thanks!
,
Aug 3
This bug has an owner, thus, it's been triaged. Changing status to "assigned".
,
Oct 3
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/959cc6d0456f3051050504add70af385b487f2f8 commit 959cc6d0456f3051050504add70af385b487f2f8 Author: Alexis Savery <asavery@chromium.org> Date: Wed Oct 03 20:59:24 2018 fio_util: Fix clat ns percentile units Newer fio uses clat_ns_percentile instead of clat_percentile. The units need to be changed to match. BUG=chromium:769150 TEST=Ran hardware_StorageFio, checked results.json Change-Id: Ifc671c2af716ca2330ed306029b48f9f7332fdfe Reviewed-on: https://chromium-review.googlesource.com/1173476 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Alexis Savery <asavery@chromium.org> Reviewed-by: Gwendal Grignou <gwendal@chromium.org> [modify] https://crrev.com/959cc6d0456f3051050504add70af385b487f2f8/client/bin/fio_util.py |
||||||||||||||||||||||||||||||||||
►
Sign in to add a comment |
||||||||||||||||||||||||||||||||||
Comment 1 by asavery@chromium.org
, Sep 27 2017