New issue
Advanced search Search tips

Issue 807882 link

Starred by 1 user

Issue metadata

Status: Verified
Owner:
Closed: Feb 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Fuchsia
Pri: 2
Type: Bug-Regression



Sign in to add a comment

DiskCacheBackendTests failing on Fuchsia/x64 FYI

Project Member Reported by w...@chromium.org, Feb 1 2018

Issue description

This started in https://ci.chromium.org/buildbot/chromium.fyi/Fuchsia/13462, which included https://chromium-review.googlesource.com/c/chromium/src/+/779742.

Could the failing tests be getting run only after the FD-limit test has run, perhaps, leading to them failing due to persistent process-global state?
 

Comment 1 by w...@chromium.org, Feb 1 2018

Owner: w...@chromium.org
Status: Assigned (was: Untriaged)
Thanks for the report. One thing that may be useful to know: what's the (whatever the equivalent is of) FD limit on Fuchsia, anyway?

One of the failures this is listing is:
DiskCacheBackendTest.Load
which is just a test change, it's not exercising the backend that the bulk of the CL touched (while .SimpleCacheLoad runs through that same test with the affected backend).
The test change is merely this:
https://chromium-review.googlesource.com/c/chromium/src/+/779742/22/net/disk_cache/backend_unittest.cc#b1044

Beyond that, the only output I see is that the tests timed out, which is pretty weird. Maybe getting where it's stuck would be insightful (for all threads...)


Comment 3 by w...@chromium.org, Feb 1 2018

Cc: w...@chromium.org
Owner: morlovich@chromium.org
Re #2: The current limit is intentionally low, at 256 FDs, I believe; I see the test you mention used to be disabled under Mac because of it having that same limit.

From your CL landing we see a 5x slow-down in these tests - I see a change to DiskCacheBackendTest::BackendLoad() which increases the number of entries from 100 to 512 (see 
https://chromium-review.googlesource.com/c/chromium/src/+/779742/22/net/disk_cache/backend_unittest.cc).

Does the number of entries in the cache need to be 512, or could we reduce it back to 100, perhaps just under Fuchsia, until we have the virtualization performance issues resolved?


I'd missed that the tests are all just timing-out; timings for this run were:
[00785.451] 03912.03959> [       OK ] DiskCacheBackendTest.ShaderCacheLoad (51150 ms)
[00542.435] 03907.03950> [13466/21684] DiskCacheBackendTest.NewEvictionLoad - can't find it listed; only the TIMED OUT summary.
[00415.418] 03912.03959> [       OK ] DiskCacheBackendTest.SimpleCacheAppCacheLoad (56111 ms)
[00542.434] 03907.03950> [13465/21684] DiskCacheBackendTest.Load - not listed
[00415.418] 03912.03959> [       OK ] DiskCacheBackendTest.SimpleCacheLoad (48840 ms)
and the new test:
[00415.428] 03912.03959> [       OK ] DiskCacheBackendTest.SimpleFdLimit (76857 ms)

Looking at the preceding net_unittests run (https://ci.chromium.org/buildbot/chromium.fyi/Fuchsia/13461) on that bot:
[00542.436] 03907.03950> [13469/21684] DiskCacheBackendTest.ShaderCacheLoad (12805 ms)
[00542.435] 03907.03950> [13466/21684] DiskCacheBackendTest.NewEvictionLoad (17632 ms)
[00248.236] 03907.03950> [5008/21684] DiskCacheBackendTest.SimpleCacheAppCacheLoad (10627 ms)
[00542.434] 03907.03950> [13465/21684] DiskCacheBackendTest.Load (11867 ms)
[00248.236] 03907.03950> [5007/21684] DiskCacheBackendTest.SimpleCacheLoad (10238 ms)

The tests already suffer substantial slow-downs due to virtualization, e.g. ShaderCacheLoad takes:
- Linux (bare metal): 20ms
- Fuchsia (QEMU+KVM, under Linux on bare metal): ~900ms
- Fuchsia (QEMU+KVM on Linux+GCE): ~12s (before) and ~50s (after)
which is a general issue that we're working to improve.
Ouch at those virtualization numbers, but they certainly make the timeouts make sense. As for changing the limit:
The test is mostly fine with anything > 64 (which is what the fixture tells it to try to keep the usage to) with respect to having the code exercised; having it try to use > 256 on Mac is good, though, since that demonstrates it actually limiting FDs rather than just setting stats claiming it did, so an #ifdef may be the best choice.

Would also likely need to change this:
https://chromium-review.googlesource.com/c/chromium/src/+/779742/22/net/disk_cache/backend_unittest.cc#4222

(Probably should have used more constants there...)

OS having an FD limit of 256 might be a problem for Chrome proper, though --- we lift it to 8192 on Mac on startup, though I guess this may be a little too early to worry about that.

Comment 5 by w...@chromium.org, Feb 1 2018

Owner: kmarshall@chromium.org
kmarshall, can you adjust the # of disk-cache entries in these tests back to 100 under Fuchsia, and 512 for other platforms, as per comments #3 and #4 - thanks!

Comment 6 by w...@chromium.org, Feb 1 2018

Status: Started (was: Assigned)
Project Member

Comment 7 by bugdroid1@chromium.org, Feb 3 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/bc1a77b0f711f72072c57a7ef3730308f46584ee

commit bc1a77b0f711f72072c57a7ef3730308f46584ee
Author: Kevin Marshall <kmarshall@chromium.org>
Date: Sat Feb 03 01:28:12 2018

Specify lower limits on Fuchsia for load tests on DiskCacheBackendTest.

Fuchsia tests run under two layers of virtualization which carries
a significant performance penalty. The DiskCacheBackendTest load tests
specified values which were too computationally intensive, causing the
tests to time out.


Bug:  807882 
Change-Id: I1f44436a9f0967b072aa9da160b5fc15bc7f730f
Reviewed-on: https://chromium-review.googlesource.com/898006
Commit-Queue: Kevin Marshall <kmarshall@chromium.org>
Reviewed-by: Maks Orlovich <morlovich@chromium.org>
Cr-Commit-Position: refs/heads/master@{#534233}
[modify] https://crrev.com/bc1a77b0f711f72072c57a7ef3730308f46584ee/net/disk_cache/backend_unittest.cc

Status: Fixed (was: Started)
Status: Verified (was: Fixed)
Fuchsia FYI bot is now green.

Sign in to add a comment