New issue
Advanced search Search tips

Issue 799995 link

Starred by 2 users
Status: Available
Owner: ----
Cc:
rmis...@chromium.org
mar...@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug

Blocked on:
issue 800030


Sign in to add a comment

Optimize exparchive for very large uploads

Project Member Reported by mar...@chromium.org, Jan 8 2018 
Example:
https://luci-milo.appspot.com/swarming/task/3a971d0b5992ff10/steps/isolate_tests/0/stdout from https://chromium-swarm.appspot.com/task?id=3a971d0b5992ff10&refresh=10:

on GCE n1-standard-16 / Debian 9.2

Hits    : 78958 (133.4Gib)
Misses  :    40 (278.9Mib)

This takes 2h26mins to run. This should be *much* faster, since this is almost all local operations.

Comment 1 by mar...@chromium.org, Jan 8 2018

Cc: rmis...@chromium.org
I forgot to note, that's the equivalent of processing 2Gb/minute or ~37MB/s.

A n1-standard-16 with a 3Tb disk like https://chromium-swarm.appspot.com/bot?id=skia-ct-gce-001 should be able to pump much more than that: https://cloud.google.com/compute/docs/disks/performance gives around 0.12*3000 = 360MB/s but the sheet claims it to be capped at 180MB/s anyway. That's still much higher than 37MB/s so there's room for optimization inside isolate-go. This is what this bug is about.

---

As a mitigation, since I suspect the performance is read bound, it would make sense to use a much smaller SSD or NVMe disk to get better read performance.

Comment 2 by mar...@chromium.org, Jan 8 2018

Blockedon: 800030
Project Member

Comment 3 by sheriffbot@chromium.org, Jan 10

Labels: Hotlist-Recharge-Cold
Status: Untriaged (was: Available)
This issue has been Available for over a year. If it's no longer important or seems unlikely to be fixed, please consider closing it out. If it is important, please re-triage the issue.

Sorry for the inconvenience if the bug really should have been left as Available.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Comment 4 by mar...@chromium.org, Jan 10

Labels: -Hotlist-Recharge-Cold
Status: Available (was: Untriaged)
►

Sign in to add a comment