New issue
Advanced search Search tips

Issue 821559 link

Starred by 0 users
Status: Assigned
Owner:
achuith@chromium.org
Cc:
bpastene@chromium.org
dgarr...@chromium.org
vapier@chromium.org
chromeos-continuous-integration-fire-hose@google.com
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 3
Type: Feature



Sign in to add a comment

Generate upload.json in UploadArchivedFile

Project Member Reported by achuith@chromium.org, Mar 13 2018 
We currently create uploaded.json in report_stages, after we've uploaded all archived files:
https://cs.corp.google.com/chromeos_public/chromite/cbuildbot/stages/report_stages.py?l=685 

GenerateUploadJSON calculates the SHA1 and SHA256 hashes for a number of large files in serial.

We could calculate these SHAs in UploadArchivedFile, which uploads these large files in parallel:
https://cs.corp.google.com/chromeos_public/chromite/cbuildbot/commands.py?l=1876

Comment 1 by achuith@chromium.org, Mar 13 2018 

From:
https://chromium-review.googlesource.com/c/chromiumos/chromite/+/956883#message-91b125d54031a5225443d0883afca604f70f38bb

I downloaded the artifacts from this build:
https://uberchromegw.corp.google.com/i/chromeos/builders/amd64-generic-chromium-pfq/builds/11729

There are 22 large files, and the SHA computations took ~51 sec on my workstation. 

From this build:
https://uberchromegw.corp.google.com/i/chromeos/builders/amd64-generic-chromium-pfq/builds/11734

We can see that the generation of uploaded.json took ~48sec:
12:27:41: INFO: GenerateUploadJSON completed in 0:00:47.631189.

So I think this is fine as a P3.

Comment 2 by vapier@chromium.org, Mar 14 2018 

ok, ~48sec on ~7.5GiB should be fine for now

Comment 3 by vapier@chromium.org, Mar 14 2018 

thanks for checking btw

Comment 4 by akes...@chromium.org, Mar 30 2018

Components: Infra>Client>ChromeOS>CI

Comment 5 by akes...@chromium.org, Mar 30 2018

Components: -Infra>Client>ChromeOS

Comment 6 by jclinton@chromium.org, Jul 13

Labels: -Type-Bug Hotlist-GoodFirstBug Type-Feature
This is a great starter bug and has some performance improvements to boot.

Comment 7 by benhenry@chromium.org, Aug 2

Status: Assigned (was: Available)
►

Sign in to add a comment