New issue
Advanced search Search tips

Issue 811989 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Mar 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug

Blocking:
issue 732531



Sign in to add a comment

Upload sha1 of cros images alongside the actual images in google storage

Project Member Reported by bpastene@chromium.org, Feb 13 2018

Issue description

So that an image like:
gs:///chromeos-image-archive/amd64-generic-full/R66-10401.0.0-b21238/chromiumos_qemu_image.tar.xz
would also have a small file at:
gs:///chromeos-image-archive/amd64-generic-full/R66-10401.0.0-b21238/chromiumos_qemu_image.tar.xz.sha1
(or something to the same effect).

This would be useful for chromium's use case so we can skip the download if the file is already mirrored in our content-addressed object storage.

 
Cc: vapier@chromium.org
Labels: OS-Chrome
Status: Started (was: Assigned)
https://chromium-review.googlesource.com/c/chromiumos/chromite/+/956883
Project Member

Comment 2 by bugdroid1@chromium.org, Mar 10 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/db8d32eef11fc3b475fe85b1fcdb530fa39069d6

commit db8d32eef11fc3b475fe85b1fcdb530fa39069d6
Author: Achuith Bhandarkar <achuith@chromium.org>
Date: Sat Mar 10 04:05:25 2018

cbuildbot: Generate and upload uploaded.json.

uploaded.json contains the filename, size, sha1, and sha256 of all
uploaded files. This allows chromium infra to cache large cros
binaries with the sha of the binary.

BUG= chromium:811989 
TEST=unittest

Change-Id: I7b0c357a0423df1704ad960980c4b5fe97a278e8
Reviewed-on: https://chromium-review.googlesource.com/956883
Commit-Ready: Achuith Bhandarkar <achuith@chromium.org>
Tested-by: Achuith Bhandarkar <achuith@chromium.org>
Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>

[modify] https://crrev.com/db8d32eef11fc3b475fe85b1fcdb530fa39069d6/cbuildbot/stages/report_stages.py
[modify] https://crrev.com/db8d32eef11fc3b475fe85b1fcdb530fa39069d6/cbuildbot/commands.py
[modify] https://crrev.com/db8d32eef11fc3b475fe85b1fcdb530fa39069d6/cbuildbot/commands_unittest.py

Status: Fixed (was: Started)
https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/amd64-generic-chromium-pfq/R67-10481.0.0-rc2

There's now an uploaded.json file with filenames, size, sha1, and sha256 hash as expected.
Project Member

Comment 4 by bugdroid1@chromium.org, Mar 13 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/6769a99f588dd78ec347e18d2bdbbe74c4a85d4e

commit 6769a99f588dd78ec347e18d2bdbbe74c4a85d4e
Author: Achuith Bhandarkar <achuith@chromium.org>
Date: Tue Mar 13 11:50:13 2018

cbuildbot: Log timing of GenerateUploadJSON.

BUG= chromium:811989 
TEST=unittest

Change-Id: I8a23f59976d9ccff4e9f87617c06a507620074c7
Reviewed-on: https://chromium-review.googlesource.com/959588
Commit-Ready: Achuith Bhandarkar <achuith@chromium.org>
Tested-by: Achuith Bhandarkar <achuith@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/6769a99f588dd78ec347e18d2bdbbe74c4a85d4e/cbuildbot/commands.py

Great, thanks Achuith!

Any idea when it'll start showing up on the full builders on https://build.chromium.org/p/chromiumos? The latest builds don't seem to have it:
https://build.chromium.org/p/chromiumos/builders/amd64-generic-full/builds/21476
Whoops; nvm. I was looking at the "Artifacts" link, but it's in the bucket:
https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/amd64-generic-full/R67-10485.0.0-b2377374
Yup, I haven't added it to the index.html file, but maybe I should.

Comment 8 by vapier@chromium.org, Mar 13 2018

all files that get uploaded should be indexed.  i thought it was already, but i must have missed something in the CL.
Everything in UPLOADED gets indexed. I think we're also missing build-events.json and the UPLOADED file itself. I can add all 3.
Project Member

Comment 11 by bugdroid1@chromium.org, Mar 15 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/a22f70d6bd7ca539ab84b36d27ac49af5fa251f4

commit a22f70d6bd7ca539ab84b36d27ac49af5fa251f4
Author: Achuith Bhandarkar <achuith@chromium.org>
Date: Thu Mar 15 00:23:16 2018

cbuildbot: Add missing files to index.

updated.json, timeline-stages.html, and timeline-slaves.html will also
be added to the index.

BUG= chromium:811989 
TEST=bots

Change-Id: Ie4ca847e5e6c2fd85aeb394c54e412590e47cfe4
Reviewed-on: https://chromium-review.googlesource.com/961444
Commit-Ready: Achuith Bhandarkar <achuith@chromium.org>
Tested-by: Achuith Bhandarkar <achuith@chromium.org>
Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>

[modify] https://crrev.com/a22f70d6bd7ca539ab84b36d27ac49af5fa251f4/cbuildbot/stages/report_stages_unittest.py
[modify] https://crrev.com/a22f70d6bd7ca539ab84b36d27ac49af5fa251f4/cbuildbot/stages/report_stages.py

Project Member

Comment 12 by bugdroid1@chromium.org, Mar 16 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b8e7c2d9a192a0e7c441c6f34284fea6ef68dcd5

commit b8e7c2d9a192a0e7c441c6f34284fea6ef68dcd5
Author: chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Fri Mar 16 22:50:01 2018

Roll src/third_party/chromite/ 3b75c9d82..3ad8f333d (31 commits)

https://chromium.googlesource.com/chromiumos/chromite.git/+log/3b75c9d82ebf..3ad8f333d567

$ git log 3b75c9d82..3ad8f333d --date=short --no-merges --format='%ad %ae %s'
2018-03-16 dgarrett Revert "Reland "pre_cq_launcher: Swarming for chromeos-infra-puppet-pre-cq.""
2018-03-16 dgarrett Reland "pre_cq_launcher: Swarming for chromeos-infra-puppet-pre-cq."
2018-03-14 ayatane autotest-pre-cq: Remove builder and stage [2/2]
2018-03-16 dgarrett Revert "pre_cq_launcher: Swarming for chromeos-infra-puppet-pre-cq."
2018-03-15 dgarrett chromeos_config: Move fuzzer builds into new bucket.
2018-03-16 dgarrett Revert "commands: RunBranchUtilTest -> RunLocalTryjob"
2018-03-13 dgarrett pre_cq_launcher: Swarming for chromeos-infra-puppet-pre-cq.
2018-02-07 dgarrett commands: RunBranchUtilTest -> RunLocalTryjob
2018-03-14 dgarrett cbuildbot_run: Switch more build links to Legoland.
2018-03-13 dgarrett swarming_lib: Remove SWARMING_TASK_ID from cmds.
2018-03-08 dgarrett moblab_vm_unitest: Fix lint issues.
2018-03-14 ihf chromeos_config: add more arcnext experimental coverage.
2018-03-14 ayatane autotest-pre-cq: Remove this [1/2]
2018-03-14 norvez chromeos_config: remove dead code
2018-03-09 dgarrett summarize_build_stats: Add blank line at beginning.
2018-01-09 dgarrett cros tryjob: Remove buildbot URL generation.
2017-09-14 craigb image_test: Remove check that kernel is not ELF.
2018-03-15 ihf Revert "chromeos_config: temporarily mark eve-arcnext-paladin experimental"
2018-03-15 ihf Revert "chromeos_config: temporarily experimental eve-arcnext-mst-android-pfq"
2018-03-13 lhchavez chromeos_config: Add betty-arcnext builder config
2018-03-13 achuith cbuildbot: Add missing files to index.
2018-03-13 akeshet completion_stages: add a has_important_slave metric to master completion
2018-03-13 dgarrett precq-launcher: Start using Legoland build details page.
2018-03-08 dgarrett chromite-pre-cq: Disable CidbIntegrationTest.
2018-03-14 akeshet chromeos_config: temporarily experimental eve-arcnext-mst-android-pfq
2018-03-13 akeshet chromeos_config: temporarily mark eve-arcnext-paladin experimental
2018-03-12 haddowk [chromite] Make guado_moblab important again
2018-03-13 chrome-bot Update config settings by config-updater.
2018-03-12 gmeinke chromium-config: replace cros_config_host_py in chromite
2018-03-12 yunlian Enable ThinLTO on all AMD64 boards.
2018-03-12 achuith cbuildbot: Log timing of GenerateUploadJSON.

Created with:
  roll-dep src/third_party/chromite
BUG=821930, 822517 , 821615 ,None,821618,821227,None,821664,821930,None,815377,747385,461595,821664,821664,811989,819419,821618,820305,821664,821664,819017,813442,707803,811989


The AutoRoll server is located here: https://chromite-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=chrome-os-gardeners@chromium.org

Change-Id: Ib6aaddf338307e994865a092ecb322a432148692
Reviewed-on: https://chromium-review.googlesource.com/967273
Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#543855}
[modify] https://crrev.com/b8e7c2d9a192a0e7c441c6f34284fea6ef68dcd5/DEPS

Project Member

Comment 13 by bugdroid1@chromium.org, Sep 20

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/589b105f17aef2ef64a5812daf3e1fba9b4276a4

commit 589b105f17aef2ef64a5812daf3e1fba9b4276a4
Author: Achuith Bhandarkar <achuith@chromium.org>
Date: Thu Sep 20 18:39:45 2018

metadata_lib: Pretty print metadata.json

Make partial-metadata.json and metadata.json human readable.

BUG= chromium:811989 
TEST=None

Change-Id: Ic5b35b5f7ba4eb88a747f9fcbcddca3d74499b37
Reviewed-on: https://chromium-review.googlesource.com/959607
Commit-Ready: Achuith Bhandarkar <achuith@chromium.org>
Tested-by: Achuith Bhandarkar <achuith@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/589b105f17aef2ef64a5812daf3e1fba9b4276a4/lib/metadata_lib.py

Sign in to add a comment