New issue
Advanced search Search tips

Issue 763514 link

Starred by 0 users

Issue metadata

Status: Available
Owner: ----
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 3
Type: Feature



Sign in to add a comment

add a builder stage to sanity check all source URIs

Project Member Reported by vapier@chromium.org, Sep 8 2017

Issue description

sometimes we run into issues where sources aren't available at the SRC_URI ... either they haven't been posted at all, or a typo was added, or the files changed on the GS mirrors (while this shouldn't happen, it has in the past).

the trouble with the builders is that they often don't detect these issues up front.  if the bot happened to fetch it in a previous run, it'll stay cached locally for quite a long time (which is great!).  but it means external changes go unnoticed for much longer than they should.

ideas:
- add a new stage that runs after SetupBoard
- have it run a query to get the full list of source files:
  emerge-$BOARD -pvf virtual/target-os .....
- extract all the ones we care about (i.e. our mirrors)
- run parallel HEAD requests on all the URIs to verify the files exist (no 404) and get the current file size
- check the file size against the Manifest files to make sure they're in sync

if the files change content but not size, we'll still miss that, but i think that's a much less common scenario, and we usually have versioning enabled on our GS buckets to recover the old files.

my concern would be running a couple hundred HEAD requests, but those should be relatively light -- a couple KiB per connection.

i guess the best place to run this stage would be pre-cq and cq ?
 

Comment 1 by sjg@chromium.org, Sep 11 2017

Cc: akes...@chromium.org
Components: Infra>Client>ChromeOS>CI
Components: -Infra>Client>ChromeOS

Sign in to add a comment