Project: chromium Issues People Development process History Sign in
New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 4 users
Status: Fixed
Owner:
Closed: Apr 19
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment
python-gflags-2.0.tar.gz and boost_1_62_0.tar.bz2 md5sum changed on mirror
Project Member Reported by semenzato@chromium.org, Apr 19 Back to list
This change:

https://chromium-review.googlesource.com/c/478456/

to an init file in platform2/init/upstart has been rejected twice for a failure to build apparently unrelated packages.  There may be technical limitations to this, but it could also be a bug.

------------------

The following build(s) failed:

caroline-no-vmtest-pre-cq: The BuildPackages stage failed: Packages failed in ./build_packages: dev-libs/boost in https://luci-milo.appspot.com/buildbot/chromiumos.tryserver/no_vmtest_pre_cq/30998

This failure was probably caused by your change.

and

The following build(s) failed:

whirlwind-no-vmtest-pre-cq: The BuildPackages stage failed: Packages failed in ./build_packages: dev-python/python-gflags in https://luci-milo.appspot.com/buildbot/chromiumos.tryserver/no_vmtest_pre_cq/31178

This failure was probably caused by your change.


 
Cc: adityakali@google.com
Components: Infra>Platform>Buildbot
Labels: -Pri-2 Pri-1

Actually its not just limiting the CLs. This is breaking lakitu-release and other builders as well: https://uberchromegw.corp.google.com/i/chromeos/builders/lakitu-release/builds/2209
=== Start output for job python-gflags-2.0 (0m0.5s) ===
python-gflags-2.0: >>> Emerging (1 of 1) dev-python/python-gflags-2.0::portage-stable for /build/lakitu/
python-gflags-2.0:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
python-gflags-2.0:                                  Dload  Upload   Total   Spent    Left  Speed
python-gflags-2.0: 
python-gflags-2.0:   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
python-gflags-2.0:   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
python-gflags-2.0: curl: (22) The requested URL returned error: 404 Not Found
python-gflags-2.0:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
python-gflags-2.0:                                  Dload  Upload   Total   Spent    Left  Speed
python-gflags-2.0: 
python-gflags-2.0:   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
python-gflags-2.0: 100 65094  100 65094    0     0  1968k      0 --:--:-- --:--:-- --:--:-- 2050k
python-gflags-2.0: !!! Fetched file: python-gflags-2.0.tar.gz VERIFY FAILED!
python-gflags-2.0: !!! Reason: Filesize does not match recorded size
python-gflags-2.0: !!! Got:      65094
python-gflags-2.0: !!! Expected: 64929
python-gflags-2.0: Refetching... File renamed to '/var/cache/chromeos-cache/distfiles/target/python-gflags-2.0.tar.gz._checksum_failure_.tsUswu'
python-gflags-2.0: 
python-gflags-2.0: !!! Couldn't download 'python-gflags-2.0.tar.gz'. Aborting.
python-gflags-2.0:  * Fetch failed for 'dev-python/python-gflags-2.0', Log file:
python-gflags-2.0:  *  '/build/lakitu/tmp/portage/logs/dev-python:python-gflags-2.0:20170419-033549.log'
python-gflags-2.0: >>> Failed to emerge dev-python/python-gflags-2.0 for /build/lakitu/, Log file:
python-gflags-2.0: >>>  '/build/lakitu/tmp/portage/logs/dev-python:python-gflags-2.0:20170419-033549.log'
python-gflags-2.0: 
python-gflags-2.0:  * Messages for package dev-python/python-gflags-2.0 merged to /build/lakitu/:
python-gflags-2.0: 
python-gflags-2.0:  * Fetch failed for 'dev-python/python-gflags-2.0', Log file:
python-gflags-2.0:  *  '/build/lakitu/tmp/portage/logs/dev-python:python-gflags-2.0:20170419-033549.log'
=== Complete: job python-gflags-2.0 (0m0.5s) ===

Also happening on R59 release branch as well: https://uberchromegw.corp.google.com/i/chromeos_release/builders/lakitu-release%20release-R59-9460.B/builds/1


I verified and the checksums _do not_ match!

$ gsutil cp gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0.tar.gz .
Copying gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0.tar.gz...
/ [1 files][ 63.6 KiB/ 63.6 KiB]                                                
Operation completed over 1 objects/63.6 KiB.                                     
$ sha256sum python-gflags-2.0.tar.gz 
0dff6360423f3ec08cbe3bfaf37b339461a54a21d13be0dd5d9c9999ce531078  python-gflags-2.0.tar.gz


[portage-stable]$ cat dev-python/python-gflags/Manifest 
...
SHA256 11066217acb8cd8519a4c872cb3fe64f02bcf105802bb761ab0de55c2386cd6

Had a typo in previous update. The sha256 in Manifest is:
SHA256 311066217acb8cd8519a4c872cb3fe64f02bcf105802bb761ab0de55c2386cd6

Also it doesn't look like the Manifest changed. So it looks like the tarball in gcs bucket got corrupted somehow?

FWIW, my local build cache still has the correct tarball.
Owner: pprabhu@chromium.org
Status: Assigned
This looks like fallout from issue 703244
Cc: -pprabhu@chromium.org nxia@chromium.org
+nxia, cros-infra-deputy.
Labels: -Pri-1 Pri-0
Status: Started
I actually expect this to fail _everything_
Labels: -current-issue
We do have versioning setup in our distfiles mirror.
Taking as example the failing package python-gflags-2.0:

pprabhu@pprabhu:chromeos-admin$ gsutil ls -la gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0*
     64929  2012-05-08T04:53:56Z  gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0.tar.gz#2  metageneration=1
     65094  2017-04-19T00:47:32Z  gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0.tar.gz#1492562852564083  metageneration=1


So, I can hotfix this if I can restore the old version of the objects.
Labels: -Pri-0 Pri-2
Summary: python-gflags-2.0.tar.gz md5sum changed on mirror (was: CL rejected for apparently unrelated BuildPackages failures)
yes, it prob is due to the script run:
$ gsutil ls -l -a gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0.tar.gz 
     64929  2012-05-08T04:53:56Z  gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0.tar.gz#2  metageneration=1
     65094  2017-04-19T00:47:32Z  gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0.tar.gz#1492562852564083  metageneration=1

i restored the old one:
$ gsutil cp gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0.tar.gz#2 ./foo
$ gsutil cp -a public-read foo gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0.tar.gz
$ gsutil ls -l -a gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0.tar.gz 
     64929  2012-05-08T04:53:56Z  gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0.tar.gz#2  metageneration=1
     65094  2017-04-19T00:47:32Z  gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0.tar.gz#1492562852564083  metageneration=1
     64929  2017-04-19T18:03:12Z  gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0.tar.gz#1492624992054035  metageneration=1
$ gsutil cat gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0.tar.gz#2 | md5sum
c3ab70218dbf945cc32c0cd64c51d162  -
$ gsutil cat gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0.tar.gz | md5sum
c3ab70218dbf945cc32c0cd64c51d162  -

pprabhu@pprabhu:2$ gsutil cp -a public-read python-gflags-2.0.tar.gz gs://chromeos-mirror/gentoo/distfiles/            Copying file://python-gflags-2.0.tar.gz [Content-Type=application/x-tar]...
/ [1 files][ 63.4 KiB/ 63.4 KiB]                                                
Operation completed over 1 objects/63.4 KiB. 


With that, 
(cr) ((3f7688f...)) pprabhu@pprabhu ~/trunk/src/scripts $ emerge-guado_moblab python-gflags

passes locally (which was failing earlier)



.........
Ahem... OK, we both did it ;)
did we also restore boost?

boost-1.62.0-r1: !!! Fetched file: boost_1_62_0.tar.bz2 VERIFY FAILED!
boost-1.62.0-r1: !!! Reason: Filesize does not match recorded size
boost-1.62.0-r1: !!! Got:      84513338
boost-1.62.0-r1: !!! Expected: 84529021
boost-1.62.0-r1: Refetching... File renamed to '/var/cache/chromeos-cache/distfiles/target/boost_1_62_0.tar.bz2._checksum_failure_.3Z9nV4'
boost-1.62.0-r1: 
boost-1.62.0-r1: !!! Couldn't download 'boost_1_62_0.tar.bz2'. Aborting.
boost-1.62.0-r1:  * Fetch failed for 'dev-libs/boost-1.62.0-r1', Log file:
boost-1.62.0-r1:  *  '/build/caroline/tmp/portage/logs/dev-libs:boost-1.62.0-r1:20170419-042015.log'
boost-1.62.0-r1: >>> Failed to emerge dev-libs/boost-1.62.0-r1 for /build/caroline/, Log file:
boost-1.62.0-r1: >>>  '/build/caroline/tmp/portage/logs/dev-libs:boost-1.62.0-r1:20170419-042015.log'
boost-1.62.0-r1: 
boost-1.62.0-r1:  * Messages for package dev-libs/boost-1.62.0-r1 merged to /build/caroline/:
boost-1.62.0-r1: 
boost-1.62.0-r1:  * Fetch failed for 'dev-libs/boost-1.62.0-r1', Log file:
boost-1.62.0-r1:  *  '/build/caroline/tmp/portage/logs/dev-libs:boost-1.62.0-r1:20170419-042015.log'
Summary: python-gflags-2.0.tar.gz and boost_1_62_0.tar.bz2 md5sum changed on mirror (was: python-gflags-2.0.tar.gz md5sum changed on mirror)
Labels: -Pri-2 Pri-0
I've restored boost as well.
I'm doing a more thorough check now.

pprabhu@pprabhu:b713226$ gsutil cat gs://chromeos-mirror/gentoo/distfiles/boost_1_62_0.tar.bz2#1475376633042000 | md5sum
8567f6faf8b400ad1fbca849300e9da8  -
pprabhu@pprabhu:b713226$ gsutil cat gs://chromeos-mirror/gentoo/distfiles/boost_1_62_0.tar.bz2#1492625734672914 | md5sum
8567f6faf8b400ad1fbca849300e9da8  -

Will lower priority once all distfiles are restored
Cc: tbroch@chromium.org jclinton@chromium.org pprabhu@chromium.org owenlin@chromium.org mnissler@chromium.org x...@chromium.org zhihongyu@chromium.org djkurtz@chromium.org
 Issue 713266  has been merged into this issue.
I've launched a ToT caroline-novmtest-pre-cq trybot: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/no_vmtest_pre_cq/builds/31234

The issues so far were uncovered on caroline, so we can open the tree once caroline passes.

In the meantime, I'm running an md5sum against all the packages that were updated yesterday. That will take a while since it involves cat'ing files from gs.
FWIW, lakitu-release seems to have successfully cleared the 'BuildPackages' stage in its latest run: https://uberchromegw.corp.google.com/i/chromeos/builders/lakitu-release/builds/2211

Thanks for the follow ups!
Labels: -Pri-0 Pri-2
caroline-novmtest-pre-cq is past BuildPackages as well. I'm calling this outage over.
Thank you for the quick fix.

I guess it's still worth asking the original question: why did a CL get marked -1 for a BuildPackages failure in seemingly unrelated packages.
Status: Fixed
Search through a large fraction of the distfiles (my script died due to no local disk space at ~90%), and found some more bad packages:

flac-1.3.2.tar.xz                                                                   
jnr-ffi-2.0.2.tar.gz                                                                
libaccounts-glib-1.21.tar.gz                                                        
netpipes-4.2-export.tar.gz                                                          
paredit-23.tar.xz                                                                   
progressbar-2.3.tar.gz                                                              
psmisc-22.21.tar.gz                                                                 
pyglet-1.1.4.tar.gz                                                                 
pylibmc-1.5.1.tar.gz                                                                
scons-2.3.5.tar.gz                                                                  

I've fixed these as well.

Calling this fixed unless someone tells me otherwise.

Status: Available
Would you be so kind to address comment #17?  Should I open a separate bug?
Status: Fixed
Re #19: We have some ability to detect irrelevant changes based on which paladin slave(s) see the failure. But, if your CL is built on the paladin that fails BuildPackages, we do not have the ability to determine whether your CL could be at fault.

This feature would require us to infer / hard-code some knowledge about what packages are self-contained, and which can affect other packages (think chromiumos-overlay, a CL in that can break pretty much anything).

You can file a feature request for this, but currently this is WAI.
Thanks!
Labels: VerifyIn-60
Labels: VerifyIn-61
Sign in to add a comment