python-gflags-2.0.tar.gz and boost_1_62_0.tar.bz2 md5sum changed on mirror |
||||||||||||||||
Issue descriptionThis change: https://chromium-review.googlesource.com/c/478456/ to an init file in platform2/init/upstart has been rejected twice for a failure to build apparently unrelated packages. There may be technical limitations to this, but it could also be a bug. ------------------ The following build(s) failed: caroline-no-vmtest-pre-cq: The BuildPackages stage failed: Packages failed in ./build_packages: dev-libs/boost in https://luci-milo.appspot.com/buildbot/chromiumos.tryserver/no_vmtest_pre_cq/30998 This failure was probably caused by your change. and The following build(s) failed: whirlwind-no-vmtest-pre-cq: The BuildPackages stage failed: Packages failed in ./build_packages: dev-python/python-gflags in https://luci-milo.appspot.com/buildbot/chromiumos.tryserver/no_vmtest_pre_cq/31178 This failure was probably caused by your change.
,
Apr 19 2017
I verified and the checksums _do not_ match! $ gsutil cp gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0.tar.gz . Copying gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0.tar.gz... / [1 files][ 63.6 KiB/ 63.6 KiB] Operation completed over 1 objects/63.6 KiB. $ sha256sum python-gflags-2.0.tar.gz 0dff6360423f3ec08cbe3bfaf37b339461a54a21d13be0dd5d9c9999ce531078 python-gflags-2.0.tar.gz [portage-stable]$ cat dev-python/python-gflags/Manifest ... SHA256 11066217acb8cd8519a4c872cb3fe64f02bcf105802bb761ab0de55c2386cd6
,
Apr 19 2017
Had a typo in previous update. The sha256 in Manifest is: SHA256 311066217acb8cd8519a4c872cb3fe64f02bcf105802bb761ab0de55c2386cd6 Also it doesn't look like the Manifest changed. So it looks like the tarball in gcs bucket got corrupted somehow? FWIW, my local build cache still has the correct tarball.
,
Apr 19 2017
This looks like fallout from issue 703244
,
Apr 19 2017
+nxia, cros-infra-deputy.
,
Apr 19 2017
I actually expect this to fail _everything_
,
Apr 19 2017
We do have versioning setup in our distfiles mirror.
Taking as example the failing package python-gflags-2.0:
pprabhu@pprabhu:chromeos-admin$ gsutil ls -la gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0*
64929 2012-05-08T04:53:56Z gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0.tar.gz#2 metageneration=1
65094 2017-04-19T00:47:32Z gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0.tar.gz#1492562852564083 metageneration=1
So, I can hotfix this if I can restore the old version of the objects.
,
Apr 19 2017
yes, it prob is due to the script run:
$ gsutil ls -l -a gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0.tar.gz
64929 2012-05-08T04:53:56Z gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0.tar.gz#2 metageneration=1
65094 2017-04-19T00:47:32Z gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0.tar.gz#1492562852564083 metageneration=1
i restored the old one:
$ gsutil cp gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0.tar.gz#2 ./foo
$ gsutil cp -a public-read foo gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0.tar.gz
$ gsutil ls -l -a gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0.tar.gz
64929 2012-05-08T04:53:56Z gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0.tar.gz#2 metageneration=1
65094 2017-04-19T00:47:32Z gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0.tar.gz#1492562852564083 metageneration=1
64929 2017-04-19T18:03:12Z gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0.tar.gz#1492624992054035 metageneration=1
$ gsutil cat gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0.tar.gz#2 | md5sum
c3ab70218dbf945cc32c0cd64c51d162 -
$ gsutil cat gs://chromeos-mirror/gentoo/distfiles/python-gflags-2.0.tar.gz | md5sum
c3ab70218dbf945cc32c0cd64c51d162 -
,
Apr 19 2017
pprabhu@pprabhu:2$ gsutil cp -a public-read python-gflags-2.0.tar.gz gs://chromeos-mirror/gentoo/distfiles/ Copying file://python-gflags-2.0.tar.gz [Content-Type=application/x-tar]... / [1 files][ 63.4 KiB/ 63.4 KiB] Operation completed over 1 objects/63.4 KiB. With that, (cr) ((3f7688f...)) pprabhu@pprabhu ~/trunk/src/scripts $ emerge-guado_moblab python-gflags passes locally (which was failing earlier) ......... Ahem... OK, we both did it ;)
,
Apr 19 2017
did we also restore boost? boost-1.62.0-r1: !!! Fetched file: boost_1_62_0.tar.bz2 VERIFY FAILED! boost-1.62.0-r1: !!! Reason: Filesize does not match recorded size boost-1.62.0-r1: !!! Got: 84513338 boost-1.62.0-r1: !!! Expected: 84529021 boost-1.62.0-r1: Refetching... File renamed to '/var/cache/chromeos-cache/distfiles/target/boost_1_62_0.tar.bz2._checksum_failure_.3Z9nV4' boost-1.62.0-r1: boost-1.62.0-r1: !!! Couldn't download 'boost_1_62_0.tar.bz2'. Aborting. boost-1.62.0-r1: * Fetch failed for 'dev-libs/boost-1.62.0-r1', Log file: boost-1.62.0-r1: * '/build/caroline/tmp/portage/logs/dev-libs:boost-1.62.0-r1:20170419-042015.log' boost-1.62.0-r1: >>> Failed to emerge dev-libs/boost-1.62.0-r1 for /build/caroline/, Log file: boost-1.62.0-r1: >>> '/build/caroline/tmp/portage/logs/dev-libs:boost-1.62.0-r1:20170419-042015.log' boost-1.62.0-r1: boost-1.62.0-r1: * Messages for package dev-libs/boost-1.62.0-r1 merged to /build/caroline/: boost-1.62.0-r1: boost-1.62.0-r1: * Fetch failed for 'dev-libs/boost-1.62.0-r1', Log file: boost-1.62.0-r1: * '/build/caroline/tmp/portage/logs/dev-libs:boost-1.62.0-r1:20170419-042015.log'
,
Apr 19 2017
,
Apr 19 2017
I've restored boost as well. I'm doing a more thorough check now. pprabhu@pprabhu:b713226$ gsutil cat gs://chromeos-mirror/gentoo/distfiles/boost_1_62_0.tar.bz2#1475376633042000 | md5sum 8567f6faf8b400ad1fbca849300e9da8 - pprabhu@pprabhu:b713226$ gsutil cat gs://chromeos-mirror/gentoo/distfiles/boost_1_62_0.tar.bz2#1492625734672914 | md5sum 8567f6faf8b400ad1fbca849300e9da8 - Will lower priority once all distfiles are restored
,
Apr 19 2017
Issue 713266 has been merged into this issue.
,
Apr 19 2017
I've launched a ToT caroline-novmtest-pre-cq trybot: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/no_vmtest_pre_cq/builds/31234 The issues so far were uncovered on caroline, so we can open the tree once caroline passes. In the meantime, I'm running an md5sum against all the packages that were updated yesterday. That will take a while since it involves cat'ing files from gs.
,
Apr 19 2017
FWIW, lakitu-release seems to have successfully cleared the 'BuildPackages' stage in its latest run: https://uberchromegw.corp.google.com/i/chromeos/builders/lakitu-release/builds/2211 Thanks for the follow ups!
,
Apr 19 2017
caroline-novmtest-pre-cq is past BuildPackages as well. I'm calling this outage over.
,
Apr 19 2017
Thank you for the quick fix. I guess it's still worth asking the original question: why did a CL get marked -1 for a BuildPackages failure in seemingly unrelated packages.
,
Apr 19 2017
Search through a large fraction of the distfiles (my script died due to no local disk space at ~90%), and found some more bad packages: flac-1.3.2.tar.xz jnr-ffi-2.0.2.tar.gz libaccounts-glib-1.21.tar.gz netpipes-4.2-export.tar.gz paredit-23.tar.xz progressbar-2.3.tar.gz psmisc-22.21.tar.gz pyglet-1.1.4.tar.gz pylibmc-1.5.1.tar.gz scons-2.3.5.tar.gz I've fixed these as well. Calling this fixed unless someone tells me otherwise.
,
Apr 19 2017
Would you be so kind to address comment #17? Should I open a separate bug?
,
Apr 19 2017
Re #19: We have some ability to detect irrelevant changes based on which paladin slave(s) see the failure. But, if your CL is built on the paladin that fails BuildPackages, we do not have the ability to determine whether your CL could be at fault. This feature would require us to infer / hard-code some knowledge about what packages are self-contained, and which can affect other packages (think chromiumos-overlay, a CL in that can break pretty much anything). You can file a feature request for this, but currently this is WAI.
,
Apr 19 2017
Thanks!
,
May 30 2017
,
Aug 1 2017
,
Jan 22 2018
|
||||||||||||||||
►
Sign in to add a comment |
||||||||||||||||
Comment 1 by adityakali@google.com
, Apr 19 2017Components: Infra>Platform>Buildbot
Labels: -Pri-2 Pri-1