Issue 597033 has been merged into this issue.

+sheriffs

It looks like we started failing InitSDK around 8052 and I don't see anything interesting in there.

https://crosland.corp.google.com/log/8051.0.0..8052.0.0

Perhaps we should clobber this builder and see if that clears it up?

Unfortunately clobber did not help, since we have not built for Pineview in in over two weeks, this should probably be Pri-0, and possibly a tree closer.

+vapier for any other ideas, is our stage3 package corrupted somehow?

cros sheriff, any idea what causes initSDK failure?

An invalid tarball would certainly do the trick.  Can we download it and verify it is OK locally?

I d/led it from gs and was able to successfully extract it.

# gsutil cp gs://chromiumos-sdk/cros-sdk-2016.03.08.233108.tar.xz ./
# md5sum cros-sdk-2016.03.08.233108.tar.xz 
bbcd060b7b5ee39f557ad1c21b7237d6  cros-sdk-2016.03.08.233108.tar.xz
# sudo tar xf cros-sdk-2016.03.08.233108.tar.xz -C cros_sdk/
# ls cros_sdk
bin    dev  home  lib32  media  opt       proc  run   sys  usr
build  etc  lib   lib64  mnt    packages  root  sbin  tmp  var
#

file info
# gsutil stat gs://chromiumos-sdk/cros-sdk-2016.03.08.233108.tar.xz                                                                                                               
gs://chromiumos-sdk/cros-sdk-2016.03.08.233108.tar.xz:
        Creation time:          Wed, 09 Mar 2016 21:23:50 GMT
        Content-Length:         1376730204
        Content-Type:           application/x-tar
        Hash (crc32c):          2ExvfA==
        Hash (md5):             u80GC3te459VetHCG3I31g==
        ETag:                   CPilroPFtMsCEAE=
        Generation:             1457558630667000
        Metageneration:         1

tried it on the latest tarball that failed and used the command in make_chroot.sh and it still worked for me.

# sudo xz -dc cros-sdk-2016.03.23.082418.tar.xz | sudo tar -xp -C test/
# ls test
bin  build  dev  etc  home  lib  lib32  lib64  media  mnt  opt  packages  proc  root  run  sbin  sys  tmp  usr  var

does the build server cache that file?  it appears to have a local path to it in the failing command

I don't believe so, it looks like it downloads it fresh everytime given: https://cs.corp.google.com/chromeos_public/chromite/scripts/cros_sdk.py?rcl=b9887e147151d5cfdd24ac58493a3bee83fad04a&l=102

I'm going to clobber this and see if it resolves the issue. (This is done by clicking Force Build with the Clobber checkbox enabled).

The clobber last week did not fix it :(.

We are having similar issue on rambi-release

https://uberchromegw.corp.google.com/i/chromeos/builders/rambi-release/builds/686/steps/SyncChrome/logs/stdio

I don't see the similarity in #18. 

https://uberchromegw.corp.google.com/i/chromeos/builders/pineview-release-group/builds/2240/steps/InitSDK/logs/stdio

10:46:59: NOTICE: Downloading SDK tarball...
10:46:59: NOTICE: Creating chroot. This may take a few minutes...

I'm not convinced that the tarball is actually being downloaded, given that it is taking less than 1s. Maybe it is somehow trying the wrong URL?

Running ['/b/cbuild/internal_master/src/scripts/sdk_lib/make_chroot.sh', '--stage3_path', '/b/cbuild/internal_master/.cache/sdks/cros-sdk-2016.03.27.171405.tar.xz', '--chroot', '/b/cbuild/internal_master/chroot', '--cache_dir', '/b/cbuild/internal_master/.cache'] failed!

Maybe it is trying to re-use that cached file and the cached file is corrupt? I can do in to the builder and delete the file...

File deleted, trying again...

Issue 598354 has been merged into this issue.

Ok, after deleting the cache it definitely downloaded the tarball, but then it failed anyway.

vapier@ can you weigh in? Is there some use flag issue? The message "make_chroot.sh: Building with a non-empty USE: -cros-debug chrome_internal" is suspicious.

---
13:11:12: NOTICE: Downloading SDK tarball...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  2 1397M    2 39.3M    0     0  56.7M      0  0:00:24 --:--:--  0:00:24 57.4M
  8 1397M    8  113M    0     0  66.9M      0  0:00:20  0:00:01  0:00:19 67.2M
 16 1397M   16  225M    0     0  83.6M      0  0:00:16  0:00:02  0:00:14 83.8M
 24 1397M   24  336M    0     0  74.1M      0  0:00:18  0:00:04  0:00:14 74.3M
 25 1397M   25  354M    0     0  75.4M      0  0:00:18  0:00:04  0:00:14 75.5M
 33 1397M   33  466M    0     0  81.8M      0  0:00:17  0:00:05  0:00:12 85.3M
 41 1397M   41  577M    0     0  86.3M      0  0:00:16  0:00:06  0:00:10 92.9M
 48 1397M   48  683M    0     0  88.8M      0  0:00:15  0:00:07  0:00:08 91.6M
 56 1397M   56  795M    0     0  91.4M      0  0:00:15  0:00:08  0:00:07  110M
 57 1397M   57  808M    0     0  83.1M      0  0:00:16  0:00:09  0:00:07 90.2M
 65 1397M   65  916M    0     0  85.7M      0  0:00:16  0:00:10  0:00:06 90.0M
 73 1397M   73 1028M    0     0  87.9M      0  0:00:15  0:00:11  0:00:04 90.0M
 81 1397M   81 1140M    0     0  89.8M      0  0:00:15  0:00:12  0:00:03 91.3M
 89 1397M   89 1251M    0     0  91.4M      0  0:00:15  0:00:13  0:00:02 91.3M
 97 1397M   97 1363M    0     0  92.8M      0  0:00:15  0:00:14  0:00:01  111M
100 1397M  100 1397M    0     0  93.1M      0  0:00:14  0:00:14 --:--:--  111M
13:11:27: NOTICE: Creating chroot. This may take a few minutes...
make_chroot.sh: Building with a non-empty USE: -cros-debug chrome_internal
This modifies the expected behaviour and can fail.

INFO    cros_sdk:make_chroot: Unpacking stage3...
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors

(that was https://uberchromegw.corp.google.com/i/chromeos/builders/pineview-release-group/builds/2241/steps/InitSDK/logs/stdio )

looks like build106-m2 is broken and should be decommissioned:

$ xzcat .cache/sdks/cros-sdk-2016.03.28.044537.tar.xz >/dev/null
Bus error

$ sudo dmesg -c
[ 7136.586131] sd 0:2:0:0: [sda] Unhandled error code
[ 7136.586148] sd 0:2:0:0: [sda]  
[ 7136.586151] Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
[ 7136.586154] sd 0:2:0:0: [sda] CDB: 
[ 7136.586156] Read(16): 88 00 00 00 00 00 00 23 3c 78 00 00 00 08 00 00
[ 7136.586174] end_request: I/O error, dev sda, sector 2309240

The following revision refers to this bug:
  http://goto.ext.google.com/viewvc/chrome-internal?view=rev&revision=85863

------------------------------------------------------------------
r85863 | dnj@google.com | 2016-03-29T14:18:52.983311Z

-----------------------------------------------------------------

We started getting Pineview builds on Friday for everything except mario (see  issue 600500 ).  Is there anything more to be done on this bug?

since dnj@ took the bots out of rotation, i don't think so