Issue 876633  has been merged into this issue.

Only ARM boards and only this specific suites seem to be affected. I've tried comparing various logs between failing and successful builds, but couldn't find anything useful. Also, python-uinput seems to be present and successful in BuildPackages, so I'm not sure why it could be missing.

P0 this is killing CQ.

Info from Alex and my investigation.

This only affects arm64 boards.

Sometime recently python's libraries moved from /usr/local/lib to /usr/local/lib64. (not sure when this happened, or why, still looking).

Python's path has not changed though, and still tries to look for /usr/local/lib/python2.7

I've confirmed on an older test image that this is the case, and Alex has checked that recent builds have the mismatched path.

TL;DR please try to add

REQUIRE_SSP = False

in control file.

For details, please refer crbug.com/541244 and crbug.com/530386

Looking at the logs for tests in the OP, the test was *not* running inside SSP, so #5 will not get us anything.

Also note that the uinput import is failing on the client, i.e., the DUT.
This suggests that there is no change on the test-infra side to be implicated.

We dropped some python modules either from the test image, or from autotest's insane test dependency installation flow.

bleung@'s comment #4 is on the mark:
On an affected DUT, I see:

localhost ~ # python -c 'import sys; print(sys.path)'
['', '/usr/local/lib/python27.zip', '/usr/local/lib/python2.7', '/usr/local/lib/python2.7/plat-linux2', '/usr/local/lib/python2.7/lib-tk', '/usr/local/lib/python2.7/lib-old', '/usr/local/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/site-packages/gtk-2.0', '/usr/local/telemetry/src/third_party/catapult/telemetry']

and,
localhost ~ # ls /usr/local/lib/python2.7/site-packages/| grep -i uinput
localhost ~ # ls /usr/local/lib64/python2.7/site-packages/| grep -i uinput
_libsuinput.so
python_uinput-0.11.2-py2.7.egg-info
uinput

So the DUT's default python search path is broken.

localhost ~ # file $(realpath -e $(which python))
/usr/local/bin/python2.7: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32, BuildID[sha1]=c4751b981c37fbd2f9c69d991b1e130e32dd3cf5, stripped

And this is on an armv7l arch machine.
Why are all the packages deployed in lib64?

Passing to the Bobby to at least find an owner.

I have already asked several times on crosoncall that clobbering the affecting builders should fix the problem. Can someone please actually do it?

As a proof, here is a nyan_big-paladin tryjob with hwtest that passed.
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8937446435147852384

lannm@ mentioned that we got a single green CQ run after a long time yesterday, and this may have started happening after this. That could be because new prebuilts got published.
Digging out crosland from the green run before that: https://crosland.corp.google.com/log/10968.0.0..10993.0.0

I don't see anything obvious...

As I expected, bob-paladin has passed after clobbering (https://uberchromegw.corp.google.com/i/chromeos/builders/bob-paladin/builds/3855)

Now only if CI team would actually listen to me and clobber the rest of broken ARM builders.

I ran:

./ccompute ri cros-beefy{80,41,60,63,27,70,32,58}-c2

Which should reinstance the builders associated with:

https://uberchromegw.corp.google.com/i/chromeos/builders/nyan_big-paladin
https://uberchromegw.corp.google.com/i/chromeos/builders/peach_pit-paladin
https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_mighty-paladin
https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_minnie-paladin
https://uberchromegw.corp.google.com/i/chromeos/builders/elm-paladin
https://uberchromegw.corp.google.com/i/chromeos/builders/hana-paladin
https://uberchromegw.corp.google.com/i/chromeos/builders/kevin-arcnext-paladin
https://uberchromegw.corp.google.com/i/chromeos/builders/kevin-paladin

I didn't think to save a chroot off of one of them (it's late, I'm tired).

We won't be able to do manual interventions of this kind as easily in the future, so we need to instead harden the system to do a better kind of automatic cleanup. If we can't, the builder fleet will become unmanageable.

Sheriffs, is this fixed after the chroot wipes?

Yes, it looks like clobbering the builders fixed this issue.

We need a specific CL that caused the problem. Backtrack to the first report of the Python failure, and look for CL's that landed in the preceding CQ run. Then, we need to identify a way to prevent this from happening again.

Lowering priority to reflect that the emergency is over.

The first build that exhibited this problem was Build ID 2865481, build 19465.

https://uberchromegw.corp.google.com/i/chromeos/builders/master-paladin/builds/19465

Should we look at build 19464, then?

Yep.

Hi, CI bobby, nyan_blaze-paladin is still failing due to the same reason:

https://uberchromegw.corp.google.com/i/chromeos/builders/nyan_blaze-paladin

Can I request a clobber for it?

I'm looking at the latest build and will clobber to clear the issue.

-- Mike

The builder in question, cros-beefy81-c2, has been clobbered.  

-- Mike

Uhm, it looks like this is back:
https://luci-milo.appspot.com/buildbot/chromeos/veyron_mighty-paladin/9569

Perhaps this comes with some CL in the CQ, which we skipped after the clobber? Let me look through them.

Dumped list of files (cd /; find) from a good and a bad DUT. The files are too big, so uploaded to a shared drive directory:
https://drive.google.com/drive/folders/1tDAYeWrLHdG5CWbu7Ri1YUc_s7q5kaip?usp=sharing

I've spent the other few hours on trying to find a working way to access -c2 builders, but in the end I couldn't find any. Giving up, sorry.

What were you going to look at on the builder that would have been of interest?

Back to P0. Sheriffs, please drop everything else and work on this full time. Don't hesitate to pull others in if needed.

Not clobbering the builders in case there's some debugging that we want to do before doing that.

Wanted to check at least various portage environment (emerge-$BOARD --info -vvv). There might be also some more detailed build logs of various python packages, which I couldn't find in our frontends.

Charles, as current oncall, could you gather the information from the builders 
requested in #30?

For the "more detailed build logs", I was expecting something to be in /var/log/portage, but in my local chroot the logs there are of very low verbosity. +vapier@ for some ideas.

Looking at this now.

For comment #30, here's the output:
!!! CONFIG_PROTECT is empty for '/build/nyan_big/'
Portage 2.2.28 (python 2.7.10-final-0, base, gcc-4.9.x, unavailable, 3.13.0-129-generic x86_64)
=================================================================
System uname: Linux-3.13.0-129-generic-x86_64-Intel-R-_Xeon-R-_CPU_@_2.30GHz-with-glibc2.2.5
KiB Mem:   214611996 total, 159792568 free
KiB Swap:          0 total,         0 free
sh bash 4.3_p48-r1
ld GNU gold (binutils-2.27.0-r11-cce69c1b1bfc46de5e1cdd4923f87215c0edb884_cos_gg 2.27.0.20170315) 1.12
ccache version 3.2.4 [disabled]
app-shells/bash:          4.3_p48-r1::portage-stable
dev-lang/python:          2.7.10-r3::chromiumos
sys-apps/baselayout:      2.2-r1::chromiumos
sys-apps/sandbox:         2.11-r2::portage-stable
sys-devel/autoconf:       2.69::portage-stable
sys-devel/automake:       1.11.6-r1::portage-stable, 1.13.4::portage-stable, 1.14.1::portage-stable, 1.15.1-r2::portage-stable
sys-devel/binutils:       2.27.0-r11::chromiumos
sys-devel/libtool:        2.4.6-r6::chromiumos
sys-devel/make:           4.2.1-r3::portage-stable
sys-kernel/linux-headers: 4.4-r24::chromiumos (virtual/os-headers)
Repositories:

portage-stable
    location: /mnt/host/source/src/third_party/portage-stable
    masters: eclass-overlay
    priority: -1000

chromiumos
    location: /mnt/host/source/src/third_party/chromiumos-overlay
    masters: portage-stable eclass-overlay
    priority: 0

eclass-overlay
    location: /mnt/host/source/src/third_party/eclass-overlay
    priority: 1

chromeos
    location: /mnt/host/source/src/private-overlays/chromeos-overlay
    masters: portage-stable chromiumos eclass-overlay
    priority: 2

chromeos-partner
    location: /mnt/host/source/src/private-overlays/chromeos-partner-overlay
    masters: portage-stable chromiumos eclass-overlay
    priority: 3

nyan
    location: /mnt/host/source/src/overlays/overlay-nyan
    masters: portage-stable chromiumos eclass-overlay
    priority: 4

nyan-private
    location: /mnt/host/source/src/private-overlays/overlay-nyan-private
    masters: portage-stable chromiumos eclass-overlay
    priority: 5

nyan_big
    location: /mnt/host/source/src/overlays/overlay-variant-nyan-big
    masters: portage-stable chromiumos eclass-overlay nyan
    priority: 6

nyan_big-private
    location: /mnt/host/source/src/private-overlays/overlay-variant-nyan-big-private
    masters: portage-stable chromiumos eclass-overlay nyan
    priority: 7

ABI="arm"
ACCEPT_KEYWORDS="arm"
ACCEPT_LICENSE="* -@EULA -@CHROMEOS @CHROMEOS"
ACCEPT_PROPERTIES="*"
ACCEPT_RESTRICT="*"
APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias"
ARCH="arm"
ARM_FPU="neon"
AT_SYS_M4DIR="${SYSROOT}/usr/share/aclocal"
AUTOCLEAN="yes"
BOARD_OVERLAY="/mnt/host/source/src/private-overlays/chromeos-overlay /mnt/host/source/src/private-overlays/chromeos-partner-overlay /mnt/host/source/src/overlays/overlay-nyan /mnt/host/source/src/private-overlays/overlay-nyan-private /mnt/host/source/src/overlays/overlay-variant-nyan-big /mnt/host/source/src/private-overlays/overlay-variant-nyan-big-private"
BOARD_USE="nyan_big"
BOOTSTRAP_USE="cxx unicode python_targets_python3_2 python_targets_python2_7"
BOTO_CONFIG="/mnt/host/source/src/private-overlays/chromeos-overlay/googlestorage_account.boto"
BUILDBOT_BUILD=""
BUILD_CC="x86_64-pc-linux-gnu-clang"
BUILD_CXX="x86_64-pc-linux-gnu-clang++"
CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author"
CAMERAS="ptp2"
CBUILD="x86_64-pc-linux-gnu"
CC="armv7a-cros-linux-gnueabi-clang"
CFLAGS="-O2 -O2 -pipe -march=armv7-a -mtune=cortex-a15 -mfpu=neon -mfloat-abi=hard -g -fno-exceptions -fno-unwind-tables   -fno-asynchronous-unwind-tables "
CHOST="armv7a-cros-linux-gnueabi"
CHOST_arm="arm-unknown-linux-gnu"
CHROMEOS_CACHEDIR="/var/cache/chromeos-cache"
CHROMEOS_KERNEL_SPLITCONFIG="chromeos-tegra"
CHROMEOS_RELEASE_APPID="{DEV-BUILD}"
CHROOT_CWD=""
CLEAN_DELAY="5"
COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog"
COLLISION_IGNORE="*.py[co] *$py.class */dropin.cache /usr/lib/debug/.build-id"
CONFIG_PROTECT=""
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CROS_WORKON_SRCROOT="/mnt/host/source"
CROS_WORKON_TREE=""
CXX="armv7a-cros-linux-gnueabi-clang++"
CXXEXCEPTIONS="0"
CXXEXCEPTIONS_FLAGS="-fno-exceptions -fno-unwind-tables   -fno-asynchronous-unwind-tables "
CXXFLAGS="-O2 -O2 -O2 -pipe -march=armv7-a -mtune=cortex-a15 -mfpu=neon -mfloat-abi=hard -g -fno-exceptions -fno-unwind-tables   -fno-asynchronous-unwind-tables "
DEFAULT_ABI="arm"
DISTDIR="/var/lib/portage/distfiles-target"
EC_FIRMWARE="big"
EDITOR="/bin/nano"
ELIBC="glibc"
EMERGE_DEFAULT_OPTS="--oneshot"
EMERGE_WARNING_DELAY="10"
ENABLE_LICENSING="Y"
EPREFIX=""
EROOT="/build/nyan_big/"
EXTERNAL_TRUNK_PATH="/b/c/cbuild/repository"
EXTRA_BUILD_ARGS="proprietary_codecs=1"
FCFLAGS="-O2"
FEATURES="allow-missing-manifests assume-digests binpkg-logs buildpkg clean-logs config-protect-if-modified distlocks fixlafiles force-mirror network-sandbox news nodoc noinfo noman parallel-fetch parallel-install protect-owned sandbox sfperms splitdebug unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FETCHCOMMAND="curl --ipv4 -y 30 -f --retry 9 -L --output ${DISTDIR}/${FILE} ${URI}"
FETCHCOMMAND_AB="/mnt/host/source/chromite/bin/abutil fetch "${URI}" "${DISTDIR}/${FILE}""
FETCHCOMMAND_GS="bash -c 'BOTO_CONFIG=/mnt/host/source/src/private-overlays/chromeos-overlay/googlestorage_account.boto /mnt/host/source/chromite/bin/gs_fetch_binpkg "${URI}" "${DISTDIR}/${FILE}"'"
FETCHCOMMAND_RSYNC="rsync -avP "${URI}" "${DISTDIR}/${FILE}""
FETCHCOMMAND_SFTP="bash -c "x=\${2#sftp://} ; host=\${x%%/*} ; port=\${host##*:} ; host=\${host%:*} ; [[ \${host} = \${port} ]] && port=22 ; eval \"declare -a ssh_opts=(\${3})\" ; exec sftp -P \${port} \"\${ssh_opts[@]}\" \"\${host}:/\${x#*/}\" \"\$1\"" sftp "${DISTDIR}/${FILE}" "${URI}" "${PORTAGE_SSH_OPTS}""
FETCHCOMMAND_SSH="bash -c "x=\${2#ssh://} ; host=\${x%%/*} ; port=\${host##*:} ; host=\${host%:*} ; [[ \${host} = \${port} ]] && port=22 ; exec rsync --rsh=\"ssh -p\${port} \${3}\" -avP \"\${host}:/\${x#*/}\" \"\$1\"" rsync "${DISTDIR}/${FILE}" "${URI}" "${PORTAGE_SSH_OPTS}""
FFLAGS="-O2"
GCC_SPECS=""
GCC_VER="4.9.x"
GENTOO_MIRRORS="https://commondatastorage.googleapis.com/chromeos-localmirror https://commondatastorage.googleapis.com/chromeos-mirror/gentoo"
GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx"
GRUB_PLATFORMS=""
HOME="/home/chrome-bot"
INFOPATH="/usr/share/info:/usr/share/gcc-data/x86_64-pc-linux-gnu/4.9.x/info:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.27.0/info"
INPUT_DEVICES="evdev cmt"
INSTALL_MASK="   /usr/lib*/*.la   /etc/init.d /etc/conf.d   /etc/logrotate.d   /etc/sandbox.d   /usr/share/bash-completion "
IUSE_IMPLICIT="prefix prefix-guest"
JAVAC="/etc/java-config-2/current-system-vm/bin/javac"
JAVA_HOME="/etc/java-config-2/current-system-vm"
JDK_HOME="/etc/java-config-2/current-system-vm"
KERNEL="linux"
LADSPA_PATH="/usr/lib/ladspa"
LANG="en_US.UTF-8"
LATEST_RELEASE_CHROME_BINHOST="gs://chromeos-prebuilt/board/peach_pit/chrome-R70-10993.0.0-rc2/packages/"
LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text"
LC_MESSAGES="C"
LDFLAGS="-Wl,-O1 -Wl,-O2 -Wl,--as-needed"
LD_LIBRARY_PATH="/usr/local/lib"
LESS="-R -M --shift 5"
LESSOPEN="|lesspipe %s"
LIBDIR_arm="lib"
LIBDIR_x86="lib"
LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer"
LINGUAS="en"
LINUX_FIRMWARE="nvidia-xusb"
LOGNAME="root"
LS_COLORS="rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.cfg=00;32:*.conf=00;32:*.diff=00;32:*.doc=00;32:*.ini=00;32:*.log=00;32:*.patch=00;32:*.pdf=00;32:*.ps=00;32:*.tex=00;32:*.txt=00;32:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:"
MAIL="/var/mail/chrome-bot"
MAKEOPTS="-j32"
MANPAGER="manpager"
MANPATH="/etc/java-config-2/current-system-vm/man:/usr/local/share/man:/usr/share/man:/usr/share/gcc-data/x86_64-pc-linux-gnu/4.9.x/man:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.27.0/man:/etc/java-config-2/current-system-vm/man/"
MARCH_TUNE="-march=armv7-a -mtune=cortex-a15 -mfpu=neon -mfloat-abi=hard"
MULTILIB_ABIS="arm"
NETBEANS="apisupport cnd groovy gsf harness ide identity j2ee java mobility nb php profiler soa visualweb webcommon websvccommon xml"
OFFICE_IMPLEMENTATION="libreoffice"
OZONE_PLATFORM="gbm default_gbm"
OZONE_PLATFORM_DEFAULT="gbm"
PAGER="/usr/bin/less"
PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin"
PHP_TARGETS="php5-5"
PKGDIR="/build/nyan_big/packages/"
PKG_CONFIG="/build/nyan_big/build/bin/pkg-config"
PKG_INSTALL_MASK="   /usr/lib*/*.la   /etc/init.d /etc/conf.d   /etc/logrotate.d   /etc/sandbox.d   /usr/share/bash-completion "
PORTAGE_ARCHLIST="alpha amd64 amd64-fbsd amd64-linux arm arm-linux arm64 hppa hppa-hpux ia64 ia64-hpux ia64-linux m68k m68k-mint mips ppc ppc-aix ppc-macos ppc-openbsd ppc64 ppc64-linux s390 sh sparc sparc-fbsd sparc-solaris sparc64-freebsd sparc64-solaris x64-freebsd x64-macos x64-openbsd x64-solaris x86 x86-cygwin x86-fbsd x86-freebsd x86-interix x86-linux x86-macos x86-netbsd x86-openbsd x86-solaris x86-winnt"
PORTAGE_BINHOST=" gs://chromeos-prebuilt/board/arm-generic/paladin-R70-10996.0.0-rc1/packages/ gs://chromeos-prebuilt/board/nyan_big/paladin-R70-10996.0.0-rc1/packages/ gs://chromeos-prebuilt/board/peach_pit/chrome-R70-10993.0.0-rc2/packages/"
PORTAGE_BINPKG_TAR_OPTS="--checkpoint=1000"
PORTAGE_BIN_PATH="/usr/lib/portage/python2.7"
PORTAGE_BUNZIP2_COMMAND="pbunzip2 --ignore-trailing-garbage=1"
PORTAGE_BZIP2_COMMAND="pbzip2"
PORTAGE_COMPRESS_EXCLUDE_SUFFIXES="css gif htm[l]? jp[e]?g js pdf png"
PORTAGE_CONFIGROOT="/build/nyan_big/"
PORTAGE_DEBUG="0"
PORTAGE_DEPCACHEDIR="/var/cache/edb/dep"
PORTAGE_ELOG_CLASSES="log warn error"
PORTAGE_ELOG_MAILFROM="portage@localhost"
PORTAGE_ELOG_MAILSUBJECT="[portage] ebuild log for ${PACKAGE} on ${HOST}"
PORTAGE_ELOG_MAILURI="root"
PORTAGE_ELOG_SYSTEM="save_summary:log,warn,error,qa echo"
PORTAGE_FETCH_CHECKSUM_TRY_MIRRORS="5"
PORTAGE_FETCH_RESUME_MIN_SIZE="350K"
PORTAGE_GID="250"
PORTAGE_GPG_SIGNING_COMMAND="gpg --sign --digest-algo SHA256 --clearsign --yes --default-key "${PORTAGE_GPG_KEY}" --homedir "${PORTAGE_GPG_DIR}" "${FILE}""
PORTAGE_INST_GID="0"
PORTAGE_INST_UID="0"
PORTAGE_INTERNAL_CALLER="1"
PORTAGE_OVERRIDE_EPREFIX=""
PORTAGE_PYM_PATH="/usr/lib64/python2.7/site-packages"
PORTAGE_PYTHONPATH="/usr/lib64/python2.7/site-packages"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_RSYNC_RETRIES="-1"
PORTAGE_SYNC_STALE="30"
PORTAGE_TMPDIR="/build/nyan_big/tmp/"
PORTAGE_USERNAME="chrome-bot"
PORTAGE_VERBOSE="1"
PORTAGE_WORKDIR_MODE="0755"
PORTAGE_XATTR_EXCLUDE="btrfs.* security.evm security.ima        security.selinux system.nfs4_acl"
PORT_LOGDIR="/build/nyan_big/tmp/portage/logs/"
PORT_LOGDIR_CLEAN="find "${PORT_LOGDIR}" -type f ! -name "summary.log*" -mtime +7 -delete"
PREFLIGHT_BINHOST="gs://chromeos-prebuilt/board/nyan_big/paladin-R70-10996.0.0-rc1/packages/"
PROFILE_ONLY_VARIABLES="ARCH ELIBC IUSE_IMPLICIT KERNEL USERLAND USE_EXPAND_IMPLICIT USE_EXPAND_UNPREFIXED USE_EXPAND_VALUES_ARCH USE_EXPAND_VALUES_ELIBC USE_EXPAND_VALUES_KERNEL USE_EXPAND_VALUES_USERLAND"
PWD="/home/chrome-bot/trunk/src/scripts"
PYTHONDONTWRITEBYTECODE="1"
PYTHON_SINGLE_TARGET="python2_7"
PYTHON_TARGETS="python2_7"
RESUMECOMMAND="curl --ipv4 -y 30 -f -C - --retry 9 -L --output ${DISTDIR}/${FILE} ${URI}"
RESUMECOMMAND_AB="/mnt/host/source/chromite/bin/abutil fetch "${URI}" "${DISTDIR}/${FILE}""
RESUMECOMMAND_GS="bash -c 'BOTO_CONFIG=/mnt/host/source/src/private-overlays/chromeos-overlay/googlestorage_account.boto /mnt/host/source/chromite/bin/gs_fetch_binpkg "${URI}" "${DISTDIR}/${FILE}"'"
RESUMECOMMAND_RSYNC="rsync -avP "${URI}" "${DISTDIR}/${FILE}""
RESUMECOMMAND_SSH="bash -c "x=\${2#ssh://} ; host=\${x%%/*} ; port=\${host##*:} ; host=\${host%:*} ; [[ \${host} = \${port} ]] && port=22 ; exec rsync --rsh=\"ssh -p\${port} \${3}\" -avP \"\${host}:/\${x#*/}\" \"\$1\"" rsync "${DISTDIR}/${FILE}" "${URI}" "${PORTAGE_SSH_OPTS}""
ROOT="/build/nyan_big/"
ROOTPATH="/opt/bin"
RPMDIR="/usr/portage/rpm"
RUBY_TARGETS="ruby19 ruby18"
SHELL="/bin/bash"
SHLVL="1"
SSH_AUTH_SOCK="/tmp/ssh-auth-sock"
STRIP_MASK="*.nexe /firmware/*.elf"
SUDO_COMMAND="/usr/bin/emerge --root-deps --info -vvv"
SUDO_GID="1019"
SUDO_UID="1018"
SUDO_USER="chrome-bot"
SYMLINK_LIB="no"
SYSROOT="/build/nyan_big"
TERM="xterm-256color"
TTY_CONSOLE=""
UNINSTALL_IGNORE=""
USE="acpi acpi_ac allow_consumer_kiosk arm bindist bluetooth builtin_fw_t124_xusb bzip2 cairo cellular compupdates cros-debug cros_ec cros_p2p crosfonts crypt cups cxx device_tree disable_flash_hw_video_decode dri evdev_gestures frecon hardfp iconv ime internal ipv6 kernel-3_10 mbim memd mist modules mosys_eventlog mudflap ncurses neon nptl nptlonly nyan_big opengles ozone pam pinweaver postscript readline ssl sysfs tegra tegra-ldk tegra124-venice2_dtb tpm tpm_slb9645 transparent_hugepage unicode usb usbguard variant_build vlan wireless38 xattr xkbcommon xorg zlib" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" BOARD_USE="nyan_big" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" EC_FIRMWARE="big" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="evdev cmt" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en" LINUX_FIRMWARE="nvidia-xusb" OFFICE_IMPLEMENTATION="libreoffice" OZONE_PLATFORM="gbm default_gbm" OZONE_PLATFORM_DEFAULT="gbm" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7" RUBY_TARGETS="ruby19 ruby18" USERLAND="GNU" VIDEO_CARDS="tegra" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
USER="root"
USERLAND="GNU"
USERNAME="root"
USE_EXPAND="ABI_MIPS ABI_PPC ABI_S390 ABI_X86 ALSA_CARDS APACHE2_MODULES APACHE2_MPMS BOARD_USE CALLIGRA_FEATURES CAMERAS COLLECTD_PLUGINS CPU_FLAGS_X86 CROSSCOMPILE_OPTS CROS_WORKON_TREE CURL_SSL DRACUT_MODULES DVB_CARDS EC_FIRMWARE EC_FIRMWARE_EXTRA ELIBC ENLIGHTENMENT_MODULES FCDSL_CARDS FFTOOLS FOO2ZJS_DEVICES FRITZCAPI_CARDS GPSD_PROTOCOLS GRUB_PLATFORMS INPUT_DEVICES KERNEL LCD_DEVICES LIBREOFFICE_EXTENSIONS LINGUAS LINUX_FIRMWARE LIRC_DEVICES LLVM_TARGETS MONKEYD_PLUGINS NETBEANS_MODULES NGINX_MODULES_HTTP NGINX_MODULES_MAIL NGINX_MODULES_STREAM OFED_DRIVERS OFFICE_IMPLEMENTATION OPENMPI_FABRICS OPENMPI_OFED_FEATURES OPENMPI_RM OZONE_PLATFORM OZONE_PLATFORM_DEFAULT PHP_TARGETS PYTHON_SINGLE_TARGET PYTHON_TARGETS QEMU_SOFTMMU_TARGETS QEMU_USER_TARGETS ROS_MESSAGES RUBY_TARGETS SANE_BACKENDS TESTS TTY_CONSOLE USERLAND UWSGI_PLUGINS U_BOOT_CONFIG_USE U_BOOT_FDT_USE VIDEO_CARDS VOICEMAIL_STORAGE XFCE_PLUGINS XTABLES_ADDONS"
USE_EXPAND_HIDDEN="ABI_MIPS ABI_PPC ABI_S390 ABI_X86 CPU_FLAGS_X86 CROSSCOMPILE_OPTS CROS_WORKON_TREE ELIBC KERNEL USERLAND"
USE_EXPAND_IMPLICIT="ARCH ELIBC KERNEL USERLAND"
USE_EXPAND_UNPREFIXED="ARCH"
USE_EXPAND_VALUES_ARCH="alpha amd64 amd64-fbsd amd64-linux arm arm-linux arm64 hppa hppa-hpux ia64 ia64-hpux ia64-linux m68k m68k-mint mips nios2 ppc ppc64 ppc64-linux ppc-aix ppc-macos ppc-openbsd riscv s390 sh sparc sparc64-freebsd sparc64-solaris sparc-fbsd sparc-solaris x64-cygwin x64-freebsd x64-macos x64-openbsd x64-solaris x86 x86-cygwin x86-fbsd x86-freebsd x86-interix x86-linux x86-macos x86-netbsd x86-openbsd x86-solaris x86-winnt"
USE_EXPAND_VALUES_ELIBC="AIX Cygwin Darwin DragonFly FreeBSD glibc HPUX Interix mintlib musl NetBSD OpenBSD SunOS uclibc Winnt"
USE_EXPAND_VALUES_KERNEL="AIX Darwin FreeBSD freemint HPUX linux NetBSD OpenBSD SunOS Winnt"
USE_EXPAND_VALUES_USERLAND="BSD GNU"
USE_ORDER="env:pkg:conf:defaults:pkginternal:repo:env.d"
VIDEO_CARDS="tegra"
XAUTHORITY="/home/chronos/.Xauthority"
XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"

How are these getting set:
PORTAGE_PYM_PATH
PORTAGE_PYTHONPATH

I've been trying to dig into this for a good bit this am trying to trace this.

I assume these variables affect the local python instance used within portage, whereas, this affects the DUT.

I was trying to trace anything that affects the python config on the DUT.  Starting with the overlays for installing python for autotest, corresponding tests and the like.

i did some local builds yesterday and they seemed to be installing into the right place

the main libdir is looked up via the get_libdir helper in the multilib.eclass.  that data is based on two env vars defined in the profile:
- ABI
- LIBDIR_${ABI}
if those are defined, then "lib" is the fallback.

env vars in the ebuild env can be inspected using:
  portageq envvar VAR
  portageq-$BOARD envvar VAR

for the sdk, ABI=amd64 and LIBDIR_amd64=lib64 which is how all sdk paths end up under /usr/lib64.  for arm32 userland (all our current arm boards), ABI=arm and LIBDIR_arm=lib which is how all arm board paths end up under /usr/lib.

python is tricky when cross-compiling because it uses `python` as a build tool for things like setup.py, but then it's also responsible for building+compiling+installing in the same `python` environment.  so when python asks questions like "what compiler do i use" or "what filesystem paths do i install under", it tends to use distutils.sysconfig to get that info.  that's also what Gentoo eclasses build on top of.

obviously the sdk's distutils.sysconfig doesn't work for us.  that's why we patch it to respect the $SYSROOT env var.  you can see this:
$ python -c 'import distutils.sysconfig; print(distutils.sysconfig.get_python_lib())'/usr/lib64/python2.7/site-packages
$ SYSROOT=/build/nyan_big python -c 'import distutils.sysconfig; print(distutils.sysconfig.get_python_lib())'
/build/nyan_big/usr/lib/python2.7/site-packages

looking at python-2.7.10-cross-distutils.patch, there does seem to be a possible bug in the search hack we have.  if something created a lib64 symlink, or some other semi-valid path, that query would break.
$ sudo ln -s lib /build/nyan_big/usr/lib64
$ SYSROOT=/build/nyan_big python -c 'import distutils.sysconfig; print(distutils.sysconfig.get_python_lib())'
/build/nyan_big/usr/lib64/python2.7/site-packages

now, nothing *should* be creating /usr/lib64 for arm boards, but we've seen packages screw up cross-compiling before.  so if something did break that, it'd have the bad side-effect of screwing up python packages installed after it.

Following up on #37, on an ARM DUT, I see 

localhost ~ # ls -a /usr/local/lib64/
.  ..  python2.7


So if some other package did create /usr/local/lib64, it did not actually install anything in there.

Update: vapier@ is going to set up some hooks so that the build will fail when anything is placed in /usr/lib64, which will help narrow down the cause of this issue.

In the meantime, bleung@ and I will be going through CLs again searching for a culprit. Hopefully all the information from #37 will help us identify potential bad CLs

samba is wrongly using /usr/lib64.  i've filed issue 877675 to track that.

the reason it's just starting to show up is because it's only started to be pulled in recently for arm targets:
  https://chromium-review.googlesource.com/1170029
which enables smbprovider on arm starting with chrome-70.0.3524+.

it passed the CQ because it would only kick in on the next Chrome PFQ run.  that run completed on Aug 16:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/0be48436f63f369714459a2a677d825b8399da31

since the bad paths wouldn't be used by python packages until after samba is installed, we get into a fun state:
- if the python package is installed before samba, it works great
- so when we build in parallel, if we build all python packages quickly, it works
- with the overly eager revdep rebuild logic ( issue 864309 ), we have incremental builders which already have samba installed, and all the rebuilt python packages transition from "ok" to "bad"

my CL https://chromium-review.googlesource.com/1188852 is working in that it caught samba right away.  but we can't land it until samba is fixed.

this means our bots are flaky dead as long as samba is broken.  once samba is fixed, and we kick the bots, they should all be fixed.

i've filed issue 877680 to try and make python more robust.

actually i have a hack for samba to put the fire out which we can revert once the samba build logic is fixed properly (it shouldn't be even building these things let along installing them into the wrong path):
  https://chromium-review.googlesource.com/1189002

to clarify comment #41, it isn't just that samba is using /usr/lib64, it's also installing a python stubfile which is what is confusing the distutils.sysconfig logic.  so comment #37 seems to be the core issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/2e5578dc287121792b0c83b64b42851de47f31d8

commit 2e5578dc287121792b0c83b64b42851de47f31d8
Author: Mike Frysinger <vapier@chromium.org>
Date: Sat Aug 25 15:18:09 2018

samba: hack out python/perl when disabled

The samba build system is using the wrong libdir when installing
perl & python files, and it's installing these files even when we
have them disabled.  They're in turn breaking other python packages
due to the conflicting libdir usage.

Hack the files out of the install until we can figure out why the
build is broken.

BUG= chromium:876634 , chromium:877675
TEST=`emerge-nyan_big samba` doesn't install any perl or python files

Change-Id: I1f2488e96a47edf93a961ec30b1285ff1c161a2b
Reviewed-on: https://chromium-review.googlesource.com/1189002
Reviewed-by: Jason Clinton <jclinton@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

[rename] https://crrev.com/2e5578dc287121792b0c83b64b42851de47f31d8/net-fs/samba/samba-4.8.0-r8.ebuild
[modify] https://crrev.com/2e5578dc287121792b0c83b64b42851de47f31d8/net-fs/samba/samba-4.8.0.ebuild

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/crosutils/+/ef121e275645abc3e62ff575d0a5f84b2913a697

commit ef121e275645abc3e62ff575d0a5f84b2913a697
Author: Mike Frysinger <vapier@chromium.org>
Date: Sun Aug 26 15:44:53 2018

install hooks: add a sanity check for bad multilib usage

We don't want lib64 being used on 32-bit systems like x86 or arm.
If it leaks in, we should abort early on rather than much later like
during test time.  So add a sanity check for these to reject lib64.

BUG= chromium:876634 
TEST=precq passes
TEST=modifying an ebuild to install into /usr/lib64 and building on arm fails

Change-Id: I10d22c304080dde90626bff874d44425b3db0e7c
Reviewed-on: https://chromium-review.googlesource.com/1188852
Reviewed-by: Benson Leung <bleung@chromium.org>
Reviewed-by: C Shapiro <shapiroc@google.com>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

[add] https://crrev.com/ef121e275645abc3e62ff575d0a5f84b2913a697/hooks/install/multilib-sanity.sh

we're at the point now where the bots need to be reset like they were before.  i'm not sure how that went down, so whoever did it before should do it again :).

I ran
./ccompute ri cros-beefy{80,41,60,63,27,70,32,58,69}-c2
which should re-image the same as Don's plus bob-paladin.

nyan_kitty-paladin failed, running on cros-beefy82-c2: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?builderName=nyan_kitty-paladin&buildNumber=5937

Same for builds on cros-beefy59-c2 and cros-beefy76-c2.

That's a different failure from this bug since it failed in BuildPackages and the symptoms of this bug are that it fails in HWTest.

Downgrading for follow-up.

#48: This is the fail-fast vapier added above at #44; same underlying issue.

jclinton:

These are related because Mike landed his lib64 sanity check in #44 to make the process fail in build packages instead of HWTest.

Okay, so the diagnosis in #41 was that this was only Samba. Looking at the build output, it seems that:
	dev-python/pyserial-2.7-r1
	dev-python/packaging-16.0
	dev-python/pexpect-3.3
	dev-python/ws4py-0.3.2
	dev-python/six-1.10.0
	dev-python/pyparsing-2.2.0-r1
	dev-util/scons-2.3.0
	dev-python/python-uinput-0.11.2-r1
	chromeos-base/touch_firmware_test-0.0.1-r106
	dev-python/contextlib2-0.4.0
	dev-python/setproctitle-1.1.8
	dev-python/imaging-1.1.7-r5
	dev-util/gdbus-codegen-2.52.3
	dev-python/pyyaml-3.11
	chromeos-base/touchbot-1.0-r128
	dev-python/dpkt-1.8-r2
	dev-python/jsonschema-2.3.0

are all failing now for the same reason. vapier@ just landed a fix (https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/1189003) so that lib64 shouldn't be coming out of these packages during environment probe. I believe that the next run should be fixed.

#53: We had to reimage affected machines to fix them before; are you thinking that https://crrev.com/c/1189003 will fix things without reimaging the above 3 failing builders?

Never mind re: the 3 specific failing builders; it looks like more are failing now. Still, do we think the above fix does not require reimaging?

Yes, because the bad state that became the steady-state after a successful build is worked-around by CL 1189003: the environment probe should no longer find the bad lib64 directory.

we shouldn't need to reimage anymore once i land some CLs to clean things up gracefully.  we have poisoned binhosts (arm-generic) so reimaging wouldn't stop the poison from flowing back out.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/portage-stable/+/3f10b4d254c91e683716b036cec734bfecf29328

commit 3f10b4d254c91e683716b036cec734bfecf29328
Author: Mike Frysinger <vapier@chromium.org>
Date: Mon Aug 27 20:41:01 2018

force uprev of python packages

For packages that might have a python/arm build with lib64 paths,
force a rebuild so the binpkgs get updated and bots/devs get them.

BUG= chromium:876634 
TEST=precq passes

Change-Id: I7e66a4a82590714dd8374704263d9725ece87902
Reviewed-on: https://chromium-review.googlesource.com/1191602
Reviewed-by: Jason Clinton <jclinton@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/idna/idna-2.5-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/net-ftp/pybootd/pybootd-1.5.0_pre20110524131526-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/cffi/cffi-1.10.0-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/setproctitle/setproctitle-1.1.8-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/urllib3/urllib3-1.22-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/ply/ply-3.4-r2.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/simplejson/simplejson-3.6.5-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/pyusb/pyusb-1.0.0_beta1-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/psutil/psutil-2.2.1-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/pyyaml/pyyaml-3.11-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/cryptography/cryptography-1.9-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/PySocks/PySocks-1.6.7-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/jsonrpclib/jsonrpclib-0_pre20110820-r2.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/setuptools/setuptools-19.4-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/httplib2/httplib2-0.10.3-r2.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/protobuf-python/protobuf-python-3.3.0-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/ipaddress/ipaddress-1.0.16-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/packaging/packaging-16.0-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/requests/requests-2.18.2-r2.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/pytest/pytest-2.3.5-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-util/gdbus-codegen/gdbus-codegen-2.52.3-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/numpy/numpy-1.9.2-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/ws4py/ws4py-0.3.2-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/jsonschema/jsonschema-2.3.0-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/imaging/imaging-1.1.7-r6.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/netifaces/netifaces-0.10.5-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/pyopenssl/pyopenssl-16.2.0-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/six/six-1.10.0-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/pyudev/pyudev-0.16.1-r2.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/pyparsing/pyparsing-2.2.0-r2.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/m2crypto/m2crypto-0.22.3-r5.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/selenium/selenium-3.0.2-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/contextlib2/contextlib2-0.4.0-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/chardet/chardet-3.0.4-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/asn1crypto/asn1crypto-0.22.0-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/py/py-1.4.20-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/pycparser/pycparser-2.10-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/pyserial/pyserial-2.7-r2.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/pexpect/pexpect-3.3-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/lxml/lxml-3.4.4-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/certifi/certifi-2017.4.17-r2.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-util/scons/scons-2.3.0-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/cherrypy/cherrypy-3.2.2-r2.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/pyxattr/pyxattr-0.5.5-r1.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/qrcode/qrcode-5.1-r2.ebuild
[add] https://crrev.com/3f10b4d254c91e683716b036cec734bfecf29328/dev-python/enum34/enum34-1.1.6-r1.ebuild

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/6c9e39cc34d16b67265703857314491ebcab6c19

commit 6c9e39cc34d16b67265703857314491ebcab6c19
Author: Mike Frysinger <vapier@chromium.org>
Date: Mon Aug 27 20:41:08 2018

force uprev of python packages

For packages that might have a python/arm build with lib64 paths,
force a rebuild so the binpkgs get updated and bots/devs get them.

BUG= chromium:876634 
TEST=precq passes

Change-Id: I7e66a4a82590714dd8374704263d9725ece87902
Reviewed-on: https://chromium-review.googlesource.com/1191163
Reviewed-by: Sean Kau <skau@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

[rename] https://crrev.com/6c9e39cc34d16b67265703857314491ebcab6c19/dev-python/mkvparse/mkvparse-1.1.0-r2.ebuild
[rename] https://crrev.com/6c9e39cc34d16b67265703857314491ebcab6c19/dev-util/mem/mem-0.0.1-r3.ebuild
[rename] https://crrev.com/6c9e39cc34d16b67265703857314491ebcab6c19/dev-util/hdctools/hdctools-0.0.1-r837.ebuild
[rename] https://crrev.com/6c9e39cc34d16b67265703857314491ebcab6c19/dev-python/gnupg/gnupg-2.3.0-r2.ebuild
[rename] https://crrev.com/6c9e39cc34d16b67265703857314491ebcab6c19/dev-python/python-evdev/python-evdev-0.6.0-r2.ebuild
[rename] https://crrev.com/6c9e39cc34d16b67265703857314491ebcab6c19/chromeos-base/touchbot/touchbot-1.0-r129.ebuild
[rename] https://crrev.com/6c9e39cc34d16b67265703857314491ebcab6c19/dev-python/btsocket/btsocket-1.0-r16.ebuild
[rename] https://crrev.com/6c9e39cc34d16b67265703857314491ebcab6c19/dev-python/dpkt/dpkt-1.8-r3.ebuild
[rename] https://crrev.com/6c9e39cc34d16b67265703857314491ebcab6c19/sys-apps/portage/portage-2.2.28-r22.ebuild
[rename] https://crrev.com/6c9e39cc34d16b67265703857314491ebcab6c19/dev-python/python-uinput/python-uinput-0.11.2-r2.ebuild
[rename] https://crrev.com/6c9e39cc34d16b67265703857314491ebcab6c19/chromeos-base/touch_firmware_test/touch_firmware_test-0.0.1-r107.ebuild
[add] https://crrev.com/6c9e39cc34d16b67265703857314491ebcab6c19/chromeos-base/pywalt/pywalt-0.1.8-r1.ebuild

here's the shell script i hacked together to detect broken packages

Deleted: check.sh 1.4 KB

check.sh 1.4 KB View Download

current CQ run seems to be cycling green, so going to optimistically close this out

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/crosutils/+/13d2973ccebd6a7e7387b7714a07c70000dc7a09

commit 13d2973ccebd6a7e7387b7714a07c70000dc7a09
Author: Mike Frysinger <vapier@chromium.org>
Date: Wed Aug 29 18:16:00 2018

chroot_version_hooks.d: clear our broken arm python lib64 usage

To make sure boards rebuild/reinstall their broken python packages,
forcibly unmerge any installed broken packages that are using lib64.
We're revbumping all the python packages to make sure they get new
versions before cleaning them out.

BUG= chromium:876634 
TEST=ran locally and it cleared out a bunch of broken packages

Change-Id: I9222ea3e5c28d93fcd07c042eee0a216ee88df6e
Reviewed-on: https://chromium-review.googlesource.com/1191302
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Sean Kau <skau@chromium.org>

[add] https://crrev.com/13d2973ccebd6a7e7387b7714a07c70000dc7a09/chroot_version_hooks.d/156_clean_arm_python_lib64