ERROR   :     libpng12.so.0 => None

guessing this is an old build.  you can manually fix by doing:
emerge-guado_moblab dev-python/matplotlib

if you want to fix it, you'll need to update dev-python/matplotlib in the portage-stable tree so the libpng dep uses subslots.

I just did repo sync and build_packages why is this manifesting then?

sounds like you're just running into normal behavior of how parallel_emerge works.  it tries its best to trigger rebuilds when libs update, but it's never been perfect, just "best effort".  if you did a partial update at some point (like just libpng), then any packages that weren't updated then wouldn't get caught in the future.  subslots gets us perfect behavior.

problem went away after delete the chroot and recreate.

This is now hitting chromite:
/mnt/host/source/src/scripts/build_library/test_image_content.sh: line 41: echo: write error: Broken pipe
ERROR   : test_elf_deps: Failed dependency check
^[^[^[ERROR   : Package: chromeos-base/chromite-0.0.2-r2863
ERROR   : /mnt/host/source/src/build/images/guado_moblab/R59-9373.0.2017_03_16_1414-a1/rootfs/usr/lib64/python2.7/site-packages/chromite/venv/.venv/lib/python2.7/site-packages/numpy/.libs/libopenblasp-r0-39a31c03.2.18.so (interpreter => None)
ERROR   :     libm.so.6 => /mnt/host/source/src/build/images/guado_moblab/R59-9373.0.2017_03_16_1414-a1/rootfs/lib64/libm.so.6
ERROR   :         ld-linux-x86-64.so.2 => /mnt/host/source/src/build/images/guado_moblab/R59-9373.0.2017_03_16_1414-a1/rootfs/lib64/ld-linux-x86-64.so.2
ERROR   :     libpthread.so.0 => /mnt/host/source/src/build/images/guado_moblab/R59-9373.0.2017_03_16_1414-a1/rootfs/lib64/libpthread.so.0
ERROR   :     libgfortran-ed201abd.so.3.0.0 => None
ERROR   :     libc.so.6 => /mnt/host/source/src/build/images/guado_moblab/R59-9373.0.2017_03_16_1414-a1/rootfs/lib64/libc.so.6



I blew away /build/guado_moblab and it still happened.

What is this "dependency check" thing? Looks like it is trying to inspect stuff in the guado image and determine if it has bad .so library deps.

And in particular, looks like it is triggering off of some library that virtualenv is dropping in the guado moblab rootfs image?

Deleting my chroot and recreating did not help...

Chromite is part of the guado_moblab image b/c autotest depends on it, if that matters. Do normal test images have chromite installed?

No, don't think they do. Just moblab.

Although the symptom is similar, this is unrelated to the OP.

sbasi@:
workaround: 
cd chromite/
rm -rf venv/.venv
rm -rf venv/.full_venv

And rebuild.

If this works:

- We are now adding dependencies for chromite via virtualenv
- emerge is not aware of these dependencies.
- When emerging chromite, we just copy over the whole directory (including anything installed under venv/.venv or venv/.full_venv by virtualenv (http://shortn/_bKee0ZAqJ7)
- Now, when test_elf_deps checks all dirs, it finds the virtualenv installs, which may have further dependencies (C libs) that emerge didn't know about. Oops.

- In this case, virtualenv installed numpy which depends on libgfortran-ed201abd.so.3.0.0, which chromite doesn't pull in via emerge.
- Note that this dependency isn't even present in the infra_virtualenv repo used by virtualenv for its dependencies. This is just a system library that happens to be a part of goobuntu but not chromeos.

There are two cases where this is a real problem:

- moblab: emerges chromite. As we put more stuff in the virtualenv, and start depending on this stuff from autotest, moblab will hit real problems, because we'll soon end up in a situation where an autotest -> chromite -> virtualenv call needs a dependency not installed on moblab.
  - I think right now autotest calls to chromite don't go via virtualenv at all, which means we may hit python deps problems even earlier.

- chromite unittests: I'm assuming that the chromite unittests will (eventually?) run from the virtualenv: we run unittests inside the chroot -- same as moblab. We need some way to pull in libgfortran and friends with chromite.

If it's a separate issue, we should make a separate bug for it.

It sounds like we can prevent the issue by building all of our Python wheels on an Ubuntu virtualenv to ensure no Goobuntu deps get compiled in.

yes, image test makes sure that a CrOS image doesn't have broken library dependencies.  and it's correctly flagging a broken installed binary:

ERROR   : /mnt/host/source/src/build/images/guado_moblab/R59-9373.0.2017_03_16_1414-a1/rootfs/usr/lib64/python2.7/site-packages/chromite/venv/.venv/lib/python2.7/site-packages/numpy/.libs/libopenblasp-r0-39a31c03.2.18.so (interpreter => None)
ERROR   :     libgfortran-ed201abd.so.3.0.0 => None

which makes me think the ebuild is wrongly installing venv stuff that was created in the host distro (Ubuntu).  the chromite ebuild probably needs changing to not bundle+install venv stuff.

also, it somewhat implies that things built for Ubuntu are bleeding into the sdk env which would be bad.

Okay bad news... but my guess is our prebuilts are polluted.

I followed Pprabhu's suggestions in CM11, hit the same error.

Then I `cros_workon-guado_moblab start chromite` and the problem went away (with the cleaned up localc chromite).

Now I'm lost to why the builders aren't all red.

On the virtualenv errors:

1. the chromite ebuild should probably not copy things in gitignore, like .venv.
2. I'll be rebuilding our virtualenv packages in a clean Ubuntu environment, that should help prevent potential issues with missing shared libraries.

at this point, it would probably be worth looking into adding a proper setup.py file to the chromite repo and let the ebuild process that like any other python package.  the chromite ebuild is a bit dumb because we've forced it to be so.

Michael/Keith, if you guys start hitting this you may want to follow vapier's suggestion to fix this. I don't have cycles right now.

So far I was not able to reproduce this locally. virtualenv is something new than when this issue was filed. I will keep an eye on this.