Example build: https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Ftryserver.chromium.linux%2Fchromium_presubmit%2F307101%2F%2B%2Frecipes%2Fsteps%2Fpresubmit%2F0%2Fstdout

I logged onto the bot in question, ran:
$ /b/build/slave/linux/build/src/infra/.recipe_deps/recipe_engine/ENV/bin/python -c 'import sys; print sys.prefix'
/b/build/slave/linux/build/src/infra/.recipe_deps/recipe_engine/ENV

This is at direct odds with the error, which complains that Python thinks the prefix is "/b/build/slave/linux/build/src/infra/.recipe_deps/recipe_engine".

Also, "recipes_test.py" works!

$ ./recipes.py --use-bootstrap simulation_test
...
Ran 52 tests in 5.297s



unittests/recipes_test.py (1.31s) failed
fatal: Needed a single revision
From https://chromium.googlesource.com/external/github.com/luci/recipes-py
   672570b..ae33ef8  master     -> origin/master
Activating environment: 'ENV'
  Removing old environment: None
  Building new environment
Traceback (most recent call last):
  File "/usr/lib/python2.7/site.py", line 562, in <module>
    main()
  File "/usr/lib/python2.7/site.py", line 544, in main
    known_paths = addusersitepackages(known_paths)
  File "/usr/lib/python2.7/site.py", line 271, in addusersitepackages
    user_site = getusersitepackages()
  File "/usr/lib/python2.7/site.py", line 246, in getusersitepackages
    user_base = getuserbase() # this will also set USER_BASE
  File "/usr/lib/python2.7/site.py", line 236, in getuserbase
    USER_BASE = get_config_var('userbase')
  File "/usr/lib/python2.7/sysconfig.py", line 577, in get_config_var
    return get_config_vars().get(name)
  File "/usr/lib/python2.7/sysconfig.py", line 476, in get_config_vars
    _init_posix(_CONFIG_VARS)
  File "/usr/lib/python2.7/sysconfig.py", line 355, in _init_posix
    raise IOError(msg)
IOError: invalid Python installation: unable to open /b/build/slave/linux/build/src/infra/.recipe_deps/recipe_engine/ENV/local/include/python2.7/pyconfig.h (No such file or directory)
ERROR: The executable ENV/bin/python is not functioning
ERROR: It thinks sys.prefix is u'/b/build/slave/linux/build/src/infra/.recipe_deps/recipe_engine' (should be u'/b/build/slave/linux/build/src/infra/.recipe_deps/recipe_engine/ENV')
ERROR: virtualenv is not compatible with this system or executable
Traceback (most recent call last):
  File "/b/build/slave/linux/build/src/infra/.recipe_deps/recipe_engine/recipes.py", line 646, in <module>
    ret = main()
  File "/b/build/slave/linux/build/src/infra/.recipe_deps/recipe_engine/recipes.py", line 568, in main
    cwd=os.path.dirname(os.path.realpath(__file__)))
  File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['/usr/bin/python', 'bootstrap/bootstrap.py', '--deps-file', 'bootstrap/deps.pyl', 'ENV']' returned non-zero exit status 100
Uncaught exception (CalledProcessError): Command '['/usr/bin/python', 'bootstrap/bootstrap.py', '--deps-file', 'bootstrap/deps.pyl', 'ENV']' returned non-zero exit status 100
Traceback (most recent call last):
  File "unittests/recipes_test.py", line 19, in <module>
    recipes_py('simulation_test')
  File "unittests/recipes_test.py", line 16, in recipes_py
    [os.path.join(INFRA_DIR, 'recipes.py'), '--use-bootstrap'] + list(args))
  File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/b/build/slave/linux/build/src/infra/recipes.py', '--use-bootstrap', 'simulation_test']' returned non-zero exit status 1


While experimenting on a bot, I also received:

ERROR:root:Error while importing testing libraries. You may be missing the pip package "coverage". Install it, or use the --use-bootstrap command line argument when calling into the recipe engine, which will install it for you.
Traceback (most recent call last):
  File "/b/build/slave/linux/build/src/infra/.recipe_deps/recipe_engine/recipes.py", line 646, in <module>
    ret = main()
  File "/b/build/slave/linux/build/src/infra/.recipe_deps/recipe_engine/recipes.py", line 608, in main
    return simulation_test(package_deps, args, op_args)
  File "/b/build/slave/linux/build/src/infra/.recipe_deps/recipe_engine/recipes.py", line 42, in simulation_test
    from recipe_engine import simulation_test
  File "/b/build/slave/linux/build/src/infra/.recipe_deps/recipe_engine/recipe_engine/simulation_test.py", line 24, in <module>
    import expect_tests
  File "/b/build/slave/linux/build/src/infra/.recipe_deps/recipe_engine/recipe_engine/third_party/expect_tests/__init__.py", line 5, in <module>
    from .main import main
  File "/b/build/slave/linux/build/src/infra/.recipe_deps/recipe_engine/recipe_engine/third_party/expect_tests/main.py", line 6, in <module>
    import multiprocessing
  File "/usr/lib/python2.7/multiprocessing/__init__.py", line 84, in <module>
    import _multiprocessing
ImportError: No module named _multiprocessing


------

Theory: bootstrap environment is broken. Either it was never working in the first place, or we're hitting issues with multiple processes setting up the same environment.

I remember at some point we had a command-line argument to tell the recipe engine to use a custom ".recipe_deps" directory, but I don't see it anymore. Where did that go? WDYT about enabling it for all of our test presubmits?