This problem will only affect boards with Android containers.

The actual error message looks like this:

01:04:42: INFO: RunCommand: /b/c/cbuild/repository/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cp -v -- gs://chromeos-arc-images/builds/git_nyc-mr1-arc-m62-linux-cheets_x86_64-user/4319723/cheets_x86_64-symbols-4319723.zip /b/c/cbuild/repository/buildbot_archive/cyan-release/R62-9901.8.0/android-symbols.zip
01:04:45: WARNING: GS_ERROR: Copying gs://chromeos-arc-images/builds/git_nyc-mr1-arc-m62-linux-cheets_x86_64-user/4319723/cheets_x86_64-symbols-4319723.zip...
==> NOTE: You are downloading one or more large file(s), which would
run significantly faster if you enabled sliced object uploads. This
feature is enabled by default but requires that compiled crcmod be
installed (see "gsutil help crcmod").
CommandException: 
Downloading this composite object requires integrity checking with CRC32c,
but your crcmod installation isn't using the module's C extension, so the
hash computation will likely throttle download performance. For help
installing the extension, please see:
  $ gsutil help crcmod
To download regardless of crcmod performance or to skip slow integrity
checks, see the "check_hashes" option in your boto config file.
NOTE: It is strongly recommended that you not disable integrity checks. Doing so
could allow data corruption to go undetected during uploading/downloading.
 

We reimaged our builders yesterday. I suspect that the crcmod value mentioned in the gsutil error messages above was either removed from the new builder image, or updated to a newer version that our old version of gsutil (4.19) is not able to use.

The TOT builders use the same builder image, but are now using a newer version of gsutil (4.27, I believe) which may either not need the external binary library, or may use a newer version matching the newer builder image.

Should be ok to upgrade gsutil to 4.27 for branches after version 9854.0.0: 

The caller removing CL is merged in 9854.0.0:
https://crosland.corp.google.com/log/9853.0.0..9854.0.0

Further checking....

The pinned version of gsutil INSIDE the chroot has access to crcmod, but on the builder, the pinned version of gsutil OUTSIDE the chroot does not.

Checking further.... on our Golo builders (which were not reimaged yesterday):

chrome-bot@build12-m2:(Linux 14.04):/b/c/cbuild/repository/.cache/common/gsutil_4.19.tar.gz/gsutil$ ./gsutil version -l
gsutil version: 4.19
checksum: 67da5fbdef140f1663fbb11e96bb9f90 (OK)


That shows that the crcmodule is available and working for gsutil 4.19 on most builders, but not the newly imaged ones. I believe that that's the root cause.

1) Our two options are getting that binary crcmod installed on the builders (which may be a huge versioning issue)

2) Rolling the release builders back to the older builder image.

3) Updating the version of gsutil used on the branches.


I have a preference for 3.

Checking the versions on all active branches to see if that's feasible based on xixuan's comment:

release-R60-9592.B
release-R61-9765.B
release-R62-9901.B

That shows we can only merge back the new gsutil to R62, which isn't far enough.

So... I'm going to suggest that we identify and reimage the R61 and R60 builders to the older builder image.

That's an ugly hack that will allow builds to proceed, but which will mean we can't ssh into into of those builders (hopefully, not needed).

Richard, go the the ccompute git repo, check out to an older revision, then run the ccompute command.

You can identify the build slaves to reimage by looking at the release waterfall buildslaves tab.

We need to do that for all R60, and R61 slaves, but no R62 slaves.

I created a list of the R60/R61 builders, and proceeded to delete
them.  When I went to recreate, I got failures like this:

cros-beefy185-c2 O> Adding instance cros-beefy185-c2...
cros-beefy185-c2 O> Running: /usr/local/google/home/jrbarnette/tools/google-cloud-sdk/bin/gcloud compute instances create cros-beefy185-c2 --machine-type n1-highmem-32 --zone us-east1-d --project chromeos-bot --image chromeos-trusty-17050300-d40e82cb8cf --image-project chromeos-bot --boot-disk-size 2048GB --boot-disk-type pd-standard --address 104.196.64.246 --scopes https://www.googleapis.com/auth/devstorage.full_control,https://www.googleapis.com/auth/gerritcodereview,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring,https://www.googleapis.com/auth/pubsub,https://www.googleapis.com/auth/userinfo.email --metadata image_name=chromeos-trusty-17050300-d40e82cb8cf --metadata-from-file cipd_deployments=/usr/local/google/home/jrbarnette/infra_internal/ccompute/scripts/cipd/buildbot-trusty.json
cros-beefy185-c2 O> Executing command: /usr/local/google/home/jrbarnette/tools/google-cloud-sdk/bin/gcloud compute instances create cros-beefy185-c2 --machine-type n1-highmem-32 --zone us-east1-d --project chromeos-bot --image chromeos-trusty-17050300-d40e82cb8cf --image-project chromeos-bot --boot-disk-size 2048GB --boot-disk-type pd-standard --address 104.196.64.246 --scopes https://www.googleapis.com/auth/devstorage.full_control,https://www.googleapis.com/auth/gerritcodereview,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring,https://www.googleapis.com/auth/pubsub,https://www.googleapis.com/auth/userinfo.email --metadata image_name=chromeos-trusty-17050300-d40e82cb8cf --metadata-from-file cipd_deployments=/usr/local/google/home/jrbarnette/infra_internal/ccompute/scripts/cipd/buildbot-trusty.json
cros-beefy185-c2 O> == Output from command: /usr/local/google/home/jrbarnette/tools/google-cloud-sdk/bin/gcloud compute instances create cros-beefy185-c2 --machine-type n1-highmem-32 --zone us-east1-d --project chromeos-bot --image chromeos-trusty-17050300-d40e82cb8cf --image-project chromeos-bot --boot-disk-size 2048GB --boot-disk-type pd-standard --address 104.196.64.246 --scopes https://www.googleapis.com/auth/devstorage.full_control,https://www.googleapis.com/auth/gerritcodereview,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring,https://www.googleapis.com/auth/pubsub,https://www.googleapis.com/auth/userinfo.email --metadata image_name=chromeos-trusty-17050300-d40e82cb8cf --metadata-from-file cipd_deployments=/usr/local/google/home/jrbarnette/infra_internal/ccompute/scripts/cipd/buildbot-trusty.json
cros-beefy185-c2 E> ERROR: (gcloud.compute.instances.create) Could not fetch resource:
cros-beefy185-c2 E>  - Invalid value for field 'resource.disks[0].initializeParams.sourceImage': 'https://www.googleapis.com/compute/v1/projects/chromeos-bot/global/images/chromeos-trusty-17050300-d40e82cb8cf'. The referenced image resource cannot be found.
cros-beefy185-c2 E> 

The old build image was auto-deleted after we stopped using it.

Our current plan of record:

The only thing we really need from the old builder images is a crcmod binary library that's old enough to work with gsutil 4.19.

That library can be pip installed.

So... we figure out a crcmod version that gsutil 4.19 can work with, and we ssh into all of the affected builders to pip install that crcmod.

Assuming the pip version overrides the system version for the chromeos-bot user, that's all we need.

We are currently working to confirm that the crc library is really the problem, and that the proposed fix works.

Notes of interest:

6 PM: All GCE builders were reimaged.
7 PM: TOT release builds failed with the same error and the branch builders.
9 PM: gsutil 4.27 was landed on TOT.

We haven't yet seen a release build pass with gsutil 4.27.

We are watching the current canary run:
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?waterfall=chromeos&buildType=canary&buildConfig=samus-release

Running a TOT release tryjob:
https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/14704

And running local branch tryjobs for samus-release on workstations with/without the PIP update having been applied locally.

Actually, a release tryjob has already been run before submitting the gsutil 4.27 CL: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/14578

If release builds fail, we may need to investigate why tryjob works but real builds fail.

But, that tryjob was run with the old builder image, which was known to work with gsutil 4.19 also.

A possibility that might have been missed earlier in this bug is that the new builder image breaks all versions of gsutil.

Also note, gsutil would only be broken when trying to download 'split' files from GS. I'm not sure when or how often our files are split at upload, so this problem might only apply to android debug symbols.

You're right, that tryjob runs gsutil 4.27 with old builder image, and also since lumpy is not an android board, it doesn't fail stage 'DownloadAndroidDebugSymbols'. Next time we may need to choose an android board to test.

This type of failure is hard to predict. I think your testing was quite reasonable.

The testing failure, it's related to how we pick builder images to use. It's never been a problem before, so building a mechanism for that has never been prioritized.

David James suggests that if we set the gsutil option "check_hashes" to "if_fast_else_skip" then we can work around this problem.

I've confirmed that that suggestion doesn't work.

On a GCE instance using gsutil 4.19:

./gsutil -o "check_hashes:if_fast_else_skip" cp gs://chromeos-arc-images/build
s/git_nyc-mr1-arc-m62-linux-cheets_x86_64-user/4319723/cheets_x86_64-symbols-4319723.zip /tmp

Copying gs://chromeos-arc-images/builds/git_nyc-mr1-arc-m62-linux-cheets_x86_64-user/4319723/cheets_x86_64-symbols-
4319723.zip...
==> NOTE: You are downloading one or more large file(s), which would
run significantly faster if you enabled sliced object uploads. This
feature is enabled by default but requires that compiled crcmod be
installed (see "gsutil help crcmod").
CommandException: 
Downloading this composite object requires integrity checking with CRC32c,
but your crcmod installation isn't using the module's C extension, so the
hash computation will likely throttle download performance. For help
installing the extension, please see:
  $ gsutil help crcmod
To download regardless of crcmod performance or to skip slow integrity
checks, see the "check_hashes" option in your boto config file.
NOTE: It is strongly recommended that you not disable integrity checks. Doing so
could allow data corruption to go undetected during uploading/downloading.


Also:

/gsutil -o "check_hashes:never" cp gs://chromeos-arc-images/builds/git_nyc-mr
1-arc-m62-linux-cheets_x86_64-user/4319723/cheets_x86_64-symbols-4319723.zip /tmp

Copying gs://chromeos-arc-images/builds/git_nyc-mr1-arc-m62-linux-cheets_x86_64-user/4319723/cheets_x86_64-symbols-
4319723.zip...
==> NOTE: You are downloading one or more large file(s), which would
run significantly faster if you enabled sliced object uploads. This
feature is enabled by default but requires that compiled crcmod be
installed (see "gsutil help crcmod").
CommandException: 
Downloading this composite object requires integrity checking with CRC32c,
but your crcmod installation isn't using the module's C extension, so the
hash computation will likely throttle download performance. For help
installing the extension, please see:
  $ gsutil help crcmod
To download regardless of crcmod performance or to skip slow integrity
checks, see the "check_hashes" option in your boto config file.
NOTE: It is strongly recommended that you not disable integrity checks. Doing so
could allow data corruption to go undetected during uploading/downloading.

We've filed b/65495572 with the GS team for this failure.

For not yet understood reasons, the apt installed package does not include the C library.

As a hack workaround, we can do "sudo pip install --ignore-installed crcmod".

This will install a second copy of the crcmod library in /usr/local/lib/python2.7/distfiles/crcmod (as opposed to /usr/lib), which comes first in the python import path.

Afterwards, all versions of gsutil seem to work.

The correct fix is to understand why the apt installed package doesn't include the binary and follow up on that.

The less dirty fix is to have puppet run the sudo pip install command. CrOps owns that version of puppet.

As a very dirty hack to get release builds working this weekend, I can implement this via cbuildbot_launch.

This CL forces the PIP install command to re-run at the start of the build (every time, not just the first time).

This is very, very wrong, but should fix our release builds until we have a better fix. After that, it will leave an extra copy of crc mod in /usr/local/ on all of our builders unless something else cleans that up.

https://chromium-review.googlesource.com/c/chromiumos/chromite/+/658041

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/90f5e7d25d3b0273f5e114f02ca43bb47c007d81

commit 90f5e7d25d3b0273f5e114f02ca43bb47c007d81
Author: Don Garrett <dgarrett@google.com>
Date: Sat Sep 09 03:00:54 2017

cbuildbot_launch: Hack to fix crcmod on builders.

This is a short term hack to fix crcmod on the ChromeOS builders to
fix release builds.

BUG= chromium:763438 
TEST=Local build + unittests.

Change-Id: I412f0367a2aab4067196351f260fc7417af82401
Reviewed-on: https://chromium-review.googlesource.com/658041
Tested-by: Don Garrett <dgarrett@chromium.org>
Trybot-Ready: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Allen Li <ayatane@chromium.org>

[modify] https://crrev.com/90f5e7d25d3b0273f5e114f02ca43bb47c007d81/scripts/cbuildbot_launch.py

remember that we have code in chromite.lib.gs specifically to try building the local copy of crcmod:
  https://chromium-review.googlesource.com/372323

that's working for me locally:
  ~/chromiumos/.cache/common/gsutil_4.27.tar.gz/gsutil/third_party/crcmod/python2/crcmod/_crcfunext.so
  $ gsutil version -l
  compiled crcmod: True

i'm guessing these new GCE instances lack a compiler & python-dev packages ?  those are needed to compile the module.  if you don't have them, you'll just get a warning early on, and then it'll never get retried (by design).
  https://chromium-review.googlesource.com/397702
  https://chromium-review.googlesource.com/379175

They are able to successfully do a pip install that compiles it, so all of
the dependencies should be there. I really don't understand what's
happening.

Since a workaround is in place, reducing priority, but we still need to understand why the workaround was needed, restore the normal solution, and undo the workaround.

Data points:


https://cs.corp.google.com/chromeos_public/chromite/lib/gs.py?rcl=f25bc86969dbbc8b9502187d5d03e10e724af328&l=350

gs.py can/will compile and install the crc binary if it doesn't exist, into the target directory .cache/common/<gsutil tar name>/gsutil/third_party/crcmod/python2/crcmod.

It can bail early if the cache directory is owned by root (which appears to be the case on all builders).

On my workstation, gsutil from a .cache directory is using crcmod installed on the system, even if the crcmod in the cache contains the binary.

Our old Golo builders do not contain crcmod installed at the system level at all. Our new build image contains the ubuntu package python-crcmod WITHOUT the binary extension. 

I believe the root cause is that the package was included on the new builder image, and that removing it would solve this problem for new builders.

However, I think the BETTER solution is to use virtualenv for our python code, so that we have reliable control over what packages exist in the runtime environment. Including crcmod mod there, would solve this problem globally, and much more reliably.

Choices going forward.

A) Switch to virtualenv everywhere (a major long term project)
B) Require python-crcmod + binary extension on new builder images + puppet
C) Improve the cbuildbot_launch hack to not install if not strictly required
D) Remove python-crcmod from new builder images, and reinstance all of builders

This asks for a change to our puppet configuration. We can follow up with changes to the builder image to fix this in future without puppet intervention.

https://crbug.com/764564

if crcmod isn't being used by anything, pruning it from the system image seems like an easy adjustment to make now

i don't think virtualenv would be quite as you describe it ... gsutil is a sep program we fork+exec, so really you're saying we need to create a venv for gsutil itself and run it through that.  not sure how well that'll work out ... we'd have to roll our own venv package which chromite.lib.gs would fetch right ?

reading the gsutil code, looks like it has a funky bug:
- it tries to import the system crcmod
- it then probes the system crcmod to see if compiled crcmod is available
- if not, it inserts the path to the local crcmod into its search path
the assumption here is that the next time python code does "import crcmod", it'll get the local version, but that's not actually what happens ... that first import pollutes the module cache which means the sys.path update does nothing.  i've filed that bug upstream:
https://github.com/GoogleCloudPlatform/gsutil/issues/462

however, we could workaround it ourselves.  in lib.gs.DoCommand, we insert a PYTHONPATH setting that forces the local copy of crcmod to always be searched first.

sent you guys a CL for that.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/d1522dbb77f28416f1f7f7461fef0d869bd9e6ec

commit d1522dbb77f28416f1f7f7461fef0d869bd9e6ec
Author: Mike Frysinger <vapier@chromium.org>
Date: Thu Sep 14 02:28:31 2017

gs: workaround broken system crcmod installs

If the system has a crcmod module installed that lacks a compiled
component, the gsutil code will pick that up instead of our local
copy (which we did compile crcmod on the fly).  Hack up the python
search path to make the local copy of crcmod be found first.

BUG= chromium:763438 
TEST=install (broken) system crcmod and see gs.DoCommand(['version', '-l']) still shows compiled crcmod

Change-Id: I79227e55174f98799c82bd0e29edba8ee16c77c6
Reviewed-on: https://chromium-review.googlesource.com/664264
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/d1522dbb77f28416f1f7f7461fef0d869bd9e6ec/lib/gs.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/0c40109df28ab5373eaabaabfbda3c69c12e5ccb

commit 0c40109df28ab5373eaabaabfbda3c69c12e5ccb
Author: Mike Frysinger <vapier@chromium.org>
Date: Thu Sep 14 15:24:25 2017

Revert "gs: workaround broken system crcmod installs"

This reverts commit d1522dbb77f28416f1f7f7461fef0d869bd9e6ec.

This is breaking bots when gsutil is cached and the system has a
working crcmod available.  The lib.gs code will check the system
and not bother trying to create a local crcmod in that case.  We
need to rework that logic before forcing the local copy.

BUG= chromium:763438 

Change-Id: Id97a7e5e6e0193bdeff9063f43cd122891ca7a5d
Reviewed-on: https://chromium-review.googlesource.com/667656
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/0c40109df28ab5373eaabaabfbda3c69c12e5ccb/lib/gs.py

The gsutil fetch error also breaks the CQ:
android-container-nyc-4332937-r1: 
android-container-nyc-4332937-r1: NOTE: It is strongly recommended that you not disable integrity checks. Doing so
android-container-nyc-4332937-r1: could allow data corruption to go undetected during uploading/downloading.
android-container-nyc-4332937-r1:  
android-container-nyc-4332937-r1: 00:01:50: ERROR: return code: 1; command: /mnt/host/source/.cache/common/gsutil_4.27.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cp -v -- gs://chromeos-arc-images/builds/git_nyc-mr1-arc-linux-cheets_arm-user/4332937/cheets_arm-target_files-4332937.zip /var/cache/chromeos-cache/distfiles/target/cheets_arm-target_files-4332937.zip.tmp
android-container-nyc-4332937-r1: Copying gs://chromeos-arc-images/builds/git_nyc-mr1-arc-linux-cheets_arm-user/4332937/cheets_arm-target_files-4332937.zip...
android-container-nyc-4332937-r1: / [0 files][    0.0 B/882.6 MiB]                                                
android-container-nyc-4332937-r1: ==> NOTE: You are downloading one or more large file(s), which would
android-container-nyc-4332937-r1: run significantly faster if you enabled sliced object downloads. This
android-container-nyc-4332937-r1: feature is enabled by default but requires that compiled crcmod be
android-container-nyc-4332937-r1: installed (see "gsutil help crcmod").
android-container-nyc-4332937-r1: 
android-container-nyc-4332937-r1: CommandException: 
android-container-nyc-4332937-r1: Downloading this composite object requires integrity checking with CRC32c,
android-container-nyc-4332937-r1: but your crcmod installation isn't using the module's C extension, so the
android-container-nyc-4332937-r1: hash computation will likely throttle download performance. For help
android-container-nyc-4332937-r1: installing the extension, please see "gsutil help crcmod".
android-container-nyc-4332937-r1: 
android-container-nyc-4332937-r1: To download regardless of crcmod performance or to skip slow integrity
android-container-nyc-4332937-r1: checks, see the "check_hashes" option in your boto config file.
android-container-nyc-4332937-r1: 
android-container-nyc-4332937-r1: NOTE: It is strongly recommended that you not disable integrity checks. Doing so
android-container-nyc-4332937-r1: could allow data corruption to go undetected during uploading/downloading.
android-container-nyc-4332937-r1: 
android-container-nyc-4332937-r1: cmd=['/mnt/host/source/.cache/common/gsutil_4.27.tar.gz/gsutil/gsutil', '-o', 'Boto:num_retries=10', 'cp', '-v', '--', 'gs://chromeos-arc-images/builds/git_nyc-mr1-arc-linux-cheets_arm-user/4332937/cheets_arm-target_files-4332937.zip', '/var/cache/chromeos-cache/distfiles/target/cheets_arm-target_files-4332937.zip.tmp'], extra env={'BOTO_CONFIG': '/mnt/host/source/src/private-overlays/chromeos-overlay/googlestorage_account.boto', 'PYTHONPATH': '/mnt/host/source/.cache/common/gsutil_4.27.tar.gz/gsutil/third_party/crcmod/python2:'}
android-container-nyc-4332937-r1: !!! Couldn't download 'cheets_arm-target_files-4332937.zip'. Aborting.
android-container-nyc-4332937-r1:  * Fetch failed for 'chromeos-base/android-container-nyc-4332937-r1', Log file:
android-container-nyc-4332937-r1:  *  '/build/veyron_tiger/tmp/portage/logs/chromeos-base:android-container-nyc-4332937-r1:20170914-070149.log'
android-container-nyc-4332937-r1: >>> Failed to emerge chromeos-base/android-container-nyc-4332937-r1 for /build/veyron_tiger/, Log file:
android-container-nyc-4332937-r1: >>>  '/build/veyron_tiger/tmp/portage/logs/chromeos-base:android-container-nyc-4332937-r1:20170914-070149.log'
android-container-nyc-4332937-r1: 
android-container-nyc-4332937-r1:  * Messages for package chromeos-base/android-container-nyc-4332937-r1 merged to /build/veyron_tiger/:
android-container-nyc-4332937-r1: 
android-container-nyc-4332937-r1:  * Fetch failed for 'chromeos-base/android-container-nyc-4332937-r1', Log file:
android-container-nyc-4332937-r1:  *  '/build/veyron_tiger/tmp/portage/logs/chromeos-base:android-container-nyc-4332937-r1:20170914-070149.log'
=== Complete: job android-container-nyc-4332937-r1 (0m1.3s) ===

Does the revert will fix the CQ breakage?

yes, that revert should fix those precq failures in android-container

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/90c9da23ca5b92d01cdd79f7d6827df3d7e29890

commit 90c9da23ca5b92d01cdd79f7d6827df3d7e29890
Author: Don Garrett <dgarrett@google.com>
Date: Fri Sep 15 01:25:45 2017

cbuildbot_launch: Only fixup crcmod when required.

A previous hack started installing crcmod on every build, but that's
often not needed. This change only installs if there is a system
crcmod that doesn't contain a required extension.

BUG= chromium:763438 
TEST=cbuildbot_launch_unittest

Change-Id: I0043a2c1a71ea4191371ccd387546490adca1a3e
Reviewed-on: https://chromium-review.googlesource.com/664055
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Trybot-Ready: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/90c9da23ca5b92d01cdd79f7d6827df3d7e29890/scripts/cbuildbot_launch.py

We've done as much here as we can until crbug.com/764564 is fixed.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/4d6cb64d70cad2e63d40f512c73ccee38bb07c8d

commit 4d6cb64d70cad2e63d40f512c73ccee38bb07c8d
Author: Don Garrett <dgarrett@google.com>
Date: Tue Oct 31 22:15:24 2017

cbuildbot_launch: Remove pip install of crc-mod.

We've gotten real puppet support, so removing this dirty hack.

BUG= chromium:763438 
TEST=Unittests

Change-Id: I2513fdb5e0b45a054937441216c431cae6f265ed
Reviewed-on: https://chromium-review.googlesource.com/744292
Commit-Ready: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: Elliott Friedman <friedman@chromium.org>

[modify] https://crrev.com/4d6cb64d70cad2e63d40f512c73ccee38bb07c8d/scripts/cbuildbot_launch.py