Important to be clear here: the problem is *python* double-free or corruption.

Right. And sorry for confusing the platforms. This is happening on Linux only so far.

Another run: https://build.chromium.org/p/tryserver.webrtc/builders/android_experimental/builds/41

We have also been seeing this on Linux, on the Dart trybots.

Issue 764688 has been merged into this issue.

This happens consistently and is a major blocker for us to make progress on LUCI migration. Some more failures:
https://build.chromium.org/p/tryserver.webrtc/builders/linux_more_configs/builds/9368
https://build.chromium.org/p/tryserver.webrtc/builders/linux_more_configs/builds/9369

The first build that you linked does not exhibit the error, or am I missing something?

Is `fasttop` the GCE fancy gsutil integration stuff?

Re #9: You're right, the error message is not showing up there, but it has the same exit code so I at least suspect it must be the same issue (or related).

The fasttop error message comes from glibc's malloc implementation: 

https://sourceware.org/git/?p=glibc.git;a=blob;f=malloc/malloc.c;h=be472b2ba3885b194f55c29474c4ad5d6a61b729;hb=eefa3be8e4c2c721a9f277d8ea2e11180231829f#l3817

If I had to guess, it's either:
1) A mismatch of system and bundled Python libraries (e.g., system pyOpenSSL + bundle SSL)
2) A mismatch between VirtualEnv pyOpenSSL/cryptography and bundled Python OpenSSL.
3) An actual bug in the bundled Python OpenSSL or cryptography OpenSSL.

It's really hard to diagnose, esp. if it's not readily reproducible. I'd start by making sure the OpenSSL and pyOpenSSL versions all agree.

And here is an example of it hitting the Dart CQ, which is full LUCI, running buildbucket, kitchen, swarming: https://ci.chromium.org/swarming/task/38c3c1347104ff10?server=chromium-swarm.appspot.com

With dnj@'s input in #14 it sounds like it would be great if someone from Chrome infra could take time to compare VMs.

Here are some input on good/bad ones:
https://build.chromium.org/p/tryserver.webrtc/builders/android_more_configs/builds/9379 (slave1218-c4) is green while
https://build.chromium.org/p/tryserver.webrtc/builders/android_compile_rel/builds/1234
https://build.chromium.org/p/tryserver.webrtc/builders/android_more_configs/builds/9379
https://build.chromium.org/p/tryserver.webrtc/builders/android_compile_dbg/builds/1431
gets the errors.

I think you made an error in your links, since they're all red and "/9379" is represented twice.

Sorry, the green one was supposed to be https://build.chromium.org/p/tryserver.webrtc/builders/android_more_configs/builds/9380

I think the traditional way we deal with this is to reprovision the VM as the first step to make sure we're starting from a known state. Has that happened yet?

I haven't asked for any re-imaging of machines. I assume someone familiar with the systems wanted to take a closer look and figure out what's wrong so the systematic problem could be addressed instead?

The problematic builds all have errors like these 
*** Error in `/b/c/vpython/68834e/bin/python': double free or corruption (fasttop): 0x00000000021ed320 ***
and
*** Error in `/b/c/vpython/79400c/bin/python': double free or corruption (fasttop): 0x00000000034c2910 ***
so I guess the problem is probably related to the vpython versions somehow. Logged in on a VM I can see that there are multiple directories in /b/c/vpython/ so I guess those are different rollouts?

The Dart error shows a different path (and dirname):
*** Error in `/b/swarming/w/ir/cache/vpython/f33e6b/bin/python': double free or corruption (fasttop): 0x0000000001e90f50 ***

It's also a bit confusing that all the places where we get this error, we also get the complaint about 'Unable to load AWS_CREDENTIAL_FILE', that never shows up in any of the green runs. That file (/b/build/site_config/.boto) should always be present, right? Or could it be some kind of race condition with puppet updating the file or something?

So I've finally spent some time probing this. So far, it looks like this is due to an interaction between the Python "pyOpenSSL" wheel and the Python bundle's baked-in OpenSSL implementation.

Some intermediate findings:

Firstly, the "AWS_CREDENTIAL_FILE" error is not related. This is due to "download_from_google_storage"'s "--no_auth" flag, which forcefully sets the AWS_CREDENTIAL_FILE environment variable to an empty string. "gsutil" attempts to load this and it obviously fails. The only reason we don't see this all the time is b/c the output is suppressed unless something else fails.

I've run variations of this command:

MALLOC_CHECK_=2 ~/temp/cipdroot/bin/python2.7 /home/chrome-bot/temp/workdir/depot_tools/external_bin/gsutil/gsutil_4.26/gsutil/gsutil -o GSUtil:software_update_check_period=0 -DD ls gs://chromium-luci/48ffe036be8eff7d39ebbdbb705bd26f0ec6f404; echo $?

This will echo "0" for success and non-zero for "abort()" from "MALLOC_CHECK_=2" when a "malloc()"-related failure is encountered.

== (A) Using system Python directly ==
$ MALLOC_CHECK_=2 python /home/chrome-bot/temp/workdir/depot_tools/external_bin/gsutil/gsutil_4.26/gsutil/gsutil -o GSUtil:software_update_check_period=0 -DD ls gs://chromium-luci/48ffe036be8eff7d39ebbdbb705bd26f0ec6f404; echo $?
0

== (B) Using system Python with "gsutil.vpython" VirtualEnv ==
$ MALLOC_CHECK_=2 vpython -vpython-spec ~/temp/workdir/depot_tools/gsutil.vpython -- /home/chrome-bot/temp/workdir/depot_tools/external_bin/gsutil/gsutil_4.26/gsutil/gsutil -o GSUtil:software_update_check_period=0 -DD ls gs://chromium-luci/48ffe036be8eff7d39ebbdbb705bd26f0ec6f404; echo $
0

== (C) Using bundled Python directly ==
$ MALLOC_CHECK_=2 ~/temp/cipdroot/bin/python2.7 /home/chrome-bot/temp/workdir/depot_tools/external_bin/gsutil/gsutil_4.26/gsutil/gsutil -o GSUtil:software_update_check_period=0 -DD ls gs://chromium-luci/48ffe036be8eff7d39ebbdbb705bd26f0ec6f404; echo $?
0

== (D) Using bundled Python with a vanilla VirtualEnv ==
$ PATH=$HOME/temp/cipdroot/bin:$PATH MALLOC_CHECK_=2 vpython -- /home/chrome-bot/temp/workdir/depot_tools/external_bin/gsutil/gsutil_4.26/gsutil/gsutil -o GSUtil:software_update_check_period=0 -DD ls gs://chromium-luci/48ffe036be8eff7d39ebbdbb705bd26f0ec6f404; echo $?
0

== (E) Using bundled Python with "gsutil.vpython" VirtualEnv ==
PATH=$HOME/temp/cipdroot/bin:$PATH MALLOC_CHECK_=2 vpython -vpython-spec /home/chrome-bot/temp/workdir/depot_tools/gsutil.vpython -- /home/chrome-bot/temp/workdir/depot_tools/external_bin/gsutil/gsutil_4.26/gsutil/gsutil -o GSUtil:software_update_check_period=0 -DD ls gs://chromium-luci/48ffe036be8eff7d39ebbdbb705bd26f0ec6f404; echo $?
Aborted (core dumped)
134

== (F) Using bundled Python with "gsutil.vpython" VirtualEnv, modified to remove PyOpenSSL wheel. ==
<Command is same as (E)>
0

=======

As you can see from (A)-(D), any combination that doesn't involve bundled Python and the "gsutil.vpython" VirtualEnv is fine. As you can see from (F), removing the PyOpenSSL binding import from "gsutil.vpython" is also fine! Therefore, it seems like the PyOpenSSL binding wheel is the source of the problem.

I can open the core in "gdb"; however, the current Python binary is stripped, so I have no symbols. I'm currently building a new Python version that is not stripped so I can hopefully get a feel for where the failure occurred.

Stack trace from (E) is:

Program received signal SIGABRT, Aborted.
0x00007ffff6ee3c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56      ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  0x00007ffff6ee3c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff6ee7028 in __GI_abort () at abort.c:89
#2  0x00007ffff6f2d355 in malloc_printerr (ptr=<optimized out>, str=<optimized out>, action=<optimized out>) at malloc.c:5002
#3  free_check (mem=<optimized out>, caller=<optimized out>) at hooks.c:298
#4  0x0000000000856f75 in OPENSSL_cleanup ()
#5  0x00007ffff6ee91a9 in __run_exit_handlers (status=0, listp=0x7ffff726f6c8 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true) at exit.c:82
#6  0x00007ffff6ee91f5 in __GI_exit (status=<optimized out>) at exit.c:104
#7  0x00007ffff6ecef4c in __libc_start_main (main=0x474440 <main>, argc=13, argv=0x7fffffffe4e8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe4d8) at libc-start.c:321
#8  0x000000000047446e in _start ()

It looks like OpenSSL's 'atexit' handler is the problem.

Breaking on "atexit", I see:

#0  0x00007ffff5dbecb0 in atexit () from /home/chrome-bot/.vpython-root/cce7fe/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so
#1  0x00007ffff5cc93a3 in ossl_init_base_ossl_ () from /home/chrome-bot/.vpython-root/cce7fe/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so
#2  0x00007ffff7bc9a80 in pthread_once () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:103
#3  0x00007ffff5d12699 in CRYPTO_THREAD_run_once () from /home/chrome-bot/.vpython-root/cce7fe/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so

The "atexit" handler is being called from the "cryptography" wheel's statically-compiled OpenSSL.

However, from the first backtrace, the "OPENSSL_cleanup" that is baked into the Python binary is the one that's being called!

In scenario (F), the same procedure reveals that the "atexit" call is from the bundle's OpenSSL instead of the "cryptography" wheel.

Conclusion ATM is that something about pyOpenSSL package is causing the cryptography OpenSSL binding's initialization to be called instead of the built-in Python's binding. This is registering the cryptography package's OpenSSL finalizer, but it is *actually* calling the built-in Python's finalizer!

---

So what to do? Some thoughts (without a conclusion):
- It's weird to have a process where symbols from a dynamically loaded shared object (cryptography's OpenSSL) have the same name as built-in symbols.
- We can't/shouldn't have cryptography not bundle OpenSSL, since that's a large portion of the point of the package.
- We probably should export OpenSSL symbols from our bundled Python, since packages may want to use them and expect them to exist.

Is there a way to have these symbols coexist? Or, at the very least, have symbol resolution in one symbol space consistently resolve to that symbol space?

not a milo bug, not an sdk bug

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/ec1dfc3fc3af7a297aca526e90a47e9d10e79544

commit ec1dfc3fc3af7a297aca526e90a47e9d10e79544
Author: Dan Jacques <dnj@chromium.org>
Date: Tue Oct 03 08:52:08 2017

[tpp/python] Suppress symbols, don't strip.

Suppress static library symbols, notably OpenSSL, from being linked by
external modules, notably "cryptography".

Remove stripping. It saves a few dozen megabytes, but makes it a lot
harder to debug actual problems. For now, it's not worth it.

BUG= chromium:763792 
TEST=local
  - Ran on exemplar Linux box, observed success case.

Change-Id: I33d23db292741fafe014e2aa184d54b4fea0cbef
Reviewed-on: https://chromium-review.googlesource.com/696645
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/ec1dfc3fc3af7a297aca526e90a47e9d10e79544/recipes/recipes/third_party_packages.expected/dry_run.json
[modify] https://crrev.com/ec1dfc3fc3af7a297aca526e90a47e9d10e79544/recipes/recipe_modules/third_party_packages/examples/python.expected/new_on_linux-amd64.json
[add] https://crrev.com/ec1dfc3fc3af7a297aca526e90a47e9d10e79544/recipes/recipe_modules/third_party_packages/resources/python/gnu_version_script.txt
[modify] https://crrev.com/ec1dfc3fc3af7a297aca526e90a47e9d10e79544/recipes/recipe_modules/third_party_packages/examples/python.expected/mac_exists.json
[modify] https://crrev.com/ec1dfc3fc3af7a297aca526e90a47e9d10e79544/recipes/recipe_modules/third_party_packages/examples/python.expected/new_on_mac-amd64.json
[modify] https://crrev.com/ec1dfc3fc3af7a297aca526e90a47e9d10e79544/recipes/recipes/third_party_packages.expected/basic.json
[modify] https://crrev.com/ec1dfc3fc3af7a297aca526e90a47e9d10e79544/recipes/recipe_modules/third_party_packages/examples/python.expected/new_on_linux-386.json
[modify] https://crrev.com/ec1dfc3fc3af7a297aca526e90a47e9d10e79544/recipes/recipe_modules/third_party_packages/python.py
[modify] https://crrev.com/ec1dfc3fc3af7a297aca526e90a47e9d10e79544/recipes/recipe_modules/third_party_packages/examples/python.expected/mac_failure.json

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/518ffa7fe27ce1746cd6e32d5db79b1cf25ac8d0

commit 518ffa7fe27ce1746cd6e32d5db79b1cf25ac8d0
Author: Dan Jacques <dnj@chromium.org>
Date: Tue Oct 03 12:23:09 2017

[tpp/python] Ensure autoconf.

A recent patch added "autoconf" requirement. However, our Mac builder
doesn't actually have "autoconf" installed. Ensure that it is installed
and in our PATH.

TBR=iannucci@chromium.org
BUG= chromium:763792 
TEST=expectations

Change-Id: I5b8eace75d0287ebc758862c696ed670a8f70a0d
Reviewed-on: https://chromium-review.googlesource.com/697548
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>

[modify] https://crrev.com/518ffa7fe27ce1746cd6e32d5db79b1cf25ac8d0/recipes/recipes/third_party_packages.expected/dry_run.json
[modify] https://crrev.com/518ffa7fe27ce1746cd6e32d5db79b1cf25ac8d0/recipes/recipe_modules/third_party_packages/examples/python.expected/mac_failure.json
[modify] https://crrev.com/518ffa7fe27ce1746cd6e32d5db79b1cf25ac8d0/recipes/recipe_modules/third_party_packages/examples/python.expected/new_on_mac-amd64.json
[modify] https://crrev.com/518ffa7fe27ce1746cd6e32d5db79b1cf25ac8d0/recipes/recipes/third_party_packages.expected/basic.json
[modify] https://crrev.com/518ffa7fe27ce1746cd6e32d5db79b1cf25ac8d0/recipes/recipe_modules/third_party_packages/examples/python.expected/new_on_linux-386.json
[modify] https://crrev.com/518ffa7fe27ce1746cd6e32d5db79b1cf25ac8d0/recipes/recipe_modules/third_party_packages/examples/python.expected/new_on_linux-amd64.json
[modify] https://crrev.com/518ffa7fe27ce1746cd6e32d5db79b1cf25ac8d0/recipes/recipe_modules/third_party_packages/python.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/536c7ff5be3efb76e92fe60bd988bf8e4ca02639

commit 536c7ff5be3efb76e92fe60bd988bf8e4ca02639
Author: Dan Jacques <dnj@chromium.org>
Date: Tue Oct 03 12:49:56 2017

[cipd_bootstrap_v2] Canary Python, bump stable Git

Canary Python version:2.7.14.chromium13
Promote canary Git to version:2.14.1.chromium12

TBR=iannucci@chromium.org
BUG= chromium:763792 
TEST=None

Change-Id: I2e0dee93c258f0632b9250e6d10205c54589a6bf
Reviewed-on: https://chromium-review.googlesource.com/697550
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>

[modify] https://crrev.com/536c7ff5be3efb76e92fe60bd988bf8e4ca02639/scripts/slave/cipd_bootstrap_v2.py

Hey kjellander@. As you can see, this is an obscure and difficult to identify/resolve problem. That said, I *think* I've found a solution that works. This solution is current deployed. I say the first part to limit your expectations :)

Would you please try again and see if the problem has been resolved?

It indeed seems to be fixed! I fired like 10 tryjobs for https://webrtc-review.googlesource.com/c/src/+/1576 and they're all green!

Awesome. I'm going to mark this as Fixed. Thanks for bearing with me!

Summary follow-on for #23: my initial assessment that we wanted to export symbols from Python was reduced a bit. I ended up updating the build code to mark many (most? all?) of the symbols that Python pulled in from static libraries, including OpenSSL, as "LOCAL", meaning that external shared objects that reference symbols with the same name will not try and link against them. This means that the "cryptography" wheel's OpenSSL will link against itself like it always intended to.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/20a66022bb61ec9295d03da1cd7c89f6d5ce739a

commit 20a66022bb61ec9295d03da1cd7c89f6d5ce739a
Author: Aaron Gable <agable@chromium.org>
Date: Tue Oct 03 16:46:41 2017

Revert "[cipd_bootstrap_v2] Canary Python, bump stable Git"

This reverts commit 536c7ff5be3efb76e92fe60bd988bf8e4ca02639.

Reason for revert: o/433309, broke all gsubtreed bots

Original change's description:
> [cipd_bootstrap_v2] Canary Python, bump stable Git
> 
> Canary Python version:2.7.14.chromium13
> Promote canary Git to version:2.14.1.chromium12
> 
> TBR=iannucci@chromium.org
> BUG= chromium:763792 
> TEST=None
> 
> Change-Id: I2e0dee93c258f0632b9250e6d10205c54589a6bf
> Reviewed-on: https://chromium-review.googlesource.com/697550
> Reviewed-by: Daniel Jacques <dnj@chromium.org>
> Commit-Queue: Daniel Jacques <dnj@chromium.org>

TBR=iannucci@chromium.org,dnj@chromium.org

Change-Id: I9d63f8630ab0125c2bea7012a1d5a69ce4252c46
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  chromium:763792 
Reviewed-on: https://chromium-review.googlesource.com/698226
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Aaron Gable <agable@chromium.org>

[modify] https://crrev.com/20a66022bb61ec9295d03da1cd7c89f6d5ce739a/scripts/slave/cipd_bootstrap_v2.py

The roll-out was reverted because I suppressed too many symbols. At least I know we're on the right path now, though. I'll figure out a way to be more surgical and re-land.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/dbfc5fe341e303490768135cd14c8f754432757d

commit dbfc5fe341e303490768135cd14c8f754432757d
Author: Dan Jacques <dnj@chromium.org>
Date: Wed Oct 04 00:13:19 2017

[tpp/python] Explicitly export symbols.

Change Python symbol export from a blacklist to a whitelist,
whitelisting only Python ABI. This is easier to understand and more
effective, and if there is a missing symbol set, users can easily add it
to the version script.

BUG= chromium:763792 
TEST=None

Change-Id: I1c2f2f48e115f851a385b28659a3e26c5f0e95a9
Reviewed-on: https://chromium-review.googlesource.com/698824
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/dbfc5fe341e303490768135cd14c8f754432757d/recipes/recipes/third_party_packages.expected/dry_run.json
[modify] https://crrev.com/dbfc5fe341e303490768135cd14c8f754432757d/recipes/recipe_modules/third_party_packages/examples/python.expected/new_on_windows-386.json
[add] https://crrev.com/dbfc5fe341e303490768135cd14c8f754432757d/recipes/recipe_modules/third_party_packages/examples/python.expected/win_exists.json
[modify] https://crrev.com/dbfc5fe341e303490768135cd14c8f754432757d/recipes/recipe_modules/third_party_packages/examples/python.expected/new_on_windows-amd64.json
[modify] https://crrev.com/dbfc5fe341e303490768135cd14c8f754432757d/recipes/recipe_modules/third_party_packages/examples/python.py
[modify] https://crrev.com/dbfc5fe341e303490768135cd14c8f754432757d/recipes/recipe_modules/third_party_packages/resources/python/gnu_version_script.txt
[modify] https://crrev.com/dbfc5fe341e303490768135cd14c8f754432757d/recipes/recipe_modules/third_party_packages/examples/python.expected/mac_exists.json
[modify] https://crrev.com/dbfc5fe341e303490768135cd14c8f754432757d/recipes/recipe_modules/third_party_packages/examples/python.expected/new_on_mac-amd64.json
[modify] https://crrev.com/dbfc5fe341e303490768135cd14c8f754432757d/recipes/README.recipes.md
[modify] https://crrev.com/dbfc5fe341e303490768135cd14c8f754432757d/recipes/recipe_modules/third_party_packages/examples/python.expected/mac_failure.json
[modify] https://crrev.com/dbfc5fe341e303490768135cd14c8f754432757d/recipes/recipes/third_party_packages.expected/basic.json
[modify] https://crrev.com/dbfc5fe341e303490768135cd14c8f754432757d/recipes/recipe_modules/third_party_packages/examples/python.expected/new_on_linux-386.json
[modify] https://crrev.com/dbfc5fe341e303490768135cd14c8f754432757d/recipes/recipe_modules/third_party_packages/examples/python.expected/new_on_linux-amd64.json
[modify] https://crrev.com/dbfc5fe341e303490768135cd14c8f754432757d/recipes/recipe_modules/third_party_packages/api.py
[add] https://crrev.com/dbfc5fe341e303490768135cd14c8f754432757d/.gitattributes
[modify] https://crrev.com/dbfc5fe341e303490768135cd14c8f754432757d/recipes/recipe_modules/third_party_packages/resources/python/python_test.py
[modify] https://crrev.com/dbfc5fe341e303490768135cd14c8f754432757d/recipes/recipe_modules/third_party_packages/python.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/829cfcad2f17456f5b17097aa59df75133f85d66

commit 829cfcad2f17456f5b17097aa59df75133f85d66
Author: Dan Jacques <dnj@chromium.org>
Date: Wed Oct 04 00:42:20 2017

[cipd_bootstrap_v2] Canary Python, bump stable Git

This is a reland of 536c7ff5be3efb76e92fe60bd988bf8e4ca02639,
using an updated Python version.

Canary Python version:2.7.14.chromium14
Promote canary Git to version:2.14.1.chromium12

TBR=iannucci@chromium.org
BUG= chromium:763792 
TEST=None

Change-Id: Ibf0c519240f0531d83f52b2c56dff065677c7e8a
Reviewed-on: https://chromium-review.googlesource.com/698950
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>

[modify] https://crrev.com/829cfcad2f17456f5b17097aa59df75133f85d66/scripts/slave/cipd_bootstrap_v2.py

kjellander@, can you try again? I rolled out a new version that should fix the old one's problem, but I'd like to confirm that it still works for you.

Yes, I can confirm things are looking fine again. Thanks!

Thanks for the quick turnaround! Awesome :D

We are still seeing this problem today on LUCI tasks: https://chromium-swarm.appspot.com/task?id=3900455f6e556510

It seems it still picks up the old python: infra/python/cpython/${platform}:version:2.7.14.chromium12

Is there something else that needs to happen before the fix will affect those tasks?

Sorry, I haven't pushed the new version to LUCI yet. Since kjellander@'s operation was the exemplar use case, I didn't want to go through with a full push until I'd confirmed that their problem was solved. I'll go ahead and move forward with a LUCI push right now.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/6fe026181ba29323b171a8f647977775988de291

commit 6fe026181ba29323b171a8f647977775988de291
Author: Dan Jacques <dnj@google.com>
Date: Wed Oct 04 13:44:38 2017