autoserv: stop using atfork + possibly delete paramiko host code |
||||||||
Issue description
autoserv has this whacky incanation near the top of it:
try:
import atfork
atfork.monkeypatch_os_fork_functions()
import atfork.stdlib_fixer
# Fix the Python standard library for threading+fork safety with its
# internal locks. http://code.google.com/p/python-atfork/
import warnings
warnings.filterwarnings('ignore', 'logging module already imported')
atfork.stdlib_fixer.fix_logging_module()
except ImportError, e:
from autotest_lib.client.common_lib import global_config
if _CONFIG.get_config_value(
'AUTOSERV', 'require_atfork_module', type=bool, default=False):
print >>sys.stderr, 'Please run utils/build_externals.py'
print e
sys.exit(1)
The purposes of atfork are sort of mysterious, but it seems like it is intended to fix deadlocks that happen when you mix locks with *threads*. I think we have shunned python threads for development for several years, so not sure why we need this code (last touched in 2008). And this whacky behavior has caused at least 1 CQ failure in https://uberchromegw.corp.google.com/i/chromeos/builders/daisy_skate-paladin/builds/9142 due to bad interaction with another CL that set up some logging before atfork had a chance to attach.
,
Mar 1 2017
+people who have touched paramiko
,
Mar 1 2017
,
Mar 1 2017
FYI: Using multiprocessing and multithreading at the same time is pretty much impossible to get right on linux. Once you have multiple threads, forking becomes unsafe. It causes many common C libraries to misbehave, and even memlock had a deadlock bug in older libc... From reading + experience, it's best to just avoid this.
,
Mar 1 2017
+1 to #4 which is why we've shunned it for a long time.
,
Mar 1 2017
I remember Miller and I investigated Paramiko and we abandoned it in exchange for master ssh connections. Feel free to delete all this relevant code.
,
Mar 2 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/96f8e17a3af6e387488ccc72951a332169411bfd commit 96f8e17a3af6e387488ccc72951a332169411bfd Author: Aviv Keshet <akeshet@chromium.org> Date: Thu Mar 02 23:12:41 2017
,
Mar 2 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/c368db304d3b59d3943e7ceb1607d0d7cbdadbcd commit c368db304d3b59d3943e7ceb1607d0d7cbdadbcd Author: Aviv Keshet <akeshet@chromium.org> Date: Thu Mar 02 23:12:41 2017 autotest: eliminate paramiko BUG= chromium:697254 TEST=None Change-Id: Idb935267f7b01b5330d4097c633323f01717b61c Reviewed-on: https://chromium-review.googlesource.com/447823 Commit-Ready: Aviv Keshet <akeshet@chromium.org> Tested-by: Aviv Keshet <akeshet@chromium.org> Reviewed-by: Simran Basi <sbasi@chromium.org> [modify] https://crrev.com/c368db304d3b59d3943e7ceb1607d0d7cbdadbcd/utils/external_packages.py [delete] https://crrev.com/4e54ae7287b9601242e99dd6a8158d8b91824997/server/hosts/paramiko_host.py [modify] https://crrev.com/c368db304d3b59d3943e7ceb1607d0d7cbdadbcd/server/hosts/factory_unittest.py [modify] https://crrev.com/c368db304d3b59d3943e7ceb1607d0d7cbdadbcd/server/hosts/factory.py [modify] https://crrev.com/c368db304d3b59d3943e7ceb1607d0d7cbdadbcd/global_config.ini
,
Mar 2 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/4e54ae7287b9601242e99dd6a8158d8b91824997 commit 4e54ae7287b9601242e99dd6a8158d8b91824997 Author: Aviv Keshet <akeshet@chromium.org> Date: Thu Mar 02 23:12:41 2017 autotest: stop using atfork CQ-DEPEND=CL:*332048 BUG= chromium:697254 TEST=None Change-Id: I3f3ae5d036e43248f7832059bccc70cb83a61263 Reviewed-on: https://chromium-review.googlesource.com/447864 Commit-Ready: Aviv Keshet <akeshet@chromium.org> Tested-by: Aviv Keshet <akeshet@chromium.org> Reviewed-by: Dan Shi <dshi@google.com> [modify] https://crrev.com/4e54ae7287b9601242e99dd6a8158d8b91824997/utils/external_packages.py [modify] https://crrev.com/4e54ae7287b9601242e99dd6a8158d8b91824997/server/autoserv [modify] https://crrev.com/4e54ae7287b9601242e99dd6a8158d8b91824997/global_config.ini
,
Mar 7 2017
Fixed now?
,
Mar 7 2017
,
May 30 2017
,
Aug 1 2017
,
Jan 22 2018
|
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by akes...@chromium.org
, Mar 1 2017