Python segfault while computing trace metrics |
|||||||||
Issue descriptionFull log: https://chrome-swarming.appspot.com/task?id=407ba8192e6e6b10&refresh=10&show_raw=1 (INFO) 2018-10-11 11:30:33,963 timeline_based_measurement._ComputeTimelineBasedMetrics:317 Starting to compute metrics on trace *** Error in `/b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python': free(): invalid pointer: 0x00007f1a48157630 *** ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f1a6d2077e5] /lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7f1a6d21037a] /lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f1a6d21453c] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python(PyThread_ReInitTLS+0x81)[0x50b0c1] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python(PyOS_AfterFork+0x37)[0x511757] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python[0x5fce86] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python(PyEval_EvalFrameEx+0x6efa)[0x4c449a] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python(PyEval_EvalCodeEx+0x7f8)[0x4c6a88] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python(PyEval_EvalFrameEx+0x6cd1)[0x4c4271] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python(PyEval_EvalCodeEx+0x7f8)[0x4c6a88] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python[0x6278ff] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python(PyObject_Call+0x3a)[0x42e56a] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python[0x432a55] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python(PyObject_Call+0x3a)[0x42e56a] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python[0x48ff3f] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python[0x48ae1a] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python(PyObject_Call+0x3a)[0x42e56a] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python(PyEval_EvalFrameEx+0x1805)[0x4beda5] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python(PyEval_EvalFrameEx+0x71f6)[0x4c4796] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python(PyEval_EvalCodeEx+0x7f8)[0x4c6a88] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python(PyEval_EvalFrameEx+0x6cd1)[0x4c4271] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python(PyEval_EvalCodeEx+0x7f8)[0x4c6a88] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python(PyEval_EvalFrameEx+0x6cd1)[0x4c4271] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python(PyEval_EvalCodeEx+0x7f8)[0x4c6a88] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python[0x6278ff] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python(PyObject_Call+0x3a)[0x42e56a] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python(PyEval_EvalFrameEx+0x213b)[0x4bf6db] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python(PyEval_EvalCodeEx+0x7f8)[0x4c6a88] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python(PyEval_EvalFrameEx+0x6cd1)[0x4c4271] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python(PyEval_EvalCodeEx+0x7f8)[0x4c6a88] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python(PyEval_EvalFrameEx+0x6cd1)[0x4c4271] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python(PyEval_EvalCodeEx+0x7f8)[0x4c6a88] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python[0x6278ff] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python(PyObject_Call+0x3a)[0x42e56a] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python(PyEval_EvalFrameEx+0x213b)[0x4bf6db] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python(PyEval_EvalFrameEx+0x71f6)[0x4c4796] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python(PyEval_EvalFrameEx+0x71f6)[0x4c4796] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python(PyEval_EvalCodeEx+0x7f8)[0x4c6a88] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python[0x627830] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python(PyObject_Call+0x3a)[0x42e56a] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python[0x432a55] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python(PyObject_Call+0x3a)[0x42e56a] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python(PyEval_CallObjectWithKeywords+0x47)[0x4bd4f7] /b/swarming/w/ir/.swarming_module_cache/vpython/f22999/bin/python[0x50f602] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f1a6dea96ba] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f1a6d29741d]
,
Oct 15
Wow, any idea what's going on or how to debug this further?
,
Oct 15
Is this failure case intermittent or consistent? I hit something similar for Chrome Remote Desktop tests. We had no idea how to fix it and ended up doing a hacky workaround. If we can get a small repro then hopefully we can get someone from python development team to help us. The other thing we could do is migrate to python3. +Dirk
,
Oct 15
unfortunately, switching to python3 isn't a short-term option.
,
Oct 16
#3: I saw this on a couple of different log files when I filed the bug, but it seems to be fairly random. I wonder if we're running out of memory while processing the trace?
,
Oct 18
Via a simple Google search: PyOS_AfterFork -> https://docs.python.org/2/c-api/sys.html#c.PyOS_AfterFork Looks like it's immediately called at the very start of the process. PyThread_ReInitTLS tells me this looks similar to https://bugs.python.org/issue30967
,
Oct 18
,
Oct 18
,
Oct 18
,
Jan 16
(6 days ago)
,
Jan 16
(6 days ago)
|
|||||||||
►
Sign in to add a comment |
|||||||||
Comment 1 by charliea@chromium.org
, Oct 15Components: -Speed>Metrics Speed>Telemetry