New issue
Advanced search Search tips

Issue 620404 link

Starred by 2 users
Status: Duplicate
Merged: issue 610902
Owner:
dgarr...@chromium.org
Closed: Jun 2016
Cc:
vapier@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug

Blocking:
issue 610902



Sign in to add a comment

upload_symbols is crashing

Project Member Reported by dgarr...@chromium.org, Jun 15 2016 
Since enabling --dedupe, the upload_symbols script is regularly crashing with these logs.

https://uberchromegw.corp.google.com/i/chromeos/builders/butterfly-release/builds/2687/steps/DebugSymbols/logs/stdio


04:48:08: INFO: uploading all symbols to http://clients2.google.com/cr/symbol from /b/cbuild/internal_master/chroot/build/butterfly/usr/lib/debug/breakpad
04:48:08: INFO: Starting new HTTPS connection (1): isolateserver.appspot.com
04:48:11: INFO: Queried 100 files, 7 cache hit
04:48:14: INFO: Queried 100 files, 10 cache hit
04:48:15: INFO: Queried 100 files, 6 cache hit
04:48:16: INFO: Queried 100 files, 7 cache hit
04:48:18: INFO: Queried 100 files, 11 cache hit
04:48:18: INFO: Queried 100 files, 9 cache hit
04:48:19: INFO: Queried 100 files, 4 cache hit
04:48:20: INFO: Queried 100 files, 5 cache hit
04:48:21: INFO: Queried 100 files, 9 cache hit
04:48:21: INFO: Queried 100 files, 5 cache hit
04:48:22: INFO: Queried 100 files, 5 cache hit
04:48:22: INFO: Queried 100 files, 7 cache hit
04:48:24: INFO: Queried 100 files, 7 cache hit
04:48:24: INFO: Queried 100 files, 6 cache hit
04:48:25: INFO: Queried 100 files, 2 cache hit
04:48:26: INFO: Queried 100 files, 5 cache hit
04:48:27: INFO: Queried 100 files, 8 cache hit
04:48:28: INFO: Queried 100 files, 5 cache hit
04:48:29: INFO: Queried 100 files, 2 cache hit
04:48:30: INFO: Queried 100 files, 7 cache hit
04:48:31: INFO: Queried 100 files, 4 cache hit
04:48:33: INFO: Queried 100 files, 13 cache hit
04:48:33: INFO: Queried 100 files, 5 cache hit
04:48:34: INFO: Queried 100 files, 5 cache hit
04:48:35: INFO: Queried 100 files, 7 cache hit
04:48:36: INFO: Queried 100 files, 9 cache hit
04:48:37: INFO: Queried 100 files, 7 cache hit
04:48:39: INFO: Queried 100 files, 7 cache hit
04:48:40: INFO: Queried 100 files, 10 cache hit
04:48:41: INFO: Queried 100 files, 5 cache hit
04:48:41: INFO: Queried 100 files, 10 cache hit
04:48:42: INFO: Queried 100 files, 7 cache hit
04:48:43: INFO: Queried 100 files, 8 cache hit
04:48:43: INFO: Queried 100 files, 8 cache hit
04:48:44: INFO: Queried 100 files, 7 cache hit
04:48:46: INFO: Queried 100 files, 5 cache hit
04:48:47: INFO: Queried 100 files, 7 cache hit
04:48:47: INFO: Queried 100 files, 5 cache hit
04:48:48: INFO: Queried 100 files, 5 cache hit
04:48:50: INFO: Queried 100 files, 5 cache hit
04:48:50: INFO: Queried 100 files, 11 cache hit
04:48:51: INFO: Queried 100 files, 8 cache hit
04:48:51: INFO: Queried 54 files, 2 cache hit
04:48:33: WARNING: could not upload: libbootstat.so.sym: HTTP 403: Forbidden
04:48:55: WARNING: could not upload: test_assoc_shutdown.sym: HTTP 403: Forbidden
04:49:17: WARNING: could not upload: lchown03_16.sym: HTTP 403: Forbidden
04:49:39: WARNING: could not upload: testopt.sym: HTTP 403: Forbidden
04:59:42: WARNING: Killing tasks: [<_BackgroundTask(_BackgroundTask-4, started)>]
04:59:42: WARNING: Killing 28645 (sig=24 SIGXCPU)
04:59:42: WARNING: RunCommand: pstree -Apals 28645
04:59:42: WARNING: (stdout):
init,1
  `-chromebuild-sta,1097 -u /opt/chromebuild/chromebuild-startup.py --log /var/log/messages/chromebuild/startup.log
      `-python,1458 /opt/infra-bot-setup/infra-python/run.py infra.tools.bot_setup.start --root_dir /b --password_file /home/chrome-bot/.password_json --slave_name cros-beefy17-c2
          `-python,3052 /b/build/slave/run_slave.py --no_save --no-gclient-sync --python buildbot.tac --nodaemon --logfile twistd.log
              `-python,10762 -u ../../../scripts/slave/annotated_run.py --use-factory-properties-from-disk --build-properties-gz=eNqdUctOwzAQ/Befm7ppns0J7pyQOCEU2c4mNTh2WDutKsS/s3lxAS6cvJrxzu7MfjBpRA9G+8CqZ6bO6HqIpAt3c6nHfu+wYy87JlFYdWYV64UPgIyQUZuGvj49PhB8DmHwFeejBFxkuuteORz2nXOdAap7rvlCOc83AUBLC5CAHAPptuYWIRgQHrYfduxJk1XHvCx2TIkAncMbdQyoe0EVgVLWPxecUOVsq7s/5KcPF4FaWHJPynZR63SoES7aa2cJPyRpkjSQp8VJSgFlchCQNlmetG2exdmxaWMh8zanzmV0Pa9d64ZV5ak4xelGrE63DKhhQPcKKsxbW92CD9EFcBo8sQiD8zosbreA1xtpOwuaNV7vRlRLyN8R/y75PhIEzT0NjdM8K08ZPRPxH8NenaEZzXSeNb/aG3GBadRcbJbR+UgCtLe4iNSR2KvDt0ZPfVzyOTA+N/Afh4rWg35+AXh26w4=
                  `-python,11465 ../../../scripts/slave/annotated_run.py --use-factory-properties-from-disk --build-properties-gz=eNqdUctOwzAQ/Befm7ppns0J7pyQOCEU2c4mNTh2WDutKsS/s3lxAS6cvJrxzu7MfjBpRA9G+8CqZ6bO6HqIpAt3c6nHfu+wYy87JlFYdWYV64UPgIyQUZuGvj49PhB8DmHwFeejBFxkuuteORz2nXOdAap7rvlCOc83AUBLC5CAHAPptuYWIRgQHrYfduxJk1XHvCx2TIkAncMbdQyoe0EVgVLWPxecUOVsq7s/5KcPF4FaWHJPynZR63SoES7aa2cJPyRpkjSQp8VJSgFlchCQNlmetG2exdmxaWMh8zanzmV0Pa9d64ZV5ak4xelGrE63DKhhQPcKKsxbW92CD9EFcBo8sQiD8zosbreA1xtpOwuaNV7vRlRLyN8R/y75PhIEzT0NjdM8K08ZPRPxH8NenaEZzXSeNb/aG3GBadRcbJbR+UgCtLe4iNSR2KvDt0ZPfVzyOTA+N/Afh4rWg35+AXh26w4=
                      `-python,11477 -u /b/build_internal/scripts/slave/recipes.py --verbose run --workdir=/b/build/slave/butterfly-release-master/build --properties-file=/b/build/slave/butterfly-release-master/build/.recipe_runtime/tmp5ISfsM/recipe_properties.json cros/cbuildbot_internal
                          `-python,11480 -u /b/build_internal/scripts/slave/.recipe_deps/recipe_engine/recipes.py --package /b/build_internal/scripts/slave/infra/config/recipes.cfg --bootstrap-script /b/build_internal/scripts/slave/recipes.py --verbose run --workdir=/b/build/slave/butterfly-release-master/build --properties-file=/b/build/slave/butterfly-release-master/build/.recipe_runtime/tmp5ISfsM/recipe_properties.json cros/cbuildbot_internal
                              `-cbuildbot,11817 /b/build/slave/butterfly-release-master/build/chromite/bin/cbuildbot --buildroot /b/cbuild/internal_master --buildbot --branch master --buildnumber 2687 --repo-cache /var/cache/chrome-infra/ccompute-setup/cros-internal --master-build-id 897914 butterfly-release
                                  `-cbuildbot,30767 chromite/bin/cbuildbot butterfly-release --buildroot /b/cbuild/internal_master --buildbot --branch master --buildnumber 2687 --repo-cache /var/cache/chrome-infra/ccompute-setup/cros-internal --master-build-id 897914 --resume --timeout 0 --notee --nocgroups --buildroot /b/cbuild/internal_master --version 8452.0.0 --metadata_dump /tmp/cbuildbot-tmpJaEU7o/metadataoBUV9g
                                      `-cbuildbot,22367 chromite/bin/cbuildbot butterfly-release --buildroot /b/cbuild/internal_master --buildbot --branch master --buildnumber 2687 --repo-cache /var/cache/chrome-infra/ccompute-setup/cros-internal --master-build-id 897914 --resume --timeout 0 --notee --nocgroups --buildroot /b/cbuild/internal_master --version 8452.0.0 --metadata_dump /tmp/cbuildbot-tmpJaEU7o/metadataoBUV9g
                                          `-cbuildbot,9072 chromite/bin/cbuildbot butterfly-release --buildroot /b/cbuild/internal_master --buildbot --branch master --buildnumber 2687 --repo-cache /var/cache/chrome-infra/ccompute-setup/cros-internal --master-build-id 897914 --resume --timeout 0 --notee --nocgroups --buildroot /b/cbuild/internal_master --version 8452.0.0 --metadata_dump /tmp/cbuildbot-tmpJaEU7o/metadataoBUV9g
                                              `-cbuildbot,9371 chromite/bin/cbuildbot butterfly-release --buildroot /b/cbuild/internal_master --buildbot --branch master --buildnumber 2687 --repo-cache /var/cache/chrome-infra/ccompute-setup/cros-internal --master-build-id 897914 --resume --timeout 0 --notee --nocgroups --buildroot /b/cbuild/internal_master --version 8452.0.0 --metadata_dump /tmp/cbuildbot-tmpJaEU7o/metadataoBUV9g
                                                  `-cbuildbot,26742 chromite/bin/cbuildbot butterfly-release --buildroot /b/cbuild/internal_master --buildbot --branch master --buildnumber 2687 --repo-cache /var/cache/chrome-infra/ccompute-setup/cros-internal --master-build-id 897914 --resume --timeout 0 --notee --nocgroups --buildroot /b/cbuild/internal_master --version 8452.0.0 --metadata_dump /tmp/cbuildbot-tmpJaEU7o/metadataoBUV9g
                                                      `-upload_symbols,27183 /b/cbuild/internal_master/chromite/bin/upload_symbols --yes --dedupe --root /b/cbuild/internal_master/chroot --board butterfly --official_build --failed-list /b/cbuild/internal_master/buildbot_archive/butterfly-release/R53-8452.0.0/failed_upload_symbols.list
                                                          `-upload_symbols,28645 /b/cbuild/internal_master/chromite/bin/upload_symbols --yes --dedupe --root /b/cbuild/internal_master/chroot --board butterfly --official_build --failed-list /b/cbuild/internal_master/buildbot_archive/butterfly-release/R53-8452.0.0/failed_upload_symbols.list
                                                              `-{upload_symbols},2247

04:59:42: WARNING: RunCommand: lsof -p 28645
04:59:42: WARNING: (stdout):
COMMAND     PID       USER   FD   TYPE   DEVICE SIZE/OFF     NODE NAME
upload_sy 28645 chrome-bot  cwd    DIR      8,1     4096  9571491 /b/cbuild/internal_master
upload_sy 28645 chrome-bot  rtd    DIR      8,1     4096        2 /
upload_sy 28645 chrome-bot  txt    REG      8,1  3345416     4421 /usr/bin/python2.7
upload_sy 28645 chrome-bot  mem    REG      8,1   101240     3139 /lib/x86_64-linux-gnu/libresolv-2.19.so
upload_sy 28645 chrome-bot  mem    REG      8,1    22952     4274 /lib/x86_64-linux-gnu/libnss_dns-2.19.so
upload_sy 28645 chrome-bot  mem    REG      8,1    43616     4287 /lib/x86_64-linux-gnu/libnss_files-2.19.so
upload_sy 28645 chrome-bot  mem    REG      8,1    25040     4467 /usr/lib/python2.7/lib-dynload/mmap.x86_64-linux-gnu.so
upload_sy 28645 chrome-bot  mem    REG      8,1    31720  1052870 /usr/lib/python2.7/dist-packages/Crypto/Cipher/_AES.x86_64-linux-gnu.so
upload_sy 28645 chrome-bot  mem    REG      8,1    15584  1052816 /usr/lib/python2.7/dist-packages/Crypto/Util/_counter.x86_64-linux-gnu.so
upload_sy 28645 chrome-bot  mem    REG      8,1   475168     3064 /usr/lib/x86_64-linux-gnu/libgmp.so.10.1.3
upload_sy 28645 chrome-bot  mem    REG      8,1    73728  1052954 /usr/lib/python2.7/dist-packages/Crypto/PublicKey/_fastmath.x86_64-linux-gnu.so
upload_sy 28645 chrome-bot  mem    REG      8,1    66488    54184 /usr/lib/python2.7/dist-packages/OpenSSL/SSL.so
upload_sy 28645 chrome-bot  mem    REG      8,1    12280    54185 /usr/lib/python2.7/dist-packages/OpenSSL/rand.so
upload_sy 28645 chrome-bot  mem    REG      8,1    96776    54191 /usr/lib/python2.7/dist-packages/OpenSSL/crypto.so
upload_sy 28645 chrome-bot  mem    REG      8,1    18936     2009 /lib/x86_64-linux-gnu/libuuid.so.1.3.0
upload_sy 28645 chrome-bot  mem    REG      8,1    33448     4481 /usr/lib/python2.7/lib-dynload/_multiprocessing.x86_64-linux-gnu.so
upload_sy 28645 chrome-bot  mem    REG      8,1    54064     4496 /usr/lib/python2.7/lib-dynload/_json.x86_64-linux-gnu.so
upload_sy 28645 chrome-bot  mem    REG      8,1    30944    11074 /usr/lib/x86_64-linux-gnu/libffi.so.6.0.1
upload_sy 28645 chrome-bot  mem    REG      8,1   136232     4491 /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so
upload_sy 28645 chrome-bot  mem    REG      8,1    24648     4464 /usr/lib/python2.7/lib-dynload/termios.x86_64-linux-gnu.so
upload_sy 28645 chrome-bot  mem    REG      8,1   387256    17685 /lib/x86_64-linux-gnu/libssl.so.1.0.0
upload_sy 28645 chrome-bot  mem    REG      8,1    38480     4483 /usr/lib/python2.7/lib-dynload/_ssl.x86_64-linux-gnu.so
upload_sy 28645 chrome-bot  mem    REG      8,1  1934624    17684 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
upload_sy 28645 chrome-bot  mem    REG      8,1    20664     4474 /usr/lib/python2.7/lib-dynload/_hashlib.x86_64-linux-gnu.so
upload_sy 28645 chrome-bot  mem    REG      8,1   109232     4492 /usr/lib/python2.7/lib-dynload/datetime.x86_64-linux-gnu.so
upload_sy 28645 chrome-bot  mem    REG      8,1  4648960    50490 /usr/lib/locale/locale-archive
upload_sy 28645 chrome-bot  mem    REG      8,1  1071552     3141 /lib/x86_64-linux-gnu/libm-2.19.so
upload_sy 28645 chrome-bot  mem    REG      8,1   100728     2138 /lib/x86_64-linux-gnu/libz.so.1.2.8
upload_sy 28645 chrome-bot  mem    REG      8,1    10680     4286 /lib/x86_64-linux-gnu/libutil-2.19.so
upload_sy 28645 chrome-bot  mem    REG      8,1    14664     3195 /lib/x86_64-linux-gnu/libdl-2.19.so
upload_sy 28645 chrome-bot  mem    REG      8,1  1840928     4283 /lib/x86_64-linux-gnu/libc-2.19.so
upload_sy 28645 chrome-bot  mem    REG      8,1   141574     4046 /lib/x86_64-linux-gnu/libpthread-2.19.so
upload_sy 28645 chrome-bot  mem    REG      8,1   149120     4091 /lib/x86_64-linux-gnu/ld-2.19.so
upload_sy 28645 chrome-bot  DEL    REG     0,20          10587321 /run/shm/sem.is69DR
upload_sy 28645 chrome-bot  DEL    REG     0,20          10587320 /run/shm/sem.xMnOyy
upload_sy 28645 chrome-bot  DEL    REG     0,20          10587319 /run/shm/sem.AW5stf
upload_sy 28645 chrome-bot  DEL    REG     0,20          10587318 /run/shm/sem.eHt8nW
upload_sy 28645 chrome-bot  DEL    REG     0,20          10587317 /run/shm/sem.W40NiD
upload_sy 28645 chrome-bot  DEL    REG     0,20          10587316 /run/shm/sem.YmYtdk
upload_sy 28645 chrome-bot  DEL    REG     0,20          10587315 /run/shm/sem.UEHa80
upload_sy 28645 chrome-bot  DEL    REG     0,20          10587314 /run/shm/sem.fUPR2H
upload_sy 28645 chrome-bot  DEL    REG     0,20          10587313 /run/shm/sem.P3nzXo
upload_sy 28645 chrome-bot  DEL    REG     0,20          10587312 /run/shm/sem.8RohS5
upload_sy 28645 chrome-bot  DEL    REG     0,20          10588815 /run/shm/sem.rrI0MM
upload_sy 28645 chrome-bot  DEL    REG     0,20          10588814 /run/shm/sem.9me9Xt
upload_sy 28645 chrome-bot  DEL    REG     0,20          10588813 /run/shm/sem.Nh7h9a
upload_sy 28645 chrome-bot  DEL    REG     0,20          10588811 /run/shm/sem.AGwrkS
upload_sy 28645 chrome-bot  DEL    REG     0,20          10588810 /run/shm/sem.r6RBvz
upload_sy 28645 chrome-bot  DEL    REG      0,4          10588809 /dev/zero
upload_sy 28645 chrome-bot  DEL    REG     0,20          10588808 /run/shm/sem.5ATNGg
upload_sy 28645 chrome-bot  DEL    REG     0,20          10588807 /run/shm/sem.l3GoSX
upload_sy 28645 chrome-bot  DEL    REG     0,20          10588806 /run/shm/sem.Ov1Z3E
upload_sy 28645 chrome-bot    0r  FIFO      0,8      0t0    44972 pipe
upload_sy 28645 chrome-bot    1w   REG      8,1      347   402109 /tmp/cbuildbot-tmpJaEU7o/chromite-parallel-KFRmK4
upload_sy 28645 chrome-bot    2w   REG      8,1      347   402109 /tmp/cbuildbot-tmpJaEU7o/chromite-parallel-KFRmK4
upload_sy 28645 chrome-bot    3r  FIFO      0,8      0t0 10588805 pipe
upload_sy 28645 chrome-bot    4w  FIFO      0,8      0t0 10588805 pipe
upload_sy 28645 chrome-bot    5r  FIFO      0,8      0t0 10588812 pipe
upload_sy 28645 chrome-bot    6w  FIFO      0,8      0t0 10588812 pipe
upload_sy 28645 chrome-bot    7u   REG      8,1      347   402109 /tmp/cbuildbot-tmpJaEU7o/chromite-parallel-KFRmK4
upload_sy 28645 chrome-bot    8r   CHR      1,3      0t0     1029 /dev/null
upload_sy 28645 chrome-bot   12r   REG      8,1     2679 18893987 /b/cbuild/internal_master/chroot/build/butterfly/usr/lib/debug/breakpad/libbootstat.so/F256AAA6FDDE31952D4CC570B515144C0/libbootstat.so.sym
upload_sy 28645 chrome-bot   13r   CHR      1,9      0t0     1034 /dev/urandom
upload_sy 28645 chrome-bot   14u  IPv4 10591579      0t0      TCP cros-beefy17-c2.c.chromeos-bot.internal:55101->vl-in-f101.1e100.net:http (CLOSE_WAIT)
upload_sy 28645 chrome-bot   15r   REG      8,1    24615 18893601 /b/cbuild/internal_master/chroot/build/butterfly/usr/lib/debug/breakpad/test_assoc_shutdown/9A6F327A25D18FBA188EE891AC57604F0/test_assoc_shutdown.sym
upload_sy 28645 chrome-bot   16u  IPv4 10614061      0t0      TCP cros-beefy17-c2.c.chromeos-bot.internal:48086->vl-in-f139.1e100.net:http (CLOSE_WAIT)
upload_sy 28645 chrome-bot   17r   REG      8,1    60378 18892924 /b/cbuild/internal_master/chroot/build/butterfly/usr/lib/debug/breakpad/lchown03_16/08752818CADAFF8C2753F4A4C808650E0/lchown03_16.sym
upload_sy 28645 chrome-bot   18u  IPv4 10614039      0t0      TCP cros-beefy17-c2.c.chromeos-bot.internal:55108->vl-in-f101.1e100.net:http (CLOSE_WAIT)
upload_sy 28645 chrome-bot   19u  IPv4 10622090      0t0      TCP cros-beefy17-c2.c.chromeos-bot.internal:55121->vl-in-f101.1e100.net:http (CLOSE_WAIT)
upload_sy 28645 chrome-bot   20r   REG      8,1    12929 18894993 /b/cbuild/internal_master/chroot/build/butterfly/usr/lib/debug/breakpad/testopt/9FBD4BF371F9541227702A6E34D1778E0/testopt.sym
upload_sy 28645 chrome-bot   21u  IPv4 10628129      0t0      TCP cros-beefy17-c2.c.chromeos-bot.internal:48097->vl-in-f139.1e100.net:http (CLOSE_WAIT)
upload_sy 28645 chrome-bot   22u  IPv4 10619359      0t0      TCP cros-beefy17-c2.c.chromeos-bot.internal:48105->vl-in-f139.1e100.net:http (CLOSE_WAIT)
upload_sy 28645 chrome-bot   24u  IPv4 10635478      0t0      TCP cros-beefy17-c2.c.chromeos-bot.internal:48104->vl-in-f139.1e100.net:http (CLOSE_WAIT)
upload_sy 28645 chrome-bot   27u  IPv4 10642503      0t0      TCP cros-beefy17-c2.c.chromeos-bot.internal:55140->vl-in-f101.1e100.net:http (CLOSE_WAIT)

04:59:42: WARNING: RunCommand: gdb --nx -q -p 28645 -ex 'set prompt'
04:59:42: WARNING: (stdout):
Attaching to process 28645

>>> info proc all

>>> info threads
No threads.

>>> thread apply all py-list

>>> thread apply all py-bt

>>> thread apply all bt

>>> detach
quit

04:59:42: WARNING: (stderr):
Could not attach to process.  If your uid matches the uid of the target
process, check the setting of /proc/sys/kernel/yama/ptrace_scope, or try
again as the root user.  For more details, see /etc/sysctl.d/10-ptrace.conf
ptrace: Operation not permitted.
No current process: you must name one.
The program is not being run.

05:00:12: WARNING: Killing 28645 (sig=15 SIGTERM)
05:01:12: WARNING: Killing 28645 (sig=9 SIGKILL)

STEP_FAILURE
05:01:12: WARNING: <_BackgroundTask(_BackgroundTask-4, started)> hung for 600 seconds
  File "/b/cbuild/internal_master/chromite/bin/upload_symbols", line 164, in <module>
    commandline.ScriptWrapperMain(FindTarget)
  File "/b/cbuild/internal_master/chromite/lib/commandline.py", line 834, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/b/cbuild/internal_master/chromite/scripts/upload_symbols.py", line 889, in main
    product_name=opts.product_name)
  File "/b/cbuild/internal_master/chromite/scripts/upload_symbols.py", line 714, in UploadSymbols
    _Upload(queue, counters, dedupe_list)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 749, in BackgroundTaskRunner
    queue.put(_AllTasksComplete())
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 545, in ParallelTasks
    task_errors = task.Wait()
  File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 362, in Wait
    traceback.print_stack()
05:01:12: INFO: finished uploading; joining background process
05:01:12: INFO: waiting up to 20 minutes for ~1 notifications
05:01:12: INFO: dedupe notification finished; exiting
upload_symbols: Unhandled exception:
Traceback (most recent call last):
  File "/b/cbuild/internal_master/chromite/bin/upload_symbols", line 164, in <module>
    commandline.ScriptWrapperMain(FindTarget)
  File "/b/cbuild/internal_master/chromite/lib/commandline.py", line 834, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/b/cbuild/internal_master/chromite/scripts/upload_symbols.py", line 889, in main
    product_name=opts.product_name)
  File "/b/cbuild/internal_master/chromite/scripts/upload_symbols.py", line 714, in UploadSymbols
    _Upload(queue, counters, dedupe_list)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 749, in BackgroundTaskRunner
    queue.put(_AllTasksComplete())
  File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 749, in BackgroundTaskRunner
    queue.put(_AllTasksComplete())
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 560, in ParallelTasks
    raise BackgroundFailure(exc_infos=errors)
chromite.lib.parallel.BackgroundFailure: <class 'chromite.lib.parallel.ProcessExitTimeout'>: <_BackgroundTask(_BackgroundTask-4, started)> hung for 600 seconds

Comment 1 by dgarr...@chromium.org, Jun 15 2016 

It's consistently on the final group of symbols processed in that background task, while waiting for the background running to shutdown.

So... there is some kind of cleanup that's hanging.

Comment 2 by dgarr...@chromium.org, Jun 15 2016

Blocking: 610902

Comment 3 by gkihumba@google.com, Jun 15 2016 

I've been able to use upload_symbols with --yes (--dedupe) and didn't have this issue. I think Mike did too a while back.

Comment 4 by vapier@chromium.org, Jun 15 2016 

this hang/crash is why we had to turn it off in the past.  was never able to root cause the issue (or really be able to reproduce it to try and triage in the first place).

Comment 5 by dgarr...@chromium.org, Jun 15 2016 

I have a big script revamp in-progress that removes much of the multiprocess stuff that might help.

Performance will suffer some, but I don't think it'll be a problem.

Comment 6 by autumn@chromium.org, Jun 21 2016

Labels: -current-issue

Comment 7 by dgarr...@chromium.org, Jun 30 2016

Mergedinto: 610902
Status: Duplicate (was: Started)
►

Sign in to add a comment