Issue 732496 link

Starred by 1 user

Issue metadata

Status:	Untriaged
Owner:	----
Cc:	█ yixiang@chromium.org gwendal@chromium.org █ jrbarnette@chromium.org █ sureshraj@chromium.org grundler@chromium.org hywu@chromium.org chromeos-continuous-integration-fire-hose@google.com
Components:	Infra>ChromeOS>CI
EstimatedDays:	----
NextAction:	----
OS:	Chrome
Pri:	3
Type:	Bug

PaygenTestCanary stage failing on whirlwind-release and arkham-release

Project Member Reported by sureshraj@chromium.org, Jun 12 2017

Issue description

https://uberchromegw.corp.google.com/i/chromeos/builders/whirlwind-release/builds/1938/steps/PaygenTestCanary

https://uberchromegw.corp.google.com/i/chromeos/builders/arkham-release/builds/1209/steps/PaygenTestCanary

The failure does seem to be flaky as it resolved itself in the latest whirlwind-release build
https://uberchromegw.corp.google.com/i/chromeos/builders/whirlwind-release/builds/1939/steps/PaygenTestCanary

Comment 1 by sureshraj@chromium.org, Jun 12 2017

Cc: jrbarnette@chromium.org

Comment 2 by sureshraj@chromium.org, Jun 12 2017

Sample log from one of the above failures (arkham-release). Complete log attached.

<SNIP>
03:30:38: INFO: Running cidb query on pid 9249, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f89583dbc10>
Preconditions for the stage successfully met. Beginning to execute stage...
03:30:38: INFO: Running cidb query on pid 9249, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f89583dbdd0>
03:30:38: INFO: RunCommand: /b/cbuild/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /tmp/cbuildbot-tmpl3ULu9/tmpkYKbeo/temp_summary.json --raw-cmd --task-name whirlwind-release/R61-9642.0.0-paygen_au_canary --dimension os Ubuntu-14.04 --dimension pool default --print-status-updates --timeout 14400 --io-timeout 14400 --hard-timeout 14400 --expiration 1200 '--tags=priority:Build' '--tags=suite:paygen_au_canary' '--tags=build:whirlwind-release/R61-9642.0.0' '--tags=task_name:whirlwind-release/R61-9642.0.0-paygen_au_canary' '--tags=board:whirlwind' -- /usr/local/autotest/site_utils/run_suite.py --build whirlwind-release/R61-9642.0.0 --board whirlwind --suite_name paygen_au_canary --pool bvt --file_bugs True --priority Build --timeout_mins 180 --retry True --suite_min_duts 2 -c
Autotest instance: cautotest
06-12-2017 [03:30:46] Submitted create_suite_job rpc
06-12-2017 [03:30:53] Created suite job: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=122926343
@@@STEP_LINK@Link to suite@http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=122926343@@@
--create_and_return was specified, terminating now.
Will return from run_suite with status: OK
03:30:54: INFO: RunCommand: /b/cbuild/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /tmp/cbuildbot-tmpl3ULu9/tmpNA7thy/temp_summary.json --raw-cmd --task-name whirlwind-release/R61-9642.0.0-paygen_au_canary --dimension os Ubuntu-14.04 --dimension pool default --print-status-updates --timeout 14400 --io-timeout 14400 --hard-timeout 14400 --expiration 1200 '--tags=priority:Build' '--tags=suite:paygen_au_canary' '--tags=build:whirlwind-release/R61-9642.0.0' '--tags=task_name:whirlwind-release/R61-9642.0.0-paygen_au_canary' '--tags=board:whirlwind' -- /usr/local/autotest/site_utils/run_suite.py --build whirlwind-release/R61-9642.0.0 --board whirlwind --suite_name paygen_au_canary --pool bvt --file_bugs True --priority Build --timeout_mins 180 --retry True --suite_min_duts 2 -m 122926343
05:54:25: WARNING: Killing tasks: [<_BackgroundTask(_BackgroundTask-7:6:7:3, started)>]
05:54:25: WARNING: Killing 9249 (sig=24 SIGXCPU)
05:54:25: WARNING: RunCommand: pstree -Apals 9249
05:54:25: WARNING: (stdout):
init,1
  `-chromebuild-sta,1050 -u /opt/chromebuild/chromebuild-startup.py --log /var/log/messages/chromebuild/startup.log
      `-python,1586 /opt/infra-bot-setup/infra-python/run.py infra.tools.bot_setup.start --root_dir /b --password_file /home/chrome-bot/.password_json --slave_name cros-beefy80-c2
          `-python,3047 /b/build/slave/run_slave.py --no_save --no-gclient-sync --python buildbot.tac --nodaemon --logfile twistd.log
              `-python,19769 -u ../../../scripts/slave/annotated_run.py --use-factory-properties-from-disk --build-properties-gz=eNqtU01vozAQ/Ssrn0MMDp8+7V566mmlPTVVZMyUuAFMbEMUVfnvO2BoU3V7Wy7Y896beTMMb6RsRAuNso7wp+cNKY3o5JFwQvA8qKYqtfvz+xEDR+d6yykdSjDyaHQL9WUrtem3tdZ1A3huqaIe0pa+JxjkCTA7edv7wJ7wH/N5ik+XPWmFdWC2q3ZPNhiUBoSD6lBePWmwYHgcp3ESRrs0TtOgPWk2RHWaNJk6W20TGRauUwjC67nOkp8VjNDoHlPXKB6VBCGlHjo3mf1cxVlfJYqLjCV5EhVpEidZ6Fmq8mheZFkYxrsUXbCCpYzFntCAsHA4weK1yNMsjOJ4QZ2o5/RP2LYWpuKXozLNRXWVx+e5HNy1By5FJ8z1LoxNv7MDA3OhO9iC43L9UtQPcnmtbBolOT5sabgsD/4rc0/7CEvdvaj6u3ITwysO3q+q+KfMHuQPopkkz7fbugJgOlwyXIEvmVdGN7S4VoRHxS7f3FlEjc9KNncOv8n0D4NkdejRURglumkZ/ZhR5AWLv3UBMd4b/QrS+T/BQK+tchoVy/08AOqqX0hYN4axDwBbIf9xVeeSo7JKd96AbcQIq2mjbVACvFzzMJAM0Ys2p0pNHmhJ51HQWUC/TC1Ypnv7C+Q5V6M=
                  `-python,20538 ../../../scripts/slave/annotated_run.py --use-factory-properties-from-disk --build-properties-gz=eNqtU01vozAQ/Ssrn0MMDp8+7V566mmlPTVVZMyUuAFMbEMUVfnvO2BoU3V7Wy7Y896beTMMb6RsRAuNso7wp+cNKY3o5JFwQvA8qKYqtfvz+xEDR+d6yykdSjDyaHQL9WUrtem3tdZ1A3huqaIe0pa+JxjkCTA7edv7wJ7wH/N5ik+XPWmFdWC2q3ZPNhiUBoSD6lBePWmwYHgcp3ESRrs0TtOgPWk2RHWaNJk6W20TGRauUwjC67nOkp8VjNDoHlPXKB6VBCGlHjo3mf1cxVlfJYqLjCV5EhVpEidZ6Fmq8mheZFkYxrsUXbCCpYzFntCAsHA4weK1yNMsjOJ4QZ2o5/RP2LYWpuKXozLNRXWVx+e5HNy1By5FJ8z1LoxNv7MDA3OhO9iC43L9UtQPcnmtbBolOT5sabgsD/4rc0/7CEvdvaj6u3ITwysO3q+q+KfMHuQPopkkz7fbugJgOlwyXIEvmVdGN7S4VoRHxS7f3FlEjc9KNncOv8n0D4NkdejRURglumkZ/ZhR5AWLv3UBMd4b/QrS+T/BQK+tchoVy/08AOqqX0hYN4axDwBbIf9xVeeSo7JKd96AbcQIq2mjbVACvFzzMJAM0Ys2p0pNHmhJ51HQWUC/TC1Ypnv7C+Q5V6M=
                      `-logdog_butler,20574 -log-level warning -project chromeos -prefix bb/chromeos/whirlwind-release/1938 -coordinator-host luci-logdog.appspot.com -output logdog -tag buildbot.master=chromeos -tag buildbot.builder=whirlwind-release -tag buildbot.buildnumber=1938 -tag logdog.viewer_url=https://luci-milo.appspot.com/buildbot/chromeos/whirlwind-release/1938 -service-account-json :gce -output-max-buffer-age 30s run -stdout tee=stdout -stderr tee=stderr -streamserver-uri unix:/b/build/.recipe_runtime/tmpMTXVat/butler.sock -- /b/build/slave/whirlwind-release-master/.recipe_cipd/logdog_annotee -log-level warning -name-base recipes -print-summary -tee annotations,text -json-args-path /b/build/slave/whirlwind-release-master/.recipe_runtime/tmpaSuYUb/logdog_annotee_cmd.json -result-path /b/build/slave/whirlwind-release-master/.recipe_runtime/tmpaSuYUb/bootstrap_result.json
                          `-logdog_annotee,20590 -log-level warning -name-base recipes -print-summary -tee annotations,text -json-args-path /b/build/slave/whirlwind-release-master/.recipe_runtime/tmpaSuYUb/logdog_annotee_cmd.json -result-path /b/build/slave/whirlwind-release-master/.recipe_runtime/tmpaSuYUb/bootstrap_result.json
                              `-python,20598 -u /b/build_internal/scripts/slave/recipes.py --verbose run --workdir=/b/build/slave/whirlwind-release-master/build --properties-file=/b/build/slave/whirlwind-release-master/.recipe_runtime/tmpaSuYUb/recipe_properties.json --output-result-json /b/build/slave/whirlwind-release-master/.recipe_runtime/tmpaSuYUb/recipe_result.json cros/cbuildbot_internal
                                  `-python,20627 -u /b/build_internal/scripts/slave/.recipe_deps/recipe_engine/recipes.py --package /b/build_internal/scripts/slave/infra/config/recipes.cfg --verbose run --workdir=/b/build/slave/whirlwind-release-master/build --properties-file=/b/build/slave/whirlwind-release-master/.recipe_runtime/tmpaSuYUb/recipe_properties.json --output-result-json /b/build/slave/whirlwind-release-master/.recipe_runtime/tmpaSuYUb/recipe_result.json cros/cbuildbot_internal
                                      `-python2,21302 /b/build/slave/whirlwind-release-master/build/chromite/scripts/cbuildbot_launch --buildroot /b/cbuild --buildbot --branch master --buildnumber 1938 --buildbucket-id 8977004366452926224 --master-build-id 1588882 --git-cache-dir /b/build/slave/cache/git --goma_dir /b/build/slave/cache/goma_client --goma_client_json /creds/service_accounts/service-account-goma-client.json whirlwind-release
                                          `-python2,13099 /b/cbuild/chromite/bin/cbuildbot whirlwind-release --buildroot /b/cbuild --buildbot --branch master --buildnumber 1938 --buildbucket-id 8977004366452926224 --master-build-id 1588882 --git-cache-dir /b/build/slave/cache/git --goma_dir /b/build/slave/cache/goma_client --goma_client_json /creds/service_accounts/service-account-goma-client.json
                                              `-python2,5258 chromite/bin/cbuildbot whirlwind-release --buildroot /b/cbuild --buildbot --branch master --buildnumber 1938 --buildbucket-id 8977004366452926224 --master-build-id 1588882 --git-cache-dir /b/build/slave/cache/git --goma_dir /b/build/slave/cache/goma_client --goma_client_json /creds/service_accounts/service-account-goma-client.json --resume --timeout 0 --notee --nocgroups --buildroot /b/cbuild --version 9642.0.0 --metadata_dump /tmp/cbuildbot-tmpl3ULu9/metadataLIzXeQ
                                                  `-python2,30409 chromite/bin/cbuildbot whirlwind-release --buildroot /b/cbuild --buildbot --branch master --buildnumber 1938 --buildbucket-id 8977004366452926224 --master-build-id 1588882 --git-cache-dir /b/build/slave/cache/git --goma_dir /b/build/slave/cache/goma_client --goma_client_json /creds/service_accounts/service-account-goma-client.json --resume --timeout 0 --notee --nocgroups --buildroot /b/cbuild --version 9642.0.0 --metadata_dump /tmp/cbuildbot-tmpl3ULu9/metadataLIzXeQ
                                                      `-python2,7325 chromite/bin/cbuildbot whirlwind-release --buildroot /b/cbuild --buildbot --branch master --buildnumber 1938 --buildbucket-id 8977004366452926224 --master-build-id 1588882 --git-cache-dir /b/build/slave/cache/git --goma_dir /b/build/slave/cache/goma_client --goma_client_json /creds/service_accounts/service-account-goma-client.json --resume --timeout 0 --notee --nocgroups --buildroot /b/cbuild --version 9642.0.0 --metadata_dump /tmp/cbuildbot-tmpl3ULu9/metadataLIzXeQ
                                                          `-python2,7560 chromite/bin/cbuildbot whirlwind-release --buildroot /b/cbuild --buildbot --branch master --buildnumber 1938 --buildbucket-id 8977004366452926224 --master-build-id 1588882 --git-cache-dir /b/build/slave/cache/git --goma_dir /b/build/slave/cache/goma_client --goma_client_json /creds/service_accounts/service-account-goma-client.json --resume --timeout 0 --notee --nocgroups --buildroot /b/cbuild --version 9642.0.0 --metadata_dump /tmp/cbuildbot-tmpl3ULu9/metadataLIzXeQ
                                                              `-python2,9249 chromite/bin/cbuildbot whirlwind-release --buildroot /b/cbuild --buildbot --branch master --buildnumber 1938 --buildbucket-id 8977004366452926224 --master-build-id 1588882 --git-cache-dir /b/build/slave/cache/git --goma_dir /b/build/slave/cache/goma_client --goma_client_json /creds/service_accounts/service-account-goma-client.json --resume --timeout 0 --notee --nocgroups --buildroot /b/cbuild --version 9642.0.0 --metadata_dump /tmp/cbuildbot-tmpl3ULu9/metadataLIzXeQ
                                                                  `-python,29681 /b/cbuild/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /tmp/cbuildbot-tmpl3ULu9/tmpNA7thy/temp_summary.json --raw-cmd --task-name whirlwind-release/R61-9642.0.0-paygen_au_canary --dimension os Ubuntu-14.04 --dimension pool default --print-status-updates --timeout 14400 --io-timeout 14400 --hard-timeout 14400 --expiration 1200 --tags=priority:Build --tags=suite:paygen_au_canary --tags=build:whirlwind-release/R61-9642.0.0 --tags=task_name:whirlwind-release/R61-9642.0.0-paygen_au_canary --tags=board:whirlwind -- /usr/local/autotest/site_utils/run_suite.py --build whirlwind-release/R61-9642.0.0 --board whirlwind --suite_name paygen_au_canary --pool bvt --file_bugs True --priority Build --timeout_mins 180 --retry True --suite_min_duts 2 -m 122926343
                                                                      `-{python},29786

05:54:25: WARNING: RunCommand: lsof -p 9249

</SNIP>