Suite job crashes on test retries by "Recipient address required" |
||||||
Issue descriptionWhen a suite job retries a test job, it tries to send a mail, and it makes the suite job crash. Suite job: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=125517051 Suite autoserv log: https://storage.cloud.google.com/chromeos-autotest-results/125517051-chromeos-test/hostless/debug/autoserv.DEBUG 06/28 04:40:29.529 DEBUG| retry_util:0201| ending retries with error: <class 'googleapiclient.errors.HttpError'>(<HttpError 400 when requesting https://www.googleapis.com/gmail/v1/users/me/messages/send?alt=json returned "Recipient address required">) 06/28 04:40:29.530 ERROR| suite:1292| Exception waiting for results Traceback (most recent call last): File "/usr/local/autotest/server/cros/dynamic_suite/suite.py", line 1289, in wait bug_template=bug_template) File "/usr/local/autotest/server/cros/dynamic_suite/suite.py", line 1339, in _record_result self._get_bug_template(result, bug_template)) File "/usr/local/autotest/server/cros/dynamic_suite/reporting.py", line 994, in send_email creds_path=site_utils.get_creds_abspath(EMAIL_CREDS_FILE)) File "/usr/local/autotest/site_utils/gmail_lib.py", line 175, in send_email backoff_factor=RETRY_BACKOFF_FACTOR) File "/usr/local/autotest/site-packages/chromite/lib/retry_util.py", line 243, in GenericRetry return _run() File "/usr/local/autotest/site-packages/chromite/lib/retry_util.py", line 176, in _Wrapper ret = func(*args, **kwargs) File "/usr/local/autotest/site-packages/chromite/lib/retry_util.py", line 242, in _run return functor(*args, **kwargs) File "/usr/local/autotest/site_utils/gmail_lib.py", line 153, in _run client.send_message(m, ignore_error=False) File "/usr/local/autotest/site_utils/gmail_lib.py", line 119, in send_message userId='me', body=message.get_payload()).execute() File "/usr/local/autotest/site-packages/chromite/third_party/oauth2client/util.py", line 140, in positional_wrapper return wrapped(*args, **kwargs) File "/usr/local/autotest/site-packages/chromite/third_party/googleapiclient/http.py", line 838, in execute raise HttpError(resp, content, uri=self.uri) HttpError: <HttpError 400 when requesting https://www.googleapis.com/gmail/v1/users/me/messages/send?alt=json returned "Recipient address required"> 06/28 04:40:29.531 INFO | server_job:0200| FAIL ---- bvt-arc timestamp=1498650029 localtime=Jun 28 04:40:29 Exception waiting for results
,
Jun 29 2017
,
Jun 29 2017
This issue makes bvt-arc suite unstable in particular, since CTS tests are often retried.
,
Jun 29 2017
The problem is in the bug template:
_BUG_TEMPLATE = {
'labels': ['bvt'],
'owner': '',
'status': None,
'summary': None,
'title': None,
}
owner is empty.
,
Jun 29 2017
,
Jun 29 2017
Hmm, many bug templates have empty owner. Is the behavior changed recently? [nya@slb test_suites]% git grep "'owner': ''" control.au: 'owner': '', control.bvt-arc: 'owner': '', control.bvt-cq: 'owner': '', control.bvt-inline: 'owner': '', control.bvt-perbuild: 'owner': '', control.cellular_mbim_compliance: 'owner': '', control.faft_bios: 'owner': '', control.faft_bios_au_1: 'owner': '', control.faft_bios_au_2: 'owner': '', control.faft_bios_au_3: 'owner': '', control.faft_bios_tot: 'owner': '', control.faft_ec: 'owner': '', control.faft_ec_au_1: 'owner': '', control.faft_ec_au_2: 'owner': '', control.faft_ec_au_3: 'owner': '', control.faft_ec_tot: 'owner': '', control.faft_flashrom: 'owner': '', control.faft_lab: 'owner': '', control.faft_lv1: 'owner': '', control.faft_lv2: 'owner': '', control.faft_lv3: 'owner': '', control.faft_lv4: 'owner': '', control.faft_lv5: 'owner': '', control.faft_normal: 'owner': '', control.faft_setup: 'owner': '', control.faft_smoke: 'owner': '', control.hotrod: 'owner': '', control.kernel_daily_benchmarks: 'owner': '', control.longevity: 'owner': '', control.longevity_two: 'owner': '', control.network3g: 'owner': '', control.network3g_att: 'owner': '', control.network3g_tmobile: 'owner': '', control.network3g_verizon: 'owner': '', control.network_nightly: 'owner': '', control.paygen_au_beta: 'owner': '', control.paygen_au_canary: 'owner': '', control.paygen_au_dev: 'owner': '', control.paygen_au_stable: 'owner': '', control.perf_v2: 'owner': '', control.power_daily: 'owner': '', control.power_requirements: 'owner': '', control.push_to_prod: 'owner': '', control.test_that_wrapper: 'owner': '', control.toolchain-tests: 'owner': '',
,
Jun 29 2017
owner can be empty. The bug template is meant to file bug, not just send out email. So it should still work without owner. On the other hand, the code should handle the empty owner case more gracefully. It should just post a warning and skip sending email (bug will still be filed).
,
Jun 29 2017
I see, then is this change enough? https://chromium-review.googlesource.com/c/554398/
,
Jun 29 2017
Above change is in CQ now.
,
Jun 29 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/b7a9ef7a60b26c3b51fdc69a3ae99662fb841dc4 commit b7a9ef7a60b26c3b51fdc69a3ae99662fb841dc4 Author: Shuhei Takahashi <nya@chromium.org> Date: Thu Jun 29 08:16:18 2017 reporting: Skip sending mails if there is no recipient. This will resolve the issue that a suite job crashes when it attempts to send a mail to no recipient. BUG= chromium:737836 TEST=run_pylint.py Change-Id: I38d7cfc155adb4546da3ea447026a6d153c1d197 Reviewed-on: https://chromium-review.googlesource.com/554398 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Dan Shi <dshi@google.com> [modify] https://crrev.com/b7a9ef7a60b26c3b51fdc69a3ae99662fb841dc4/server/cros/dynamic_suite/reporting.py
,
Jul 5 2017
,
Jan 22 2018
|
||||||
►
Sign in to add a comment |
||||||
Comment 1 by nya@chromium.org
, Jun 29 2017