New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 694139 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Mar 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

Enable failure emails for bvt-perbuild suite

Project Member Reported by wuchengli@chromium.org, Feb 20 2017

Issue description

We found test suite:video sent failures emails. But suite:bvt-perbuild didn't. Is there a setting in cros infra that needs to be changed? video_VideoDecodeAccelerator is in bvt-perbuild. video_VideoEncodeAccelerator is in suite:video.

Here's a sample of the email.

This report is automatically generated to track the following Failure: 
Test: video_VideoEncodeAccelerator. 
Suite: video. 
Chrome Version: 58.0.3015.0. 
Build: elm-release/R58-9296.0.0. 

Reason: 
video_encode_accelerator_unittest failed! Command <HOME=/tmp/tmpFfsdY6 
CR_SOURCE_ROOT=/usr/local/autotest/deps/chrome_test/test_src 
CHROME_DEVEL_SANDBOX=/opt/google/chrome/chrome-sandbox 
/usr/local/autotest/deps/chrome_test/test_src/out/Release/video_encode_accelerator_unittest 
--test_stream_data="/usr/local/autotest/tmp/tmpbPtG2H_video_VideoEncodeAccelerator/tulip2-320x180-55be7124b3aec1b72bfb57f433297193.yuv:320:180:1:/usr/local/autotest/tmp/tmpbPtG2H_video_VideoEncodeAccelerator/tulip2-320x180-55be7124b3aec1b72bfb57f433297193.yuv.out:100000" 
--ozone-platform=gbm> failed, rc=1, Command returned non-zero exit 
status. 
build artifacts: 
https://storage.cloud.google.com/?arg=chromeos-image-archive/elm-release/R58-9296.0.0. 
results log: http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/102301366-chromeos-test/chromeos2-row7-rack10-host19/debug/. 
status log: http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/102301366-chromeos-test/chromeos2-row7-rack10-host19/status.log. 
buildbot stages: NA. 
job link: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=102301366. 

You may want to check the test history on wmatrix: 
https://wmatrix.googleplex.com/unfiltered?hide_missing=True&tests=video_VideoEncodeAccelerator 
You may also want to check the test retry dashboard in case this is a 
flakey test: https://wmatrix.googleplex.com/retry_teststats/?days_back=30&tests=video_VideoEncodeAccelerator 
 
Components: Infra>Client>ChromeOS
I think we disabled email notifications for bvt-perbuild as there was no active monitoring of auto-filed bugs and the filing frequency was very high.
Cc: dchan@chromium.org
We don't need auto-filed bugs. We only want the email notifications for bvt-perbuild. Is it OK to turn it on? Rohit. Who's the right owner for it?
Labels: videoshortlist
Labels: -Pri-2 Pri-1
I found control.bvt-perbuild has another bug template (#1).

According to #2, 
The settings in this bug template will be merged to the bug template defined in suite control file. That is, labels and cc defined in both test and suite control files will be merged and applied to the bug filed." It looks like the bug template in test control is not applied.

#1: https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/test_suites/control.bvt-perbuild?dr=C

#2: https://www.chromium.org/chromium-os/testing/test-suites
Cc: dshi@chromium.org ayatane@chromium.org
Summary: BUG_TEMPLATE not working if the suite is bvt-perbuild (was: tests in suite:bvt-perbuild doesn't send failure emails)
Add dshi and ayatane.

video_VideoDecodeAccelerator has a BUG_TEMPLATE that cc chromeos-video-test-failures@google.com. But we haven't received any emails. Do you know why?

https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/client/site_tests/video_VideoDecodeAccelerator/control.vp8?dr=C&l=24
Cc: fdeng@chromium.org

Comment 9 Deleted

Comment 10 by dshi@chromium.org, Mar 1 2017

Are you referring to the task:
[NightlyStormBvtPerBuild]
run_on: nightly
hour: 14
suite: bvt-perbuild
branch_specs: ==tot
boards: storm
pool: suites

It needs a "file_bugs: True" setting to enable bug filing
Re #3: AFAIK, Dan is the one :-)
Re 10: I don't need bug filing. I want to know why emails are not sent to cc list of video_VideoDecodeAccelerator.

By the way, I cannot find which source file contains the task [NightlyStormBvtPerBuild]. Which file is that?
I found an old email syaing "Recently we disabled the auto-filing bug feature for all suites except the tree-closing suites ('au', 'bvt-.*', 'paygen_.*', 'sanity'), all the other suites will default to just send email notifications to groups that developers had specified in the bug template when the tests fail."

This explained why video suite has file_bugs=True but it doesn't have auto-filed bugs. We still receive failure emails.
I found the suites that will file bugs.
_FILE_BUG_SUITES = ['au', 'bvt', 'bvt-cq', 'bvt-inline', 'paygen_au_beta',
                    'paygen_au_canary', 'paygen_au_dev', 'paygen_au_stable',
                    'sanity', 'push_to_prod']

https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/server/cros/dynamic_suite/suite.py?q=package:%5Echromeos_public$+%5C%27bvt-&l=44&dr=C
Owner: johnylin@chromium.org
Status: Assigned (was: Available)
Summary: Enable failure emails for bvt-perbuild suite (was: BUG_TEMPLATE not working if the suite is bvt-perbuild)
This means we can set file_bugs=True for bvt-perbuild. Theoretically we will receive failure emails and won't have auto-filing bugs. That's exactly what we need. But maybe we need to tell people before we merge the CL.
https://cs.corp.google.com/search/?q=package:chromeos_public+suite:bvt-perbuild+BUG_TEMPLATE+%22%27cc%27%22&type=cs

According to code-search except for video tests there are only two affected tests:
logging_CrashServices
'cc': ['vapier@chromium.org', 'benchan@chromium.org'],

autoupdate_CatchBadSignatures
'cc': ['chromeos-installer-alerts@google.com'],

We could tell them in advance.
https://chrome-internal-review.googlesource.com/c/332643/ is in review.

Re 16: Yes. Please notify them.

Comment 18 by dshi@chromium.org, Mar 2 2017

re #14
right, you need to add bvt-perbuild to that list.
re #18

Since what we expect is to receive failure mails and not auto-file bugs, I think what we need to do is to add "file_bugs: True" in NightlyStormBvtPerBuild (as CL below), not add bvt-perbuild to _FILE_BUG_SUITES list.

Is this correct?

https://chrome-internal-review.googlesource.com/c/332643/

Comment 20 by dshi@chromium.org, Mar 2 2017

Re #19
that's correct, no need to add the suite to that list.
Project Member

Comment 21 by bugdroid1@chromium.org, Mar 2 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/624a8116b063a7e34db30a1bb0586a2bca53cb32

commit 624a8116b063a7e34db30a1bb0586a2bca53cb32
Author: Pin-chih Lin <johnylin@google.com>
Date: Thu Mar 02 10:45:31 2017

Status: Fixed (was: Assigned)
Dan. I don't see any test failure email yet for video_VideoDecodeAccelerator. Is there anything else to be done?
Status: Assigned (was: Fixed)
There is an example (we should receive failure mails in chromeos-video-test-failures@google.com but still no)

test:video_VideoDecodeAccelerator
platform:guado
build:R58-9332.0.0
https://wmatrix.googleplex.com/testrun/unfiltered?test_ids=442366635

Check suite log autoserv.DEBUG under this link
https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/104414300-chromeos-test/hostless/debug
03/02 15:30:48.875 INFO |        server_job:0184| END FAIL	104436633-chromeos-test/chromeos4-row3-rack8-host3/video_VideoDecodeAccelerator	video_VideoDecodeAccelerator	timestamp=1488497383	localtime=Mar 02 15:29:43
......
03/02 15:30:50.754 DEBUG|         gmail_lib:0120| Email sent: 15a915c9dcfd0747

"Email sent" should mean it had sent mails to someone else, but not chromeos-video-test-failures@google.com

And actually there is a mail in crosinfra2@gmail.com mailbox but the recipient is empty. (thanks for dshi@'s help)
---------------------------------------------------
From: <crosinfra2@gmail.com>
Date: Thu, Mar 2, 2017 at 3:30 PM
Subject: [bvt-perbuild] video_VideoDecodeAccelerator Failure on guado-release/R58-9332.0.0
To: 
---------------------------------------------------



Additional note: how to get suite log?
step1. Go to "details" page of failure test in Wmatrix
step2. Click "Job page"
step3. Click "parent job id", then click on "Debug logs" under "Hosts for this job"
Project Member

Comment 26 by bugdroid1@chromium.org, Mar 9 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/67c222ff0b93a75d9bbacb4fe429a3b2602cfa7d

commit 67c222ff0b93a75d9bbacb4fe429a3b2602cfa7d
Author: Pin-chih Lin <johnylin@google.com>
Date: Thu Mar 09 04:50:30 2017

Status: Fixed (was: Assigned)

Sign in to add a comment