New issue
Advanced search Search tips

Issue 681756 link

Starred by 1 user

Issue metadata

Status: Verified
Owner:
Closed: Mar 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Feature



Sign in to add a comment

Send alert emails when tests failures exceed a threshold

Project Member Reported by wuchengli@chromium.org, Jan 17 2017

Issue description

Some tests are flaky and we don't have the time to fix them all. So we ended up muting the alert emails because there were too many. I still want to be notified if the failures are stable. For example, video_VideoDecodeAccelerator fails on all devices or fails on one device for three builds in a row. Maybe we can write a script to filter emails from crosinfra@gmail.com.

https://groups.google.com/a/google.com/forum/#!topic/chromeos-video-test-failures/bEc_KffiYRA
 
Owner: johnylin@chromium.org
Status: Assigned (was: Available)
It looks like we can use google app script to filter the emails in chromeos-video-test-failures@google.com.
https://developers.google.com/apps-script/reference/gmail/
https://developers.google.com/gmail/api/quickstart/apps-script
There's time-driven trigger similar to a cron job.
https://developers.google.com/apps-script/guides/triggers/installable
Thanks for the information.

The idea we draft first is to combine Google-App-Script to Gmail. I could run a script on chromeos-test-video-failure archived mails of my account, to parse the valuable information.
Pawel has an idea. We can make a summary of test failures like the number of devices and the number of failure in a row. 

Comment 5 by vsu...@google.com, Jan 26 2017

Cc: vsu...@chromium.org
Johny. Any update?
Code implementation is on-going.

My idea now is using Google Apps Script to parse the following two steps:
aggregate cros mails --(1)-> (SpreadSheet) dashboard --(2)-> (Mail) sends alert mails

Step(1) coding is finished.

Here is the implkementation:
https://drive.google.com/open?id=1W2VTwjuTniTm6OCWH2ViXOzXHgpGkFbpmiIyyOh_4tlgNt6hlrXSTSFW

I finished an alpha version of mail parser.

I just found crosinfra doesn't autofile failure mails for suite:bvt-perbuild (https://bugs.chromium.org/p/chromium/issues/detail?id=694139), if this could be fixed then we can get both suite:video and suite:bvt-perbuild failures.
There are two bugs tracking why bvt-cq and bvt-perbuild didn't send mails to chromeos-video-test-failures@google.com
https://bugs.chromium.org/p/chromium/issues/detail?id=694139
https://bugs.chromium.org/p/chromium/issues/detail?id=697742

Since we still have many test failures, I set up a priority test group and right now it will report test failures in the priority group only (for others it still collects data, but not report on alert mail)

Priority group right now:
["video_VideoEncodeAccelerator",
"video_VideoDecodeAccelerator",
"video_ChromeRTCHWDecodeUsed",
"video_VEAPerf",
"video_JpegDecodeAccelerator"]

Please add video_VideoDecodeAccelerator to video suite before 694139 is fixed.
Status: Fixed (was: Assigned)

Comment 12 by dchan@google.com, May 30 2017

Labels: VerifyIn-60
Labels: VerifyIn-61
Cc: avkodipelli@chromium.org
Status: Verified (was: Fixed)

Sign in to add a comment