New issue
Advanced search Search tips

Issue 848781 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Jun 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

Tryjobs no longer sending email

Project Member Reported by mikenichols@chromium.org, Jun 1 2018

Issue description

There have been a series of changes, both to the Chromite libraries as well as luci-notify, related to tryjobs and emails.  We discovered yesterday, in testing, that we are no longer seeing email reports for trjobs.  

Now that email templates have been rolled out in luci-notify, need to pinpoint the root cause and resolve so we can being utilizing the new functionality. 

 
Let's send a quick email PSA to chromeos-infra-discuss and chromium-os-dev.
An example tryjob from TOT (no email expected):
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8944904522481847056

It's buildbucket entry includes:

"parameters_json": "{\"builder_name\": \"Prod\", \"email_notify\": [{\"email\": \"dgarrett@google.com\"}], \"properties\": {\"cbb_branch\": \"master\", \"cbb_config\": \"success-build\", \"cbb_display_label\": \"tryjob\", \"cbb_email\": \"dgarrett@google.com\", \"cbb_extra_args\": [\"--remote-trybot\", \"-b\", \"master\"]}}",
  "id": "8944904522481847056",


Here is an example tryjob that DID generate email:
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8946229564617436496


"parameters_json": "{\"builder_name\": \"Try\", \"email_notify\": [{\"email\": \"dgarrett@google.com\"}], \"properties\": {\"cbb_branch\": \"master\", \"cbb_config\": \"chromeos-infra-go-tryjob\", \"cbb_display_label\": \"tryjob\", \"cbb_email\": \"dgarrett@google.com\", \"cbb_extra_args\": [\"--remote-trybot\", \"-b\", \"master\"]}}",
Project Member

Comment 4 by bugdroid1@chromium.org, Jun 1 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/7057582a2a99cbcd89108b3841d54d8a678f52ac

commit 7057582a2a99cbcd89108b3841d54d8a678f52ac
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Jun 01 19:06:41 2018

[luci-notify] Log build id in pubsub handler

Log id of the build being processed in a pubsub handler.

Bug:  848781 
Change-Id: Ie7d69659debd3b41edcaf9236a735253bef1dedd
Reviewed-on: https://chromium-review.googlesource.com/1082957
Reviewed-by: Mike Nichols <mikenichols@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/7057582a2a99cbcd89108b3841d54d8a678f52ac/luci_notify/notify/pubsub.go

Comment 5 by no...@chromium.org, Jun 1 2018

luci-notify now requires luci-notify.cfg to be in the project, even if it does not define notifiers. This is how it determines that luci-notify is enabled for the project. Please add one, perhaps with a comment.

Sorry I didn't realize chromeos does no have luci-notify.cfg
Project Member

Comment 7 by bugdroid1@chromium.org, Jun 1 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/a26863935bedd43c7183298b9dd6ccadb521070f

commit a26863935bedd43c7183298b9dd6ccadb521070f
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Jun 01 20:03:31 2018

[luci-notify] require luci-notify.cfg

Update docs to explain that luci-notify.cfg is required for email templates
to work, even if it does not define any notifiers.

Bug:  848781 
Change-Id: I5613288592adf526be47199898af5af8d8caae11
Reviewed-on: https://chromium-review.googlesource.com/1082984
Reviewed-by: Mike Nichols <mikenichols@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/a26863935bedd43c7183298b9dd6ccadb521070f/luci_notify/doc/email_templates.md

Project Member

Comment 8 by bugdroid1@chromium.org, Jun 1 2018

Labels: merge-merged-config
The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/manifest-internal/+/d5772ce62dec4dbd135cc2732f54aee13cc5d6b3

commit d5772ce62dec4dbd135cc2732f54aee13cc5d6b3
Author: Mike Nichols <mikenichols@chromium.org>
Date: Fri Jun 01 22:12:40 2018

I was hoping that implementing the luci-notify.cfg would resolve the issue, but the latest tryjob that I kicked off never delivered a notification:

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8944883052375054160
https://ci.chromium.org/p/chromeos/builds/b8944883052375054160#
https://apis-explorer.appspot.com/apis-explorer/?base=https://cr-buildbucket.appspot.com/_ah/api#p/buildbucket/v1/buildbucket.get?id=8944883052375054160&_h=1&

I checked both the request log and activity log for luci-notify but never found a matching entry for the buildbucket ID.  

Will continue to dig and see what I can find.

-- Mike
Cc: no...@chromium.org
Nodir discovered a permissions issue with the Chromeos config (chromeos is internal) and has updating the permissions to allow luci-config to access. 

Another job has been kicked off to test.

-- Mike
Status: Fixed (was: Assigned)
This is now resolved.  There is an issue with the default template, that I just pushed a change to fix, but otherwise this is now working as expected. 

Thanks Nodir for your assistance in getting this resolved!

-- Mike

Sign in to add a comment