This looks like the cron entry:

# Send sheriff reminder emails every day.
MAILTO="mmoss@google.com"
0 3 * * Mon-Fri /home/chrome-bot/buildbot/build_internal/scripts/tools/sheriff/reminder_emailer.py --oauth_file /home/chrome-bot/buildbot/build_internal/scripts/tools/sheriff/.sheriff_oauth --message_from_address chrome-team-noreply@chromium.org --config chrome --config bling_europe --config bling_us --config chrome --config chromeos --config chromeos_lab --config chromeos.other --config chromeos_test --config clank --config clank_merge --config gpu --config nacl --config network --config troopers --config valgrind --config webkit --config perf --loglevel error

Errors are mailed to mmoss, unfortunately I think that means there are no error logs on the master.

Are you sure that's the right cron entry? I thought we got rid of all the separate "--config" args, so we didn't have to change cron every time we added/removed a rotation. Also, I haven't gotten any cron failure emails.

OK, that cron is definitely old. hinoka@, where did you come up with that? If that's still live somewhere, it should really get cleaned up.

Here's the right one:

https://chrome-internal.googlesource.com/infra/puppet.git/+/master/puppetm/etc/puppet/modules/chrome_infra/files/setup/cron/master1.golo.chromium.org/chrome-infra-cron#82

and looks like emails should go to chrome-golo-alerts@google.com

It could be caused by the recent master split. I remember being unsure about whether or not we needed to copy those over.

crontab -e was empty, but i saw a cron file in the homedir and thought that it might be in use.  If it is infact not the right cron file, then it should probably be removed to avoid further confusing.

confusion*

Assigning to hinoka@, since he's been looking into this. Please re-add to trooper queue with a status update if you are no longer working on this, or close if it's fixed!

We're having the same issue with network.rotation.json.  It looks like it may have coincided with extending the rotation.  No one's been sheriffing for the past 9 days as a result.  This seems like a P-1 to me.

Same issue for monorail rotation, causing rotation to be missed

Hinoka, any update?

Assigning to benhenry@, the listed owner for sheriff rotations.  Can you take a look?
https://chrome-internal.googlesource.com/infra/infra_internal/+/master/doc/services.md#sheriff-trooper-rotation

I don't know anything about a master split.

Nope, sorry. How about Stephen.

I'm not an owner of this. I'll look at this though.

https://groups.google.com/a/google.com/forum/#!topic/chrome-golo-alerts/PZIFfUompPA looks very suspicious

I think it's a bad config. CL out: https://chrome-internal-review.googlesource.com/c/357943/

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/infra_internal/+/7cf1479a586e27aca8157faf3d28d5ddaef720b0

commit 7cf1479a586e27aca8157faf3d28d5ddaef720b0
Author: Stephen Martinis <martiniss@google.com>
Date: Wed Apr 19 22:09:57 2017

The cron should run again tonight (3 AM PST), and it should pass this time. The thread linked in #14 will get another email if it fails again.

Can one bad rotation config file make other rotations stop sending emails?  I assume that file's only used when updating that particular rotation, and not parsed when updating other rotations.

I'm pretty sure it'll stop all rotations from sending emails. The logic for sending emails is here: https://chrome-internal.googlesource.com/infra/infra_internal.git/+/master/infra_internal/services/sheriff/reminder_emailer.py

It loops through all known configs. If it hits an error, I think it fails.

mmoss@ would know more though. He wrote that code.

Thanks!  Makes sense, I just assumed all of that was configured on rotation update, and integrated into gcal's reminder feature somehow.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/infra_internal/+/3dcf8425f5eb0156fb92cf19f3bdc40f91de2fb6

commit 3dcf8425f5eb0156fb92cf19f3bdc40f91de2fb6
Author: John Budorick <jbudorick@google.com>
Date: Thu Apr 20 00:08:13 2017