New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 830054 link

Starred by 1 user
Status: Fixed
Owner:
█ jkop@chromium.org
Last visit > 30 days ago
Closed: Apr 2018
Cc:
akes...@chromium.org
█ jkop@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

crosinfra account unable to send e-mail

Reported by jrbarnette@chromium.org, Apr 6 2018 
The crosinfra account is unable to send e-mail.

The most recent lab-inventory run encountered this error:
2018-04-06 14:52:35 | WARNING    | Will retry error <HttpError 429 when requesting https://www.googleapis.com/gmail/v1/users/me/messages/send?alt=json returned "User-rate limit exceeded.  Retry after 2018-04-06T22:07:35.336Z (Mail sending)">

This is probably affecting all e-mail based notifications, not
just the lab inventory.  In particular, the infra-alerts alias has
been vewy vewy quiet since yesterday.

Comment 1 by jrbarnette@chromium.org, Apr 9 2018

Cc: akes...@chromium.org
Owner: jkop@chromium.org
Passing along.

Please note that currently, we're flying blind.  If there's an outage
being alerted, the mail can't get through.

Comment 2 by akes...@chromium.org, Apr 9 2018 

^ our monarch alerts should not be affected by this. That account is not used to send monarch alert emails.

Comment 3 by jrbarnette@chromium.org, Apr 9 2018 

> ^ our monarch alerts should not be affected by this. That account is not used to send monarch alert emails.

OK.  But the cros-infra account is still mysteriously silent.
Almost nothing has been sent since last Thursday; that's not
normal.

Comment 4 by jkop@chromium.org, Apr 9 2018 

Pcon queries are definitely broken, and with no alerts being sent, even the noisy useless ones, I suspect we _are_ flying blind. I'm filing another bug for that now.

Comment 5 by jkop@chromium.org, Apr 9 2018 

Only a small set of queries (or even just one) were broken. Probably.

Comment 6 by jkop@chromium.org, Apr 9 2018

Status: Started (was: Assigned)
This is most likely the result of a puppet change. https://chrome-internal-review.googlesource.com/593786 moved the creds for the crosinfra account on Wednesday.

Comment 7 by jkop@chromium.org, Apr 17 2018

Status: Fixed (was: Started)
►

Sign in to add a comment