New issue
Advanced search Search tips

Issue 755132 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Aug 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

Sheriff-o-matic reporting 7 of last 13 builds failed when none of last 20 did

Project Member Reported by sullivan@chromium.org, Aug 14 2017

Issue description

Screenshot attached. Error says:

  system_health.memory_desktop failing on chromium.perf/Mac Pro 10.11 Perf
  Mac Pro 10.11 Perf [7 out of the last 13 builds have failed]

That links to this page:
https://uberchromegw.corp.google.com/i/chromium.perf/builders/Mac%20Pro%2010.11%20Perf

Showing 20 builds passed.

What's going on?
 
Screen Shot 2017-08-14 at 10.10.17 AM.png
446 KB View Download
Labels: -Pri-3 Pri-1
Status: Available (was: Untriaged)
/_cron/analyze/chromium.perf has been hitting some OOM errors on prod, so it could be out of date. 

Though it looks like it's been 500ing for a quite a while (header says "Last updated: 8/08/2017, 3:59 pm PDT (6 days ago)").  So that's why the alerts don't match the builder pages.

Owner: seanmccullough@chromium.org
Project Member

Comment 3 by bugdroid1@chromium.org, Aug 14 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/7fa4298302733f82325ceaa1f57bcd3054b23af6

commit 7fa4298302733f82325ceaa1f57bcd3054b23af6
Author: Sean McCullough <seanmccullough@chromium.org>
Date: Mon Aug 14 20:39:32 2017

[som] Fix retry logic so 4xx responses don't get retried.

Bug:  755132 
Change-Id: Id5dc5ae070973359ec0ff949cc0a938839b4af52
Reviewed-on: https://chromium-review.googlesource.com/614316
Commit-Queue: Sean McCullough <seanmccullough@chromium.org>
Reviewed-by: Stephen Martinis <martiniss@chromium.org>

[modify] https://crrev.com/7fa4298302733f82325ceaa1f57bcd3054b23af6/go/src/infra/monitoring/client/simpleclient.go
[add] https://crrev.com/7fa4298302733f82325ceaa1f57bcd3054b23af6/go/src/infra/monitoring/client/simpleclient_test.go

This fix is live on staging, and appears to have worked: https://sheriff-o-matic-staging.appspot.com/chromium.perf
Status: Fixed (was: Available)
Fix is now live on prod as well: https://sheriff-o-matic.appspot.com/chromium.perf

Sign in to add a comment