New issue
Advanced search Search tips
Starred by 4 users

Issue metadata

Status: Fixed
Closed: Feb 15
EstimatedDays: ----
NextAction: ----
OS: Android
Pri: 1
Type: ----

Sign in to add a comment

Issue 853248: Android test failure not showing up on Sheriff-o-Matic

Reported by, Jun 15 2018 Project Member

Issue description

Recent example failure:

When looking at Sheriff-o-Matic, the test failure doesn't show up.

Comment 1 by, Jun 15 2018

Labels: sheriff-android

Comment 2 by, Jun 16 2018

Labels: Milestone-Data

Comment 3 by, Jun 25 2018

This bot is still red and is still not showing up in sheriff-o-matic. In fact, when looking at sheriff-o-matic there are currently no failing bots but when looking at the waterfall there are quite a few red bots.

Any updates here?

Comment 4 by, Jun 27 2018

Indeed the bot remains red, and still not showing up. Is this tablet bot expected to be excluded from SoM? I couldn't find it in:

+jbudorick FYI

Comment 5 by, Aug 10

Components: Infra>Client>Chrome
All of the "tablet" testers are now purple (appear to be timing out after 3 hours)?

And indeed they do not show up on SoM. Is this intended?

Comment 6 by, Aug 10

SoM team can look at why it's not showing up on SoM. Might be that internal failures somehow get filtered out?

The bots are having issues because the tasks are pending for much too long. Example build of is a chrome_public_apk task which has a pending time of 2 hours. The overall build timeout is 3 hours, so it times out before all the tasks are done. Probably a capacity issue?

Comment 7 by, Aug 11

Status: Available (was: Untriaged)
The problem we have right now is the LUCI swarming tasks are hitting a 3 hour execution timeout. That can be remedied fairly easily. We are still out of capacity though. I'll make a CL to bump the timeout for the tablet builders.

Comment 8 by, Aug 11

Status: Assigned (was: Available)
Assigning to sean to figure out what's happening with it not showing up on sheriff-o-matic.

Comment 9 by, Aug 11

Project Member
The following revision refers to this bug:

commit 711748ea30b592183f2d37f54091dda9f4fee9c2
Author: Stephen Martinis <>
Date: Sat Aug 11 01:49:40 2018

Bump tablet tester execution timeout

They seem to need longer timeout for now. Hopefully will be temporary.


Bug:  853248 
Change-Id: I6665013085631c69e6d78fe1f43f195e12d3295c
Reviewed-by: Stephen Martinis <>
Commit-Queue: Stephen Martinis <>
Cr-Commit-Position: refs/heads/master@{#582420}

Comment 10 by, Aug 13


Comment 11 by, Aug 22

ping, is this still happening, and is the owner working on this issue?

Comment 12 by, Sep 13

All tablet testers listed in #5 still failing.

Do we care about those bots? Should they be removed?

Comment 13 by, Sep 13

I care about these bots. We have tablet-only code and tests that need to run somewhere and be monitored.

Comment 14 by, Oct 11

It looks like the K, L, and M tablets are all still failing consistently. I'd love to get these bots greened up, which I think is likely going to require getting them back on sheriff-o-matic so sheriffs are actually looking the failures and filing bugs.

Did we ever get to the bottom of why these bots aren't appearing?

Comment 15 by, Nov 1

Labels: -Pri-2 Pri-1
All still very red, and still not showing up on SoM.

SoM team any clue why is that?

Raising Pri to get some attention to this, as it has been going like this for quite a while.

Comment 16 by, Nov 20


Comment 17 by, Nov 20

No mention of these builders in the analyzer logs, so I don't think SoM even checks them.

I don't see any of these builders (or "" master, for that matter) mentioned in

Are these builders part of a tree that's supposed to be handled by gatekeeper? That's the config file that SoM gets it's "Tree" definitions from.

martiniss@, hinoka@ do you know if builders are supposed to show up in GetCompressedMasterJSON responses from Milo?

Comment 18 by, Nov 20

Yes it should.

Using this:
I see:
$ ./
KitKat Tablet Tester [12127, 12126, 12125, 12124, 12123, 12122]

It looks like the build number wasn't incremented when it was flipped (luci is at 1840, buildbot was at 12127), so Milo keeps returning the buildbot results.  I incremented the build number here:

Comment 19 by, Nov 20

More explanation: Milo knows the builder is a builder by looking at the "mastername" property in the buildbucket config.

Comment 20 by, Nov 20

It looks like is still building 1841, so I presume it will only start using the larger build numbers on the next build?

hinoka@: can you or someone else just write a script to go through and reset all of the build numbers for builders who've migrated to luci so they're ahead of the buildbot build numbers? This isn't the first time this particular issue has hidden failures from sheriffs, and I have a bad feeling it won't be the last.

Comment 21 by, Nov 21

"KitKat Tablet Tester" faillures now show up in SoM, but the Lollipop Tablet Tester and Marshmallow Tablet Tester still do not.

I'm not sure what values to set in the form at safely, or how monitor potential side effects.

Handing over to hinoka to complete those steps.

Comment 22 by, Dec 10

ping, this is sheriff P1 - does it still warrant that priority?

Comment 23 by, Dec 12

Ping, new sherriff here, is this still a P1?

Comment 24 by, Dec 12

Priority is the sheriff's call, I think. Are there still broken Android builds missing from SoM alerts?

Comment 25 by, Jan 2

New sheriff -- I can confirm that "Lollipop tablet tester" failures (most recent one from today) aren't showing up on SoM. P1 still applies.

Comment 26 by, Feb 11

New sheriff. Lollipop Phone Tester, Lollipop Tablet Tester and Marshmallow Tablet Tester failures still aren't showing up on SoM.

Comment 27 by, Feb 15

Status: Fixed (was: Assigned)
I bumped the build numbers by marking the builder as non-prod, then prod again, on luci-migration.  They show up on SoM now:

This should only happen to builders that were migrated early before automatic build number bumping was added.

Sign in to add a comment