Alec is on vacation, so putting it through the standard CI triage.

If there is a workaround then I would like to close this and fix it in the new UI.

Really this should have been in place before we switched the builders over to swarming, we don't seem to have a real UI to monitor the individual builder status for a branch, so this seems like it should be fairly high priority?

How are we able to watch in status builds on the branch today?

We can start them using https://luci-scheduler.appspot.com/jobs/chromeos but only monitor the master there.

Alright, let me dig into why Legoland isn't displaying the build status on the builder summary page. Would that be sufficient for what you're looking for Bernie? If it's a systemic issue and the effort to fix Legoland is too great then it should probably go into the new UI which should rollout in two months (ideally in a month, but that's very optimistic as there are a lot of interruptions on the CI team :p)

That sounds sufficient, we just need some way of monitoring the builders.

I'm hitting a lot of walls trying to look into this :( Is there something different about these builders?

@Jinjing, any ideas on where to look for this? Also I've tried adding `logger.atWarning().log("...")` statement but I don't see them anywhere, how do I get at Java logs when running locally? I see some logs in the Hexa frontend, but not mine.

Talked with Jinjing, need to get a debugger setup on my workstation to look into this one.

The only weird thing about these builders is that they are on a branch, and on swarming. That hasn't been done before for a master/slave group.

David, can you take a look at this some time in the next few weeks?

Looking at this in the debugger, the page noted above:

https://cros-goldeneye.corp.google.com/chromeos/legoland/builderSummary?builderGroups=release%2Cchrome_pfq%2Cmst_android_pfq%2Cnyc_android_pfq%2Cpi_android_pfq&limit=3&buildBranch=release-R69-10895.B

causes the backend to call getRecentMasterBuilds:

https://cs.corp.google.com/piper///depot/google3/java/com/google/chrome/crosbuilds/console/areas/legoland/service/BuilderGroupServiceImpl.java?l=299-307&rcl=201428128

ten times. That is two invocations for each of the builderGroups listed in the URL: "release", "chrome_pfq", "mst_android_pfq", "nyc_android_pfq", "pi_android_pfq". This queries the CIDB table "buildTable" on the "builder_name" column. The first set of invocations sets a limit of 1 and returns the following (results are limit found masterBuilderName):

1 0 master-release release-R69-10895.B
1 0 master-chromium-pfq release-R69-10895.B
1 0 master-mst-android-pfq release-R69-10895.B
1 0 master-nyc-android-pfq release-R69-10895.B
1 0 master-pi-android-pfq release-R69-10895.B

So zero matching rows for everything. The second round of invocations sets a limit of 3 and not surprisingly gets a similar result:

3 0 master-chromium-pfq release-R69-10895.B
3 0 master-mst-android-pfq release-R69-10895.B
3 0 master-nyc-android-pfq release-R69-10895.B
3 0 master-pi-android-pfq release-R69-10895.B
3 0 master-release release-R69-10895.B

Looking at the data in the CIDB table buildTable, there doesn't appear to be any records that contain "R69" in the builder_name column, which certainly explains the finding of zero matched rows:

mysql> SELECT COUNT(*) FROM buildTable WHERE builder_name LIKE '%-R69-%';
+----------+
| COUNT(*) |
+----------+
|        0 |
+----------+
1 row in set (1.07 sec)

Whereas prior release numbers have plenty of rows:

mysql> SELECT COUNT(*) FROM buildTable WHERE builder_name LIKE '%-R68-%';
+----------+
| COUNT(*) |
+----------+
|     8990 |
+----------+
1 row in set (1.13 sec)

mysql> SELECT COUNT(*) FROM buildTable WHERE builder_name LIKE '%-R67-%';
+----------+
| COUNT(*) |
+----------+
|     6742 |
+----------+
1 row in set (1.12 sec)

I believe that for swarming based builds, CIDB should contain the builder name of "master-release" even for branched builds.

I hope it also contains a "branch" field.

While there is no "branch" field there is what looks to be a related "full_version" field. It does not contain the literal "release-R69-10895.B" that these other searches used but does have that field starting with the R69-10895 prefix. For example, a query on the "master-nyc-android-pfq" one:

mysql> SELECT builder_name, full_version FROM buildTable WHERE full_version LIKE 'R69-10895.%' AND builder_name = 'master-nyc-android-pfq';                                                                               
+------------------------+-------------------+
| builder_name           | full_version      |
+------------------------+-------------------+
| master-nyc-android-pfq | R69-10895.0.0-rc1 |
| master-nyc-android-pfq | R69-10895.0.0-rc5 |
+------------------------+-------------------+
2 rows in set (1.26 sec)

That has a different meaning. I believe we need to start populating with an explicit branch.

Fix put in place in http://cl/208866631.

Thanks for the fix! 

We now can see R69 builds:
https://cros-goldeneye.corp.google.com/chromeos/legoland/builderSummary?buildBranch=release-R69-10895.B&builderGroups=release&limit=3&email=&buildConfig=

I wonder if we can add these to the list of builders on the left (optimally the three most recent release branches), perhaps that is another bug, and can wait for the CI migration.

It does NOT easily display the chrome/android PFQ builders on the branches, though the links in the initial bug description should still work.

Bernie, should we call it fixed, or some up with a solution for them?

If we are two months away from having the CI console version of Legoland, which will presumably fix this, then we can probably leave this fixed. We do want back all the functionality we had on the waterfall though as soon as reasonable, I defer to the CI team on how best to prioritize resources to get there from the current state.