Maybe we should have separate builders, one per kernel, instead. Right now amd64-generic-pre-cq and arm-generic-pre-cq run on chromeos-4.4 but nowhere else. It would be great to have those running at least on chromeos-4.14 and chromeos-4.19 as well.

we have real devices that use those older kernel versions.  what value is there in having generic builders also doing it ?

i'm not sure what you mean by amd64-generic-pre-cq running on chromeos-4.4 ... as Doug noted, they're on chromeos-4.14 now.

groeck@groeck0:~/src/linux-chrome$ git describe
v4.4.168-15572-g28b634e269ee
groeck@groeck0:~/src/linux-chrome$ cat COMMIT-QUEUE.ini 
# Copyright 2016 The Chromium OS Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.

# Per-project Commit Queue settings.
# Documentation: http://goo.gl/5J7oND

[GENERAL]

pre-cq-configs: amd64-generic-pre-cq
                arm-generic-pre-cq
                gru-pre-cq
                reef-pre-cq

i don't know what you're trying to show.  the amd64-generic-pre-cq is clearly building 4.14.  look at the snippet Doug provided in comment #0, and just look up the builders themselves:
https://cros-goldeneye.corp.google.com/chromeos/legoland/builderHistory?buildConfig=amd64-generic-pre-cq&buildBranch=master
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8926674866708567760
https://luci-logdog.appspot.com/logs/chromeos/buildbucket/cr-buildbucket.appspot.com/8926674866708567760/+/steps/BuildPackages/0/stdout

[ebuild  N     ] sys-kernel/chromeos-kernel-4_14-4.14.88-r626::chromiumos to /build/amd64-generic/ ...

#4: That may well be, but it doesn't help that both amd64-generic-pre-cq and arm-generic-pre-cq are listed in COMMIT-QUEUE.ini of chromeos-4.4 (and not in chromeos-4.14). This sounds like a pre-requisite for commits into chromeos-4.4 is a passing pre-cq test with chromeos-4.14.
Of course, my understanding may be completely wrong. Anything chromite is still a mystery to me. If so, my apologies for the confusion.

i don't see the COMMIT-QUEUE.ini file having any bearing here.  if the v4.4 repo has outdated entries, then they can be deleted.  but that can be done entirely independently of updating the generic builders to 4.19.

the only thing the v4.4/COMMIT-QUEUE.ini file is saying is, for CLs in this repo, kick off these configs in the precq.

#6: I assume you are saying that amd64-generic-pre-cq in COMMIT-QUEUE.ini of a given kernel branch always results in that kernel branch being built, independently of the information in the various overlays.
In other words, we can add amd64-generic-pre-cq to all kernel branches, and this will result in test builds for all CLs affecting the those kernel branches, using that kernel branch as base. Is this correct ?

i don't really understand what you're describing

there is no direct relationship between COMMIT-QUEUE.ini and the overlay settings (USE flags).  the only thing COMMIT-QUEUE.ini does is tell the infrastructure (precq/cq) which set of bot configs to run whenever a CL is posted in that repo.  it doesn't know whether that board will actually build any code from that repo ... that's the responsibility of the COMMIT-QUEUE.ini author.

so you could list amd64-generic-pre-cq in every kernel repo's COMMIT-QUEUE.ini, but it'd be a waste of resources when the only kernel amd64-generic builds today is v4.14.

the config file is documented here:
https://dev.chromium.org/chromium-os/build/bypassing-tests-on-a-per-project-basis

Then why do you object to creating per-branch builders ?

i don't know what you mean by "per-branch builders".  if you mean "let's add a bot config for every kernel version for amd64-generic", then as i said, i don't see the point: we already have pre-cq configs that provide coverage.
v3.8: wolf/daisy/peach_pit
v3.10: nyan/rambi (the rambi setting here is also outdated)
v3.14: samus/arm-generic/whirlwind/veyron (arm-generic here is outdated)
v3.18: oak/cyan/glados/gale
v4.4: amd64-generic/arm-generic/gru/reef (arm-generic & amd64-generic is outdated)
v4.14: grunt (could add generics here since they're using 4.14 atm)
v4.19: cheza (will add generics here once they move to 4.19 in the overlay)

so what value exactly is there with creating 12 extra configs to build for a board that no one otherwise is testing or looking at when we already have real devices providing coverage ?

again: COMMIT-QUEUE.ini settings are orthogonal to Doug's proposal of changing the generics to 4.19 (which we should do).

lets look at linux-3.10 a concrete example.  at this point, only nyan boards are using that version.  we'll never use that version for any other board let alone ship it anywhere else.  so practically speaking, what use is there building that version for any other config ?  if we don't get any value out of it, don't we save engineering time in not worrying about breakage they might introduce for other configs when backporting things ?

i agree that for newer versions that are actively developed (like 4.14 or 4.19), having generic configs tracking those makes sense.  but once we've wrapped up development on a specific version, what do we gain from building generic configs on them ?

We still make modifications to all kernel versions kernels as old as chromeos-3.8, no matter if they are in active development or not. I guess you are suggesting that there are no benefits to actually _test_ those kernels with generic configurations. That is a matter of opinion.

Test builds, static sanitizer runs, fuzzing, and test boots of chromeos-4.4 and later are nowadays running with lots of generic configurations, only not as part of CQ. They are done by Intel's 0day infrastructure, syzbot, and with a test infrastructure I am running on a set of GCE instances. Without those, merging stable releases would be practically impossible, since we would have no means to sanity test the merges, and merge regressions would pile up over time. If we would not care about building generic configurations, we would also lose the benefit of running 0day on ChromeOS (we gave up on it for chromeos-3.18 since pretty much all configurations other than ours fail to build by now).
I also plan to start running file system regression tests on chromeos-4.4 and later. This would also not be practical since those tests rely on generic configurations.

We don't do any of the additional testing for chromeos-3.18 and earlier due to the many failures seen with those releases. As a result, those release don't get stable release merges (nor would that even be possible by now), and thus don't automatically get security fixes either. I don't think that is a good situation, and I would like to avoid it for later kernel branches.

Guenter: I think it should probably be a separate task and a separate bug to get broader testing of our kernels and we can have the debate there.  For now, I'd advocate that we at least move the current generic builders to 4.19 since those have the least amount of other testing going on.  Is that OK?

$ cros tryjob amd64-generic-full-tryjob -g '1399601 1390378'
Tryjob submitted!
To view your tryjobs, visit:
  https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8924950821313166080

$ cros tryjob arm-generic-full-tryjob -g '1399601 1390378'
Tryjob submitted!
To view your tryjobs, visit:
  https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8924950807852227408

$ cros tryjob arm64-generic-full-tryjob -g '1399601 1390378'
Tryjob submitted!
To view your tryjobs, visit:
  https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8924950798090989872

$ cros tryjob x32-generic-full-tryjob -g '1399601 1390378'
Tryjob submitted!
To view your tryjobs, visit:
  https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8924950767321945984