Is there a problem with this particular device? I don't see anything particularly bad in the crosland list there, and I also see that the DUT actually changed between the "Good" and "Bad":

https://stainless.corp.google.com/search?view=matrix&row=hostname&col=build&first_date=2018-04-12&last_date=2018-06-08&suite=wifi_endtoend%7Cwifi_matfunc%7Cwifi_perf%7Cwifi_update_router%7Cwifi_release%7Cwifi_lucidsleep&build=R66-10452.*&model=blacktip&exclude_cts=false&exclude_not_run=false&exclude_non_release=true&exclude_au=true&exclude_acts=true&exclude_retried=true&exclude_non_production=false

See how the "Good" is on chromeos15-row4-rack12-host4 and the "Bad" is on chromeos15-row1-rack2-host2.

Anyway, chromeos15-row1-rack2-host2 still seems to be in bad shape. From one of the failures in stainless above:

[   13.236493] iwlwifi 0000:01:00.0: enabling device (0000 -> 0002)
[   13.267903] iwlwifi 0000:01:00.0: Direct firmware load for iwl-dbg-cfg.ini failed with error -2
[   13.300108] iwlwifi 0000:01:00.0: loaded firmware version 17.318154.0 op_mode iwlmvm
[   13.495946] iwlwifi 0000:01:00.0: Detected Intel(R) Dual Band Wireless AC 7265, REV=0xFFFFFFFF
[   13.520846] iwlwifi 0000:01:00.0: iwlwifi transaction failed, dumping registers
[   13.520858] iwlwifi 0000:01:00.0: iwlwifi device config registers:
[   13.520898] iwlwifi 0000:01:00.0: 00000000: 095a8086 00100406 02800059 00000000 00000004 00000000 00000000 00000000
[   13.520903] iwlwifi 0000:01:00.0: 00000020: 00000000 00000000 00000000 50108086 00000000 000000c8 00000000 00000100
[   13.520906] iwlwifi 0000:01:00.0: iwlwifi device memory mapped registers:
[   13.520936] iwlwifi 0000:01:00.0: 00000000: ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
[   13.520940] iwlwifi 0000:01:00.0: 00000020: ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
[   13.520947] iwlwifi 0000:01:00.0: iwlwifi device AER capability structure:
[   13.520973] iwlwifi 0000:01:00.0: 00000000: 14010001 00100000 00000000 00462031 00002000 00002000 00000014 40000001
[   13.520976] iwlwifi 0000:01:00.0: 00000020: 0000000f c2a0000c 00000000
[   13.520980] iwlwifi 0000:01:00.0: iwlwifi parent port (0000:00:14.0) config registers:
[   13.520997] iwlwifi 0000:00:14.0: 00000000: 5ad68086 00100006 060400fb 00810010 00000000 00000000 40010100 228000f0
[   13.521001] iwlwifi 0000:00:14.0: 00000020: c2a0c2a0 0001fff1 00000000 00000000 00000000 00000040 00000000 00030000
[   13.521008] iwlwifi 0000:00:14.0: 00000000: 5ad68086 00100006 060400fb 00810010 00000000 00000000 40010100 228000f0
[   13.521012] iwlwifi 0000:00:14.0: 00000020: c2a0c2a0 0001fff1 00000000 00000000 00000000 00000040 00000000 00030000
[   13.521015] iwlwifi 0000:01:00.0: Read failed at 0x40
[   13.521020] ------------[ cut here ]------------
[   13.521036] WARNING: CPU: 1 PID: 1724 at ../../../../../tmp/portage/sys-kernel/chromeos-kernel-4_4-4.4.118-r1422/work/chromeos-kernel-4_4-4.4.118/drivers/net/wireless/iwl7000/iwlwifi/pcie/trans.c:2094 iwl_trans_pcie_grab_nic_access+0xd4/0xf2 [iwlwifi]
[   13.521039] Timeout waiting for hardware access (CSR_GP_CNTRL 0xffffffff)
[   13.521041] Modules linked in: iwlmvm(+) iwl7000_mac80211 ax88179_178a(+) usbnet uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_core snd_soc_skl snd_soc_skl_ipc snd_soc_sst_ipc iwlwifi snd_soc_sst_dsp snd_soc_sst_match snd_hda_ext_core snd_hda_core acpi_als snd_soc_max98357a zram snd_soc_da7219 bridge stp llc ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_mark fuse snd_seq_dummy snd_seq snd_seq_device iwl7000_cfg80211 cros_ec_sensors_ring cros_ec_sensors_core kfifo_buf industrialio ip6table_filter r8152 mii joydev btusb btrtl btbcm btintel bluetooth
[   13.521113] CPU: 1 PID: 1724 Comm: modprobe Tainted: G     U          4.4.118-13213-gd81e1fa0bcfd #1
[   13.521116] Hardware name: Google Coral/Coral, BIOS Google_Coral.10068.59.0 05/01/2018
[   13.521119]  0000000000000086 1a163b143453ed46 ffff88005eb87a80 ffffffffbed04f38
[   13.521126]  ffff88005eb87ad0 0000000000000000 ffff88005eb87ac0 ffffffffbea6d0e8
[   13.521133]  0000082e5eb87b40 ffff880163da0028 0000000000003000 ffff880163da9370
[   13.521139] Call Trace:
[   13.521150]  [<ffffffffbed04f38>] dump_stack+0x4d/0x63
[   13.521155]  [<ffffffffbea6d0e8>] __warn+0xca/0xe5
[   13.521160]  [<ffffffffbea6d1eb>] warn_slowpath_fmt+0x5a/0x76
[   13.521168]  [<ffffffffc0414803>] ? iwl_read32+0x25/0x97 [iwlwifi]
[   13.521176]  [<ffffffffc0426d0d>] ? iwl_trans_pcie_write8+0x16/0x16 [iwlwifi]
[   13.521184]  [<ffffffffc0428928>] iwl_trans_pcie_grab_nic_access+0xd4/0xf2 [iwlwifi]
[   13.521191]  [<ffffffffc041450c>] iwl_write_prph+0x3b/0x76 [iwlwifi]
[   13.521199]  [<ffffffffc04297af>] iwl_pcie_apm_init+0x16c/0x1a6 [iwlwifi]
[   13.521207]  [<ffffffffc042a5db>] iwl_trans_pcie_start_hw+0x6b/0x107 [iwlwifi]
[   13.521214]  [<ffffffffc042a570>] ? iwl_pcie_check_hw_rf_kill+0x86/0x86 [iwlwifi]
[   13.521241]  [<ffffffffc0592e8a>] iwl_op_mode_mvm_start+0x736/0xa1e [iwlmvm]
[   13.521250]  [<ffffffffc0415051>] _iwl_op_mode_start+0x68/0x92 [iwlwifi]
[   13.521257]  [<ffffffffc0415109>] iwl_opmode_register+0x8e/0xb5 [iwlwifi]
[   13.521276]  [<ffffffffc0454035>] iwl_mvm_init+0x35/0x1000 [iwlmvm]
[   13.521280]  [<ffffffffc0454000>] ? 0xffffffffc0454000
[   13.521285]  [<ffffffffbea004a9>] do_one_initcall+0x18c/0x1a1
[   13.521291]  [<ffffffffbeac7582>] do_init_module+0x5f/0x1cd
[   13.521295]  [<ffffffffbeac9609>] load_module+0x1ca6/0x1cca
[   13.521299]  [<ffffffffbeac96ad>] SYSC_finit_module+0x80/0xa7
[   13.521303]  [<ffffffffbeac96ad>] ? SYSC_finit_module+0x80/0xa7
[   13.521307]  [<ffffffffbeac9865>] SyS_finit_module+0xe/0x10
[   13.521312]  [<ffffffffbf1d440d>] entry_SYSCALL_64_fastpath+0x21/0x95
[   13.521316] ---[ end trace 6b9d20ee6188432b ]---


Anyway, I would suggest investigating this device in particular, to see whether the OS regressed, or if this device just has some kind of problem.

Have we ruled out https://b.corp.google.com/issues/78198279
Need to check that the kernel and firmware versions on this device are such that we dont hit that bug

Re#2: I couldn't login to the bad DUT chromeos15-row1-rack2-host2 to verify the firmweare version, but we never backported the fix to R66. SO if it was this issue, it would show up in R66. BTW, R67 results look OK to me:

https://stainless.corp.google.com/search?view=matrix&row=model&col=build&first_date=2018-05-30&last_date=2018-06-12&suite=wifi_endtoend%7Cwifi_matfunc%7Cwifi_perf%7Cwifi_update_router%7Cwifi_release%7Cwifi_lucidsleep&build=R67-10575*&model=blacktip&exclude_cts=false&exclude_not_run=false&exclude_non_release=true&exclude_au=true&exclude_acts=true&exclude_retried=true&exclude_non_production=false

Looks like chromeos15-row1-rack2-host2 is down in "repair failed" state. Will have lab team take a look, bring it back up.

In terms of the failures on M66, chromeos15-row4-rack12-host4 was a DVT version and chromeos15-row1-rack2-host2 is a PVT. Why would this issue "No interfaces managed by shill" only show up on PVT device but not on DVT device on M66.