Compile is failing on the bot, so it's not even getting to the stage where it can complain about devices being online or offline.

It looks like the devices are offline, or at least messed up. https://build.chromium.org/p/chromium.fyi/builders/ClangToTAndroid%20x64/builds/167/steps/device_recovery/logs/stdio has a lot of errors. Looks like all the phones are just offline?

Labs can you take a look at the phones on build133-b1? Not super urgent, because the code isn't compiling right now.

device_recovery is failing on those devices, and this is showing up in the log:

AdbCommandFailedError: (device: 0B897394) adb wait-for-device: failed with exit status 1 and output:
- error: could not install *smartsocket* listener: Address already in use
- ADB server didn't ACK
- * failed to start daemon *
- error: cannot connect to daemon
- * daemon not running. starting it now on port 5037 *

probably not a labs issue.

Compile is currently failing on all the clang tot bots, yes, I filed https://llvm.org/bugs/show_bug.cgi?id=30976 for that. It should compile again in a few hours though.

Ok, compile should cycle green on the next build.

Looks like the first problem was at https://build.chromium.org/p/chromium.fyi/builders/ClangToTAndroid%20x64/builds/137 where the devices disappeared in the middle of running tests, and they've been missing since.

I believe that these are the 64-bit x86 Intel tablets.  

Power cycled them all and now they adb aware.

List of devices attached
0B8973CE	device
0B897394	device
16E74304	device
17E0007E	device
238F7DF5	device
16E7431D	device
03E5276C	device

Thanks Peter. I wonder how long these devices will stay up again. If they keep having trouble, I might need to think about moving them to swarming so they can get their hands on the device watchdog.

It's failing again, starting with https://build.chromium.org/p/chromium.fyi/builders/ClangToTAndroid%20x64/builds/188

https://build.chromium.org/p/chromium.fyi/builders/ClangToTAndroid%20x64/builds/182 was the first passing run. One build cycles in about 6 h, so it worked fine for 36h / 2.5d.

Of note: In https://build.chromium.org/p/chromium.fyi/builders/ClangToTAndroid%20x64/builds/187 the bot needed 10h to cycle. Before 7h https://build.chromium.org/p/chromium.fyi/builders/ClangToTAndroid%20x64/builds/186 and before that https://build.chromium.org/p/chromium.fyi/builders/ClangToTAndroid%20x64/builds/185 6h for several bots. Not sure if that's because maybe some of the slaves died earlier on those two builds?

(reopening)

Yikes. It's taking longer and longer to cycle because the devices are dropping offline as time goes on. It had 5 available during build #185, 3 on #186, and only 1 tablet left on build #187.

I'll swing by the lab later today if they haven't been fixed yet to take a look. It might be that these types of failures can be mostly mitigated by installing the watchdog we have all swarmed devices.

I brought them back up, but I'm sure they'll drop offline again. I'll start working on getting the watchdog up and running on them.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra.git/+/bfee5ab5ccc05bcd21afc79ac741b436b00908ec

commit bfee5ab5ccc05bcd21afc79ac741b436b00908ec
Author: Benjamin Pastene <bpastene@google.com>
Date: Tue Nov 15 00:50:25 2016

Correct the compiler flags when building go for x86 and x64 on android.

BUG= 664190 

Change-Id: If66a47c025e609db8d6b93f37c5824e68ae44414
Reviewed-on: https://chromium-review.googlesource.com/411451
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Benjamin Pastene <bpastene@chromium.org>

[modify] https://crrev.com/bfee5ab5ccc05bcd21afc79ac741b436b00908ec/go/mobile_env.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra.git/+/50a58d7226055897b485fa62d0b14af013ca86f9

commit 50a58d7226055897b485fa62d0b14af013ca86f9
Author: Benjamin Pastene <bpastene@google.com>
Date: Tue Nov 15 01:36:53 2016

Add android-x64 to the Go packages built by infra-continuous-precise-64.

This should increase the cycle time by only a few seconds since
the gomobile env only needs to be initialized once. If not, I can break
all android compiling into its own bot.

BUG= 664190 

Change-Id: Ib99818e015820ae4830db8d47ac9e66626ba7b72
Reviewed-on: https://chromium-review.googlesource.com/411462
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Benjamin Pastene <bpastene@chromium.org>

[modify] https://crrev.com/50a58d7226055897b485fa62d0b14af013ca86f9/recipes/recipes/infra_continuous.expected/infra-cross-compile.json
[modify] https://crrev.com/50a58d7226055897b485fa62d0b14af013ca86f9/recipes/recipes/infra_continuous.py

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/infra_internal.git/+/72bf6432145a55dcf113c4c6f2bf47fad1985985

commit 72bf6432145a55dcf113c4c6f2bf47fad1985985
Author: Benjamin Pastene <bpastene@google.com>
Date: Wed Nov 16 01:20:54 2016

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/puppet/+/bd0a88a96e61689ea00e0a9202a26bf987a64975

commit bd0a88a96e61689ea00e0a9202a26bf987a64975
Author: Benjamin Pastene <bpastene@google.com>
Date: Thu Nov 17 01:54:15 2016

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/puppet/+/454b051597e1d98405db26f2d5771b59f2f8006a

commit 454b051597e1d98405db26f2d5771b59f2f8006a
Author: Benjamin Pastene <bpastene@google.com>
Date: Thu Nov 17 19:03:27 2016

K, I threw build135-b1 into swarming for the time-being: https://chromium-swarm-dev.appspot.com/bot?id=build135-b1

I'll watch it and see if the watchdog helps the devices live longer.

https://build.chromium.org/p/chromium.fyi/builders/Android%20x64%20Tests will remain offline for the duration of the experiment.

https://build.chromium.org/p/chromium.fyi/builders/ClangToTAndroid%20x64 is still failing provision_devices on every run. Is this supposed to work now?

Ben, let me know if you want me to take a look at the devices.

Yeah, could you reboot the ones on build133-b1 and build135-b1? I'm going to switch them to swarming. That'll make them not suck as much.

Done.

chrome-bot@build133-b1:(Linux 14.04):~$ ./adb devices
List of devices attached
0B897394	device
16E74304	device
238F7DF5	device
0B8973CE	device
17E0007E	device
03E5276C	device
16E7431D	device

chrome-bot@build135-b1:(Linux 14.04):~$ ./adb devices
List of devices attached
2B895576	device
238F7DFF	device
29891291	device
1BE74F14	device
2E8CCA31	device
25890A3D	device
0B8973CB	device

Thanks Peter!

https://codereview.chromium.org/2521113002/ and https://chromium-review.googlesource.com/c/413307/ to swarm-erize them.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/infra_internal.git/+/9fdb66526fb17b5c310b19154c173c490f32d03c

commit 9fdb66526fb17b5c310b19154c173c490f32d03c
Author: Benjamin Pastene <bpastene@google.com>
Date: Mon Nov 21 22:42:22 2016

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/infra_internal.git/+/5c0f69ab99cefc9ec80278f13a98d08c09df5c8d

commit 5c0f69ab99cefc9ec80278f13a98d08c09df5c8d
Author: Benjamin Pastene <bpastene@google.com>
Date: Tue Nov 22 01:01:25 2016

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/master-manager.git/+/4310beb5c33e649ef79343ee071866bcd2e426a8

commit 4310beb5c33e649ef79343ee071866bcd2e426a8
Author: Benjamin Pastene <bpastene@google.com>
Date: Tue Nov 22 01:49:28 2016

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/master-manager.git/+/4310beb5c33e649ef79343ee071866bcd2e426a8

commit 4310beb5c33e649ef79343ee071866bcd2e426a8
Author: Benjamin Pastene <bpastene@google.com>
Date: Tue Nov 22 01:49:28 2016

K, most everything is done here. https://codereview.chromium.org/2521113002/ just needs to be committed. The bots will fail all tests until then, but they'll still compile.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0470f9af87d6dd520c928b9468370c058b2022d1

commit 0470f9af87d6dd520c928b9468370c058b2022d1
Author: bpastene <bpastene@chromium.org>
Date: Fri Dec 02 02:28:40 2016

Swarm the tests on the two x64 android testers on chromium.fyi

Also missing tests that are on the cq but weren't here.

Needs https://chromium-review.googlesource.com/c/413307/

BUG= 664190 , 534815

Review-Url: https://codereview.chromium.org/2521113002
Cr-Commit-Position: refs/heads/master@{#435830}

[modify] https://crrev.com/0470f9af87d6dd520c928b9468370c058b2022d1/testing/buildbot/chromium.fyi.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7a9d577b4d27f64420d5a6a0b3e5e6c2bdcbf6b5

commit 7a9d577b4d27f64420d5a6a0b3e5e6c2bdcbf6b5
Author: bpastene <bpastene@chromium.org>
Date: Sat Dec 03 02:49:46 2016

chromium.fyi: Set min num of devices for swarmed x64 android tests.

The tasks aren't getting picked up because it defaults to 6 and the
swarming bots only have 1 available. (No clue why; I'll look at that later.)
This should at least get them running, albeit really slowly.

BUG=534815,  664190 

Review-Url: https://codereview.chromium.org/2544223003
Cr-Commit-Position: refs/heads/master@{#436150}

[modify] https://crrev.com/7a9d577b4d27f64420d5a6a0b3e5e6c2bdcbf6b5/testing/buildbot/chromium.fyi.json

Moving Infra>Client>Android -> Infra>Client>Chrome+OS=Android

Guessing this is old (bot link doesn't work)