Now failing 100%:
https://stainless.corp.google.com/search?view=matrix&row=test&col=build&first_date=2018-03-07&last_date=2018-03-13&test=%5Eautoupdate_Cellular&exclude_cts=true&exclude_not_run=false&exclude_non_release=true&exclude_au=true&exclude_acts=true&exclude_retried=true&exclude_non_production=true

Will take a look.

+my team as FYI on why the cellular AU tests are failing 

Any update here?

All cellular tests are failing (not just AU):
https://stainless.corp.google.com/search?test=%5Ecellular_&exclude_non_release=true&exclude_cts=true&col=build&row=test&view=matrix&first_date=2018-03-28&last_date=2018-04-03

I will check the sim cards on lab devices.

Up on looking at the cellular devices status in pool:cellular, below are some findings: 

1. Due to issue on Altair modem (Daisy_spring & Quawks), cellular connection is failing. We dont have fw fix yet from vendors. Hence could not connect. crbug.com/809724

2. Lumpy device is not getting any updates starting M-66.

3. Devices that should work fine: Lucas_Daisy (Gobi3K modem), Link, Astronaut and anyother DUT with external modem. I see that tests ran fine on Astronaut and Daisy.
https://stainless.corp.google.com/search?view=matrix&row=model&col=build&first_date=2018-03-07&last_date=2018-04-03&suite=%5Ecellular%5C_endtoend%24&status=GOOD&exclude_cts=true&exclude_not_run=false&exclude_non_release=true&exclude_au=true&exclude_acts=true&exclude_retried=true&exclude_non_production=false

4. Will look for issues on Link and any other DUTs where tests are failing.

There are bunch of cellular devices which has valid SIM and test fail is being investigated at crbug.com/829123

Regarding c#6, point4: Link had an expired SIM. Replaced it with working SIM. chromeos1-row1-rack7-host6 should be good now.

I just ran tried to run a test against that link and it failed:
FAIL: INIT_ERROR: An ethernet connection is required between the test server and the device under test.

Any update here? All of the cellular tests are failing with the error:
https://stainless.corp.google.com/search?test=%5Ecellular_&exclude_non_release=true&exclude_cts=true&col=build&row=test&view=matrix&first_date=2018-04-06&last_date=2018-04-12

Hi Ben, Can you please take a look at this issue? 

#1 and #10 are different issues.

#10 fails because ethtool is somehow missing from the system (probably due to some recent change that sys-apps/ethtool is no longer pulled in). Given that ethtool seems to be used in a few locations in autotest, autotest should make sys-apps/ethtool as an explicit dependency instead of relying some transitive dependency to pull it in.

The failures from #1 are no longer appearing for me. I guess that is from the work done by Dinesh in #6 and #9 (Thank you)

Ben, is the change you talked about in #13 something you can do?

It does seem that only cellular tests are affected by "INIT_ERROR: An ethernet connection is required between the test server and the device under test." error.

https://stainless.corp.google.com/search?view=matrix&row=test&col=build&first_date=2018-03-17&last_date=2018-04-13&reason=INIT_ERROR%3A+An+ethernet+connection+is+required+between+the+test+server+and+the+device+under+test.&exclude_cts=true&exclude_not_run=true&exclude_non_release=true&exclude_au=true&exclude_acts=true&exclude_retried=true&exclude_non_production=false

Re #15: nope, other tests are affected as well (note that the error message may be different)

https://stainless.corp.google.com/search?test=%5Enetwork%5C_EthernetStressPlug%24&exclude_non_release=true&exclude_cts=true&col=build&row=model&view=matrix&first_date=2018-04-07&last_date=2018-04-13

Here're the fixes:

https://chromium-review.googlesource.com/#/c/chromiumos/overlays/chromiumos-overlay/+/1011702 autotest-tests: add sys-apps/ethtool dependency
https://chromium-review.googlesource.com/#/c/chromiumos/overlays/chromiumos-overlay/+/1011703 autotest-tests-cellular: add sys-apps/ethtool dependency
https://chromium-review.googlesource.com/#/c/chromiumos/overlays/chromiumos-overlay/+/1011704 autotest-tests-chrome: add sys-apps/ethtool dependency

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/fa2a74d8e7690d0e3237414b01dfd0dae55227bf

commit fa2a74d8e7690d0e3237414b01dfd0dae55227bf
Author: Ben Chan <benchan@chromium.org>
Date: Fri Apr 13 08:43:19 2018

autotest-tests: add sys-apps/ethtool dependency

ethtool is used in the following tests:
- network_EthCaps
- network_EthernetStressPlug
- network_NegotiatedLANSpeed

BUG=chromium:816655
TEST=`equery-$BOARD g autotest-tests-9999` shows sys-apps/ethtool

Change-Id: I660b400ec6c715ab6dce02b4aaf33e5825449ff7
Reviewed-on: https://chromium-review.googlesource.com/1011702
Commit-Ready: Ben Chan <benchan@chromium.org>
Tested-by: Ben Chan <benchan@chromium.org>
Reviewed-by: Harpreet Grewal <harpreet@chromium.org>

[modify] https://crrev.com/fa2a74d8e7690d0e3237414b01dfd0dae55227bf/chromeos-base/autotest-tests/autotest-tests-9999.ebuild

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/b7a3ff41ec419d6b7974ff4d5acf3b1f040f0ccd

commit b7a3ff41ec419d6b7974ff4d5acf3b1f040f0ccd
Author: Ben Chan <benchan@chromium.org>
Date: Fri Apr 13 08:43:20 2018

autotest-tests-cellular: add sys-apps/ethtool dependency

Most of the cellular tests use client/cros/backchannel.py, which uses ethtool.

BUG=chromium:816655
TEST=`equery-$BOARD g autotest-tests-cellular-9999` shows sys-apps/ethtool

Change-Id: Ide251109100bd2d5ef40b31b3ef274e7fc36140b
Reviewed-on: https://chromium-review.googlesource.com/1011703
Commit-Ready: Ben Chan <benchan@chromium.org>
Tested-by: Ben Chan <benchan@chromium.org>
Reviewed-by: Harpreet Grewal <harpreet@chromium.org>

[modify] https://crrev.com/b7a3ff41ec419d6b7974ff4d5acf3b1f040f0ccd/chromeos-base/autotest-tests-cellular/autotest-tests-cellular-9999.ebuild

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/525f158917a812b99debf07d45199e368ed0d434

commit 525f158917a812b99debf07d45199e368ed0d434
Author: Ben Chan <benchan@chromium.org>
Date: Fri Apr 13 08:43:21 2018

autotest-tests-chrome: add sys-apps/ethtool dependency

The following tests use client/cros/backchannel.py, which uses ethtool:
- cellular_ModemControl
- power_LoadTest

BUG=chromium:816655
TEST=`equery-$BOARD g autotest-tests-chrome-9999` shows sys-apps/ethtool

Change-Id: I694a82fcf2ce3022f7c89c2c65e19c483e5d1ba3
Reviewed-on: https://chromium-review.googlesource.com/1011704
Commit-Ready: Ben Chan <benchan@chromium.org>
Tested-by: Ben Chan <benchan@chromium.org>
Reviewed-by: Harpreet Grewal <harpreet@chromium.org>

[modify] https://crrev.com/525f158917a812b99debf07d45199e368ed0d434/chromeos-base/autotest-chrome/autotest-chrome-9999.ebuild

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/f44a9e6ea9d993ef8b67c308bb7c7c26e041c8fe

commit f44a9e6ea9d993ef8b67c308bb7c7c26e041c8fe
Author: Ben Chan <benchan@chromium.org>
Date: Thu Apr 19 00:35:34 2018

target-chromium-os-test: add sys-apps/ethtool as dependency

ethtool is used by some autotest modules (e.g.
client/bin/net/net_utils.py, client/cros/backchannel.py) and several
tests (e.g. network_EthCaps, network_NegotiatedLANSpeed).

BUG=chromium:816655
TEST=Build test image and verify /usr/sbin/ethtool exists on image.

Change-Id: I6bff20d63f65cb386746c434ea6f4dc615c4857c
Reviewed-on: https://chromium-review.googlesource.com/1016174
Commit-Ready: Ben Chan <benchan@chromium.org>
Tested-by: Ben Chan <benchan@chromium.org>
Reviewed-by: Harpreet Grewal <harpreet@chromium.org>

[modify] https://crrev.com/f44a9e6ea9d993ef8b67c308bb7c7c26e041c8fe/virtual/target-chromium-os-test/target-chromium-os-test-1.ebuild
[rename] https://crrev.com/f44a9e6ea9d993ef8b67c308bb7c7c26e041c8fe/virtual/target-chromium-os-test/target-chromium-os-test-1-r66.ebuild

Thanks for the changes Ben. 

Now we are back to the errors at the start of this bug.

E.g failures here:
https://stainless.corp.google.com/search?exclude_retried=true&first_date=2018-05-29&master_builder_name=&builder_name_number=&shard=&exclude_acts=true&builder_name=&master_builder_name_number=&owner=&retry=&exclude_cts=false&exclude_non_production=false&hostname=&board=&test=%5Eautoupdate_Cellular%5C.delta%24&suite=%5Ecellular_au&build=%5ER69%5C-10751%5C.0%5C.0%24&status=FAIL&status=ERROR&status=ABORT&reason=&waterfall=&exclude_not_run=false&last_date=2018-06-04&exclude_non_release=true&exclude_au=false&model=&view=list

https://stainless.corp.google.com/search?exclude_retried=true&exclude_cts=false&exclude_non_production=false&exclude_acts=true&exclude_non_release=true&exclude_au=false&suite=%5Ecellular_au&exclude_not_run=false&row=test&col=build&view=matrix&first_date=2018-05-29&last_date=2018-06-04

Error, "Cannot find cellular device in shill. Is the modem plugged in?" is persistent on Link (chromeos1-row1-rack7-host6) and Squawks (chromeos1-row2-rack8-host2) - 

https://stainless.corp.google.com/search?view=matrix&row=test&col=board_model&first_date=2018-05-09&last_date=2018-06-05&reason=Cannot+find+cellular+device+in+shill&exclude_cts=false&exclude_not_run=false&exclude_non_release=true&exclude_au=false&exclude_acts=true&exclude_retried=true&exclude_non_production=false

06/05 04:13:57.219 ERROR|             utils:2730| Will raise error TestError('Cannot find cellular device in shill. Is the modem plugged in?',) due to unexpected return: None
06/05 04:13:57.226 ERROR|  test_environment:0129| Error during test initialization:
Traceback (most recent call last):
  File "/usr/local/autotest/cros/cellular/test_environment.py", line 113, in __enter__
    self._verify_sim()
  File "/usr/local/autotest/cros/cellular/test_environment.py", line 235, in _verify_sim
    modem_device = self._get_shill_cellular_device_object()
  File "/usr/local/autotest/cros/cellular/test_environment.py", line 151, in _get_shill_cellular_device_object
    timeout=shill_proxy.ShillProxy.DEVICE_ENUMERATION_TIMEOUT)
  File "/usr/local/autotest/common_lib/utils.py", line 2731, in poll_for_condition
    raise exception
TestError: Cannot find cellular device in shill. Is the modem plugged in?


@Ben, what should we look for here? Tests on these 2 devices have been failing for a while because of this error though there are some tests that sometimes pass on Link. See...

https://stainless.corp.google.com/search?view=matrix&row=test&col=board_model&first_date=2018-05-09&last_date=2018-06-05&hostname=chromeos1-row1-rack7-host6%7Cchromeos1-row2-rack8-host2&exclude_cts=false&exclude_not_run=false&exclude_non_release=true&exclude_au=false&exclude_acts=true&exclude_retried=true&exclude_non_production=false





daisy_spring (chromeos1-row1-rack7-host1) complains about "There is no SIM in the modem." for all cellular tests.

06/02 03:03:28.952 ERROR|  test_environment:0129| Error during test initialization:
Traceback (most recent call last):
  File "/usr/local/autotest/cros/cellular/test_environment.py", line 113, in __enter__
    self._verify_sim()
  File "/usr/local/autotest/cros/cellular/test_environment.py", line 248, in _verify_sim
    raise error.TestError('There is no SIM in the modem.')
TestError: There is no SIM in the modem.


@Dinesh, please check this device / sim.





Error "An ethernet connection is required between the test server and the device under test." should have been fixed with Ben's changes.

@Ben, looks like this was mostly not an issue starting with R68-10715.0.0 till R69-10736.0.0 but tests starting failing again due to this error starting with R69-10738.0.0. See..

https://stainless.corp.google.com/search?view=matrix&row=test&col=build&first_date=2018-03-17&last_date=2018-06-05&reason=INIT_ERROR%3A+An+ethernet+connection+is+required+between+the+test+server+and+the+device+under+test.&exclude_cts=true&exclude_not_run=true&exclude_non_release=true&exclude_au=true&exclude_acts=true&exclude_retried=true&exclude_non_production=false

>> TestError: Cannot find cellular device in shill. Is the modem plugged in?

It seems like the modem on chromeos1-row1-rack7-host6 got into a weird state and didn't respond to ModemManager. I'd suggest power cycling the device, launch a round of cellular_ota tests and see if the modem still behaves correctly.

The device reboots every time a new build is installed / tested. Isn't this enough or a complete cold reboot is needed? If so, maybe we should resurrect cellular_StaleModemReboot. Though I would want to clean up this test and make it a simple reboot that runs nightly.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/infra/suite_scheduler/+/da71cc01a0aea0f132956fe3a793a15e63c82ede

commit da71cc01a0aea0f132956fe3a793a15e63c82ede
Author: harpreet <harpreet@google.com>
Date: Fri Jun 08 06:33:52 2018

autotest: cellular: Update cellular_modem_repair suite

cellular_modem_repair suite should run on all devices in pool:celluar.
We also want it to run at 0 hour nightly without delay in order to make
sure the cellular modems in devices in this pool are in good health
before other cellular suites are run.

BUG=chromium:816655
TEST=None

Change-Id: I85ccabb693d1ba4d2bb85281c1407a2dffacf9f5
Reviewed-on: https://chromium-review.googlesource.com/1089991
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Harpreet Grewal <harpreet@chromium.org>
Reviewed-by: Ben Chan <benchan@chromium.org>

[modify] https://crrev.com/da71cc01a0aea0f132956fe3a793a15e63c82ede/configs/suite_scheduler.ini

dsunkara@ - the mode repair (dut reboot) test is only running on 2 devices and consistently failing on 1 of them. Can you please take a look to see why its not running on all devices in pool:cellular and also check why its failing on chromeos1-row2-rack8-host2. 


https://stainless.corp.google.com/search?exclude_retried=true&exclude_cts=true&exclude_non_production=false&exclude_acts=true&exclude_non_release=true&exclude_au=true&suite=%5Ecellular%5C_modem%5C_repair%24&exclude_not_run=false&row=hostname&col=build&view=matrix&days=15


Automated cellular test suites in general can use some tlc, looks like most tests/suites are consistently failing. I'll open a separate bug for this.

On Quwaks and daisy_spring, we had firmware issue for a long time which was causing cellular connection issue. 

Ben provided me a firmware which I am now trying to install and test. If it succeeds, I shall flash the same on devices in lab and shall provide more updates here. 

Also there is another issue where modem is not detected at all. Even rebooting the dut is not helping. I shall get those devices to my workstation for further debugging. 

I shall provide more updates soon.

Ben confirmed that chromeos1-row2-rack8-host2 has unsupported modem. This SKU never shipped.

As per the request from Ben, this dut will be decommissioned and removed from lab.

LTE module on chromeos1-row2-rack8-host2 has been changed with updated firmware. This dut now has cellular connectivity and can resume testing. 

Messi-V image on this LTE module is still under test. Once the testing is complete, this could get merged.

chromeos1-row1-rack8-host2 is not scheduling any tests. Also, when I scheduled manually, smoke test failed with below error:

https://stainless.corp.google.com/browse/chromeos-autotest-results/217837790-dsunkara/chromeos1-row1-rack8-host2/

07/16 10:31:20.805 DEBUG|      base_sysinfo:0124| Loggable saves logs to /usr/local/autotest/results/default/cellular_Smoke/sysinfo/iteration.1/schedstat.after
07/16 10:31:20.806 DEBUG|             utils:0218| Running 'logger "autotest finished iteration /usr/local/autotest/results/default/cellular_Smoke/sysinfo/iteration.1"'
07/16 10:31:20.824 DEBUG|              test:0420| after_iteration_hooks completed
07/16 10:31:20.825 WARNI|              test:0637| The test failed with the following exception
Traceback (most recent call last):
  File "/usr/local/autotest/common_lib/test.py", line 631, in _exec
    _call_test_function(self.execute, *p_args, **p_dargs)
  File "/usr/local/autotest/common_lib/test.py", line 831, in _call_test_function
    return func(*args, **dargs)
  File "/usr/local/autotest/common_lib/test.py", line 495, in execute
    dargs)
  File "/usr/local/autotest/common_lib/test.py", line 362, in _call_run_once_with_retry
    postprocess_profiled_run, args, dargs)
  File "/usr/local/autotest/common_lib/test.py", line 400, in _call_run_once
    self.run_once(*args, **dargs)
  File "/usr/local/autotest/site_tests/cellular_Smoke/cellular_Smoke.py", line 100, in run_once
    with test_env, shill_context.ServiceAutoConnectContext(
  File "/usr/local/autotest/cros/cellular/test_environment.py", line 131, in __enter__
    raise error.TestError('INIT_ERROR: %s' % str(e))
TestError: INIT_ERROR: org.chromium.flimflam.Error.Failure: Serial command timed out
07/16 10:31:20.840 DEBUG|              test:0642| Running cleanup for test.
07/16 10:31:20.841 DEBUG|   logging_manager:0627| Logging subprocess finished
07/16 10:31:20.844 DEBUG|   logging_manager:0627| Logging subprocess finished

@Ben, can you please take a look?

Issue happens on Nautilus too. 

Any update here? The cellular tests have been failing a long time now

dsunkara@, could you help update the latest status of the cellular DUTs?

Hi Ben,
All the below hosts are in good state with active cellular connection. 
chromeos1-row1-rack7-host5
chromeos1-row1-rack8-host2
chromeos1-row2-rack7-host1
chromeos1-row2-rack7-host3
chromeos1-row2-rack7-host4
chromeos1-row2-rack8-host3
However, au_tests fail on these hosts:
https://stainless.corp.google.com/search?view=list&first_date=2018-07-31&last_date=2018-08-06&suite=%5Ecellular%5C_au%24&hostname=chromeos1-row1-rack7-host5%7Cchromeos1-row1-rack8-host2%7Cchromeos1-row2-rack7-host1%7Cchromeos1-row2-rack8-host3&exclude_cts=true&exclude_not_run=false&exclude_non_release=true&exclude_au=true&exclude_acts=true&exclude_retried=true&exclude_non_production=false

Also, on these tests, there are two new errors (other than INIT_ERRO)
1. reason: Update attempt failed: Command <update_engine_client -update -omaha_url=http://127.0.0.1:34779/update > failed, rc=1, Command returned non-zero exit status
2. Did not find expected string in update_engine log: Allowing updates over cellular as permission preference is set to true.

Thanks Dinesh. The errors look AU specific. Back to dhaddock@ to take a look at the AU tests.

Huh? Here are the latest results:
https://stainless.corp.google.com/search?exclude_retried=true&exclude_cts=false&exclude_non_production=false&exclude_acts=true&exclude_non_release=true&exclude_au=false&suite=%5Ecellular_au&exclude_not_run=false&row=test&col=build&view=matrix&first_date=2018-08-03&last_date=2018-08-09

In the last day the tests have failed on these DUTs with these cellular errors

chromeos1-row1-rack7-host1: INIT_ERROR: There is no SIM in the modem.)
chromeos1-row2-rack7-host1: INIT_ERROR: Cannot find cellular device in shill. Is the modem plugged in?)
chromeos1-row1-rack8-host2: INIT_ERROR: An ethernet connection is required between the test server and the device under test.).
chromeos1-row2-rack7-host3: INIT_ERROR: Cannot find cellular device in shill. Is the modem plugged in?

And this one too:

chromeos1-row2-rack7-host4: INIT_ERROR: Cannot find cellular device in shill. Is the modem plugged in?

Below two hosts at present had issues with cellular connectivity and are blocked/failed on tests because of b/112042309 or b/111852637
chromeos1-row2-rack7-host4
chromeos1-row2-rack7-host3 

Identified regression on chromeos1-row1-rack7-host1 and is being tracked @  crbug.com/873732 

Working on other duts for RCA.

(probably a dumb question) What is RCA?

Root Cause Analysis. :)

+dsunkara,  the 2 bugs you point out are really just one b/111852637, since  b/112042309 is dup to b/111852637.

But 111852637 is marked as wont fixed, so back to this bug, test are still failure https://stainless.corp.google.com/search?view=matrix&row=model&col=build&first_date=2018-10-06&last_date=2018-11-02&suite=%5Ecellular_au&exclude_cts=false&exclude_not_run=false&exclude_non_release=true&exclude_au=false&exclude_acts=true&exclude_retried=true&exclude_non_production=false


And looks like we should run force AU on nautiluslte instead of nautilus?

The devices we have in lab are nautiluslte. But for some reason atest shows them as nautilus. 
dsunkara@dsunkara:~/Downloads$ ssh root@chromeos1-row2-rack7-host3
localhost ~ # 
localhost ~ # mosys platform id
Google
Nautilus
nautiluslte
NAUTILUSLTE
1
SMAL
nautiluslte
nautiluslte
rev2
Google_Nautilus

As for the test issues, there are some changes that Ben submitted that will fix cellular connectivity/timing issues (for auto tests) and hopefully that should solve these errors. 

Below are CLs:
https://chromium-review.googlesource.com/c/chromiumos/third_party/autotest/+/1313538
https://chromium-review.googlesource.com/c/chromiumos/third_party/autotest/+/1313537
https://chromium-review.googlesource.com/c/chromiumos/third_party/autotest/+/1314049

Hi David,
Nautilus is now failing with below reasons on auto update. Can you please check?

https://stainless.corp.google.com/search?view=list&first_date=2018-10-20&last_date=2018-11-02&suite=%5Ecellular%5C_au%24&model=nautilus&exclude_cts=true&exclude_not_run=false&exclude_non_release=true&exclude_au=true&exclude_acts=true&exclude_retried=true&exclude_non_production=false

autoupdate_CannedOmahaUpdate client test did not pass (reason: Update attempt failed: Command <update_engine_client -update -omaha_url=http://127.0.0.1:39709/update > failed, rc=1, Command returned non-zero exit status).
autoupdate_StartOOBEUpdate client test did not pass (reason: Update status was unexpectedly IDLE when we were waiting for the update to complete. Please check the update engine logs.).

Those 4 failures look like actual failures by update_engine to complete the update via cellular. That should be a different bug. Let's keep this one to be failures due to cellular setup. 

Filed crbug.com/901565 to look at that 

For this bug it looks like just daisy_spring is still affected?

I will check on this device (daisy_spring) manually to see if it has cellular connectivity.
Checking it remotely says that modem is not up and does not detect any cellular connection. I also could not start/initiate cellular connectivity remotely.

+dsunkara, I filed https://bugs.chromium.org/p/chromium/issues/detail?id=901929 for the mis-label issue.

Latest run has another INIT_ERROR this time on quawks chromeos1-row1-rack8-host2:
https://stainless.corp.google.com/search?exclude_retried=true&first_date=2018-11-02&master_builder_name=&builder_name_number=&shard=&exclude_acts=true&builder_name=&master_builder_name_number=&owner=&retry=&exclude_cts=false&exclude_non_production=false&hostname=&board=&branch=&test=%5Eautoupdate_Cellular%5C.full%24&suite=%5Ecellular_au&build=%5ER72%5C-11241%5C.0%5C.0%24&status=FAIL&status=ERROR&status=ABORT&reason=&waterfall=&exclude_not_run=false&last_date=2018-11-08&exclude_non_release=true&exclude_au=false&model=&view=list