Create an autotest testing an autoupdate over cellular connection |
||||
Issue descriptionWe currently execute an autoupdate over cellular *manually* on every new milestone. I want to create an autotest to do this. benchan@ has given me steps to run a test that sets the cellular connection as primary (so we can use cellular connection but not lose the ssh connection to the DUT). Currently, our autoupdate tests use a lab devserver as the omaha instance. When we switch to cellular connection, these devservers will not be accessible by the DUT. Some thoughts/options on writing this test: 1. Some of the autoupdate tests copy the devserver and payload to the DUT and do the whole update via localhost. Keeping that setup won't really be a good test for cellular though. 2. If we have the test communicate with the production omaha instance (https://omaha.sandbox.google.com/update/service2) we would only be able to run the test on images earlier than the publicly released version. There could also be problems with pinging the production server from a test image. 3. I think the best solution here is to have the test setup an omaha instance on the DUT with a payload stored on a public location (like google storage). That way we have a omaha instance that can serve a payload no matter what version we are running against. It will also test the payload being downloaded via cellular. And test that update_engine does the right thing when the update_over_cellular flag is set
,
Dec 21 2017
> 3. I think the best solution here is to have the test setup an omaha instance on the DUT with a payload stored on a public location (like google storage). That way we have a omaha instance that can serve a payload no matter what version we are running against. It will also test the payload being downloaded via cellular. And test that update_engine does the right thing when the update_over_cellular flag is set If you are running an omaha instance inside the DUT, then what is the point of testing the UE? in this case UE is getting the payload from localhost. Its like testing only the cellular connection. Doing a full update looks to me just an overkill for testing the update_over_cellular flag with UE. I think a better test would be to test the UE on slow and interrupted connections instead!? But I agree that we should have some autotests for these kind of things.
,
Dec 21 2017
I was thinking about repurposing some of the code from this test: https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/client/site_tests/autoupdate_CannedOmahaUpdate/autoupdate_CannedOmahaUpdate.py This will give us a very lite omaha instance on the DUT that will always return an "update available" response to pings. The payload is not stored on the DUT though. That will be on Google Storage and will need to be downloaded via cellular.
,
Dec 21 2017
Yeah, That make more sense now. Thanks. autoupdate_CannedOmahaUpdate.py is using a canned full update. But if you need to make sure update succeeds by downloading the payload, but you don't want to download more than 1GB of payload on cellular network. We could generate a delta payload from some other version. paygen build and tests in canary, already making the delta's, so it may be possible to merge them there!
,
Dec 21 2017
ahassani: For your other point from #2: I definitely have a plan to automate an update with a bunch of interruptions while an update in progress. Basically all of the manual tests from section 1.2 here I hope to automate next year: https://testtracker.googleplex.com/testplans/details/450 For this bug though I want an end to end cellular test.
,
Dec 21 2017
Re #4: An N-to-N delta payload is generated for every build. I can use that instead of the full payload for this test (Not sure yet who owns/pays for the sim cards in the lab - we don't need to run this test everyday anyway) The N-to-N delta is what I used for the P2P test :)
,
Dec 21 2017
An N-to-N's payload is very small I believe. For example in the crbug.com/796692, we see it downloads like 10% of the payload and it fails afterwards. N-to-N tests may not catch this scenario. (Although we still don't know the source of crbug.com/796692 :/ ) e.g. chromeos_R65-10235.0.0_R65-10235.0.0_edgar_delta_dev.bin 189.02 KB
,
Dec 21 2017
So you mean to reuse one of the other delta payloads produced by the builder right? The same ones that are used by autoupdate_EndToEndTest / paygen_au_dev suite That will mean we need to first put the DUT on the correct FROM version (in a server side test) and then call a client test to do the cellular update to the delta payload
,
Dec 21 2017
Correct. in #6 you mentioned that 'we don't need to run this test everyday'. Can you give more context how often these tests will be run? Will they get initiated manually like test_that by someone? Or we have some sort of tests are run every few days or something?
,
Dec 21 2017
It definitely can be run on every build automatically like the rest of the AU tests. Or scheduled weekly etc etc I just added that line to say we don't HAVE to run it on every build if the celluar budget is low or whatever. I'll check with the cellular team what they do with the rest of their tests.
,
Dec 22 2017
I have hacked together a basic setup for this test. However I am hitting a problem that even though we are on cellular, UE will not do the update [1222/095938:WARNING:libpolicy.cc(37)] Could not load the device policy file. [1222/095938:INFO:update_attempter.cc(314)] No device policies/settings present. [1222/095938:INFO:connection_manager.cc(77)] Disabling updates over cellular as device policy fails to be loaded. [1222/095938:INFO:connection_manager.cc(114)] There's no device policy loaded yet. [1222/095938:INFO:omaha_request_action.cc(1559)] Allowing updates over cellular as permission preference is set to true. [1222/095938:INFO:omaha_request_action.cc(1630)] We are connected via cellular, Updates allowed: No [1222/095938:INFO:omaha_request_action.cc(1530)] Update is not allowed over current connection.
,
Dec 22 2017
Normally, the user has to ack the fact that update is performing over the cellular. Have you passed that?
,
Jan 5 2018
Nope, I haven't dealt with that confirmation dialog. I have been doing all of this remotely so I forgot it does that :D Ansar sent me the attached screenshot of when he done it manually to show me what it looks like. I wonder if it only shows that dialog when you click the "Check for Update" button though. Will it be triggered by calling update_engine_client directly? I ask because the error message from #12 seem to be unrelated to not acknowledging the confirmation dialog. It is a problem with not loading the device policy file. e.g similar to issue 799185 .
,
Jan 5 2018
> I wonder if it only shows that dialog when you click the "Check for Update" button though. Will it be triggered by calling update_engine_client directly? I think that is the case, it will show after check for update. there are two flags for update_engine_client: --show_update_over_cellular --update_ver_cellular I don't know the exact usage of them, but you might want to try them out and see how to use them. There is a fix for the aforementioned bug, you might want to try with that too.
,
Jan 26 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/22da8b4e876cd7980becc28b0bede90c49b835fe commit 22da8b4e876cd7980becc28b0bede90c49b835fe Author: David Haddock <dhaddock@chromium.org> Date: Fri Jan 26 03:23:59 2018 Update autoupdate_CannedOmahaResponse to handle more AU cases. This CL adds support for delta payloads and updates over cellular. BUG= chromium:797047 TEST=Testing with upcoming cellular AU CL. TEST=autoupdate_CatchBadSignatures still works. Change-Id: I0462483d316f6ef21acc4d32e0f2d27d8b0d3218 Reviewed-on: https://chromium-review.googlesource.com/869465 Commit-Ready: David Haddock <dhaddock@chromium.org> Tested-by: David Haddock <dhaddock@chromium.org> Reviewed-by: Katherine Threlkeld <kathrelkeld@chromium.org> [modify] https://crrev.com/22da8b4e876cd7980becc28b0bede90c49b835fe/client/site_tests/autoupdate_CannedOmahaUpdate/autoupdate_CannedOmahaUpdate.py
,
Jan 30 2018
,
Feb 6 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/suite_scheduler/+/de1e25a54344c2683d5204292a5522dbc336b5d4 commit de1e25a54344c2683d5204292a5522dbc336b5d4 Author: David Haddock <dhaddock@chromium.org> Date: Tue Feb 06 12:55:58 2018 Add cellular_au suite. BUG= chromium:797047 TEST=None Change-Id: Ib7c2bf1f863505e2d50dce32c8bf7f43612e3939 Reviewed-on: https://chromium-review.googlesource.com/903242 Commit-Ready: David Haddock <dhaddock@chromium.org> Tested-by: David Haddock <dhaddock@chromium.org> Reviewed-by: Harpreet Grewal <harpreet@chromium.org> [modify] https://crrev.com/de1e25a54344c2683d5204292a5522dbc336b5d4/configs/suite_scheduler.ini
,
Feb 7 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/01c657f683b149acdca06950c998251d61c69921 commit 01c657f683b149acdca06950c998251d61c69921 Author: David Haddock <dhaddock@chromium.org> Date: Wed Feb 07 02:18:38 2018 Add a cellular_au test suite. BUG= chromium:797047 TEST=None Change-Id: I82286c6b896f935389ae4b011c1e72adae55e280 Reviewed-on: https://chromium-review.googlesource.com/903226 Commit-Ready: David Haddock <dhaddock@chromium.org> Tested-by: David Haddock <dhaddock@chromium.org> Reviewed-by: Harpreet Grewal <harpreet@chromium.org> [add] https://crrev.com/01c657f683b149acdca06950c998251d61c69921/test_suites/control.cellular_au
,
Feb 8 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/4a60e5d847b2eb02d6c21614cc10dfdbd758f15e commit 4a60e5d847b2eb02d6c21614cc10dfdbd758f15e Author: David Haddock <dhaddock@chromium.org> Date: Thu Feb 08 05:27:37 2018 Add autoupdate_Cellular test to ebuild. BUG= chromium:797047 TEST=None Change-Id: I997af4c858f48ee46cd93fa38e2ac7f46198ebe5 Reviewed-on: https://chromium-review.googlesource.com/903248 Commit-Ready: David Haddock <dhaddock@chromium.org> Tested-by: David Haddock <dhaddock@chromium.org> Reviewed-by: David Haddock <dhaddock@chromium.org> Reviewed-by: Amin Hassani <ahassani@chromium.org> [modify] https://crrev.com/4a60e5d847b2eb02d6c21614cc10dfdbd758f15e/chromeos-base/autotest-server-tests/autotest-server-tests-9999.ebuild
,
Feb 8 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/44dea631fbf655ff58a3e2a977de6f2f6a69e7a2 commit 44dea631fbf655ff58a3e2a977de6f2f6a69e7a2 Author: David Haddock <dhaddock@chromium.org> Date: Thu Feb 08 09:24:38 2018 autoupdate test over cellular connection. While connected to cellular, the DUT will not be able to access a devserver. So we need to copy the payload to a publicly accessible google storage location that we will then download over cellular. We will use an omaha instance on the DUT (via autoupdate_CannedOmahaUpdate) that will point to that google storage location. We will then check that the update engine logs contain the cellular entries if the update completes successfully. Other small things in this CL: 1. I have reworked the get_delta_payload() function to be able to return either the full or delta payload url. 2. I've add a function which calls the devservers /api/fileinfo API to return information about staged files (SHA256 and size) because gs://chromeos-image-archive/ payloads don't have the .json file with this info. BUG= chromium:797047 TEST=autoupdate_Cellular.full passes TEST=autoupdate_Cellular.delta passes Change-Id: I14cd15615a5606a3ed2ead4721c253baf5212b21 Reviewed-on: https://chromium-review.googlesource.com/900362 Commit-Ready: David Haddock <dhaddock@chromium.org> Tested-by: David Haddock <dhaddock@chromium.org> Reviewed-by: David Haddock <dhaddock@chromium.org> [add] https://crrev.com/44dea631fbf655ff58a3e2a977de6f2f6a69e7a2/server/site_tests/autoupdate_Cellular/control.delta [add] https://crrev.com/44dea631fbf655ff58a3e2a977de6f2f6a69e7a2/server/site_tests/autoupdate_Cellular/control.full [modify] https://crrev.com/44dea631fbf655ff58a3e2a977de6f2f6a69e7a2/server/cros/update_engine/update_engine_test.py [add] https://crrev.com/44dea631fbf655ff58a3e2a977de6f2f6a69e7a2/server/site_tests/autoupdate_Cellular/autoupdate_Cellular.py
,
Feb 8 2018
,
Feb 17 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/59492ffd99108d2b0477af71ee05d198086268b5 commit 59492ffd99108d2b0477af71ee05d198086268b5 Author: David Haddock <dhaddock@chromium.org> Date: Sat Feb 17 23:00:03 2018 Fix for failing AU cellular tests. The pool:cellular DUTs are in a restricted subnet so tests cannot make direct calls to a devserver. The staging code already takes care of this by sshing and then using curl so I will make the get_payload_info function do the same. See client/common_lib/cros/dev_server.py run_ssh_call() I've also added a suite to run the DELTA cellular au tests nightly. BUG= chromium:797047 TEST=autoupdate_Cellular.delta passes on the DUT it failed on yesterday Change-Id: I2b0c25eb37053a4e509c6f1781476dc1929d8bee Reviewed-on: https://chromium-review.googlesource.com/923215 Commit-Ready: David Haddock <dhaddock@chromium.org> Tested-by: David Haddock <dhaddock@chromium.org> Reviewed-by: David Haddock <dhaddock@chromium.org> [modify] https://crrev.com/59492ffd99108d2b0477af71ee05d198086268b5/server/site_tests/autoupdate_ForcedOOBEUpdate/control.cellular.delta [modify] https://crrev.com/59492ffd99108d2b0477af71ee05d198086268b5/server/site_tests/autoupdate_Cellular/control.delta [add] https://crrev.com/59492ffd99108d2b0477af71ee05d198086268b5/test_suites/control.cellular_nightly [modify] https://crrev.com/59492ffd99108d2b0477af71ee05d198086268b5/server/cros/update_engine/update_engine_test.py [modify] https://crrev.com/59492ffd99108d2b0477af71ee05d198086268b5/site_utils/attribute_whitelist.txt
,
Dec 7
|
||||
►
Sign in to add a comment |
||||
Comment 1 by dhadd...@chromium.org
, Dec 21 2017