Moblab failing to provision devices |
||
Issue description
Provisioning and Repair steps are failing when trying to update the DUTs (see error from status.log below).
I'm using a guado with official 10032.90.0 image. I tried powerwash and re-configured the device, but the issue persists. The only issue reported on MobMonitor is that there are no servo devices connected.
I'm not sure where to look next to debug and fix this, so any help would be greatly appreciated.
== Snip from repair step's status.log ==
Failed to install device image using payload at http://192.168.231.1:8080/update/grunt-release/R67-10531.0.0 on 192.168.231.105. : command execution error
* Command:
/usr/bin/ssh -a -x -o ControlPath=/tmp/_autotmp_Th22Apssh-
master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
-o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o
ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22
192.168.231.105 "export LIBC_FATAL_STDERR_=1; if type \"logger\" >
/dev/null 2>&1; then logger -tag \"autotest\"
\"server[stack::_run|_base_update_handler_no_retry|run] ->
ssh_run(/usr/bin/update_engine_client --update
--omaha_url=http://192.168.231.1:8080/update/grunt-
release/R67-10531.0.0)\";fi; /usr/bin/update_engine_client --update
--omaha_url=http://192.168.231.1:8080/update/grunt-release/R67-10531.0.0"
Exit status: 1
Duration: 1.30677914619
stderr:
[0405/100539:INFO:update_engine_client.cc(486)] Forcing an update by setting app_version to ForcedUpdate.
[0405/100539:INFO:update_engine_client.cc(488)] Initiating update check and install.
[0405/100539:INFO:update_engine_client.cc(517)] Waiting for update to complete.
[0405/100540:ERROR:update_engine_client.cc(232)] Update failed, current operation is UPDATE_STATUS_IDLE, last error code is ErrorCode::kOmahaResponseInvalid(34)
,
Apr 6 2018
Drew found the root cause - a non backward compatible change to devserver This change changed the appid from a hardcoded value to some other value. Not sure what it's used for exactly. The response from the moblab (as logged in update_engine.log) is the old hard-coded value. The requests from the DUT have a different appid. https://chromium.googlesource.com/chromiumos/platform/dev-util/+/d7a913a3455a4e857a11889516f9c7bf6204bf23%5E%21/#F0 This looks like the code where the "Omaha Response does not have version in manifest" error originates. In line 1043, there is first a check that the appid matches some expected appid. If that matches, then we set output_object->version on the response. Because the appid doesn't match, we never set the version, and thus the error. https://cs.corp.google.com/chromeos_public/src/aosp/system/update_engine/omaha_request_action.cc?rcl=ffb6d804bfeb671a793f781d17cd305499e83eef&l=1057
,
Apr 6 2018
Also seeing this issue in the Jetstream lab on our autotest server and moblabs. See b/76117708.
,
Apr 6 2018
We are not going to officially release a fix for this until next week due to the timing of 65 stable ( we do not want to risk breaking partners sending CTS results ) For internal users you can update to the version of 65 moblab we are testing for release next week by the command sudo update_engine_client -update --omaha_url="https://omaha.sandbox.google.com/service/update2" This has been confirmed to fix the provisioning issue, but the build has not had much testing yet so there might be other issues.
,
May 12 2018
|
||
►
Sign in to add a comment |
||
Comment 1 by haddowk@chromium.org
, Apr 6 2018I see this in the updater logs [0405/105753:INFO:omaha_request_action.cc(1099)] Omaha request response: <?xml version="1.0" encoding="UTF-8"?> <response protocol="3.0"> <daystart elapsed_seconds="39472"/> <app appid="{87efface-864d-49a5-9bb3-4b050a7c227a}" status="ok"> <ping status="ok"/> <updatecheck status="ok"> <urls> <url codebase="http://192.168.231.1:8080/static/grunt-release/R67-10531.0.0/"/> </urls> <manifest version="999999.0.0"> <packages> <package hash="Q413zizznelgKvgCGcO9tja0XQ4=" name="update.gz" size="1021491165" required="true"/> </packages> <actions> <action event="postinstall" ChromeOSVersion="999999.0.0" sha256="q2VOTPAtERrRgQy/v3AndFStG8wrxoBlAvYLXw3fp20=" needsadmin="false" IsDeltaPayload="false" MetadataSize="65509" /> </actions> </manifest> </updatecheck> </app> </response> [0405/105753:INFO:omaha_request_action.cc(935)] Found 1 <app>. [0405/105753:INFO:omaha_request_action.cc(1021)] Update for <app> 0 [0405/105753:ERROR:omaha_request_action.cc(1057)] Omaha Response does not have version in manifest! [0405/105753:INFO:metrics_reporter_omaha.cc(152)] Sending 3 for metric UpdateEngine.Check.Result (enum) [0405/105753:INFO:prefs.cc(122)] metrics-check-last-reporting-time not present in /var/lib/update_engine/prefs [0405/105753:INFO:action_processor.cc(116)] ActionProcessor: finished OmahaRequestAction with code ErrorCode::kOmahaResponseInvalid [0405/105753:INFO:action_processor.cc(121)] ActionProcessor: Aborting processing due to failure.