New issue
Advanced search Search tips

Issue 786225 link

Starred by 1 user

Issue metadata

Status: Verified
Owner:
Closed: Jan 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 3
Type: Bug-Regression



Sign in to add a comment

Autoupdate from and to build details are shown the same in update_engine log file

Project Member Reported by sdantul...@chromium.org, Nov 17 2017

Issue description

Google Chrome	64.0.3270.0 (Official Build) dev (64-bit)
Revision	0
Platform	10134.0.0 (Official Build) dev-channel samus

What steps will reproduce the problem?
(1) Trigger AU of device from 10134.0.0 ==> 10135.0.0
(2) After download of updates and post installation, check 'From' and 'To' build details in update_engine log file 

What happens ?
Both from and to build details are the same

Attached update_engine log file.
 
update_engine.20171116-170940.txt
44.9 KB View Download
Cc: ahass...@chromium.org
log file info....

[1116/171525:INFO:action_processor.cc(143)] ActionProcessor: starting PostinstallRunnerAction
[1116/171525:INFO:postinstall_runner_action.cc(166)] Performing postinst (postinst at /tmp/.org.chromium.Chromium.Eb6KbZ/postinst) installed on device /dev/sda5 and mountable device /dev/sda5
[1116/171525:INFO:postinstall_runner_action.cc(173)] Format file for new postinst is: data
[1116/171600:INFO:subprocess.cc(156)] Subprocess output:
dm:dm bht[DEBUG] Setting block_count 448000
dm:dm bht[DEBUG] Setting depth to 3.
dm:dm bht[DEBUG] depth: 0 entries: 1
dm:dm bht[DEBUG] depth: 1 entries: 28
dm:dm bht[DEBUG] depth: 2 entries: 3500
PostInstall Configured: (B, /dev/sda5, /dev/sda4, /dev/sda12)

FROM (rootfs):
CHROMEOS_ARC_ANDROID_SDK_VERSION=25
CHROMEOS_ARC_VERSION=4454992
CHROMEOS_AUSERVER=https://tools.google.com/service/update2
CHROMEOS_BOARD_APPID={F67500C1-C6D8-5287-E4EC-F9BBB4AEE5C5}
CHROMEOS_CANARY_APPID={90F229CE-83E2-4FAF-8479-E368A34938B1}
CHROMEOS_DEVSERVER=
CHROMEOS_RELEASE_APPID={F67500C1-C6D8-5287-E4EC-F9BBB4AEE5C5}
CHROMEOS_RELEASE_BOARD=samus-signed-mp-v3keys
CHROMEOS_RELEASE_BRANCH_NUMBER=0
CHROMEOS_RELEASE_BUILDER_PATH=samus-release/R64-10135.0.0
CHROMEOS_RELEASE_BUILD_NUMBER=10135
CHROMEOS_RELEASE_BUILD_TYPE=Official Build
CHROMEOS_RELEASE_CHROME_MILESTONE=64
CHROMEOS_RELEASE_DESCRIPTION=10135.0.0 (Official Build) dev-channel samus 
CHROMEOS_RELEASE_NAME=Chrome OS
CHROMEOS_RELEASE_PATCH_NUMBER=0
CHROMEOS_RELEASE_TRACK=dev-channel
CHROMEOS_RELEASE_VERSION=10135.0.0
DEVICETYPE=CHROMEBOOK
GOOGLE_RELEASE=10135.0.0

FROM (stateful):
CHROMEOS_RELEASE_TRACK=dev-channel

TO:
CHROMEOS_ARC_ANDROID_SDK_VERSION=25
CHROMEOS_ARC_VERSION=4454992
CHROMEOS_AUSERVER=https://tools.google.com/service/update2
CHROMEOS_BOARD_APPID={F67500C1-C6D8-5287-E4EC-F9BBB4AEE5C5}
CHROMEOS_CANARY_APPID={90F229CE-83E2-4FAF-8479-E368A34938B1}
CHROMEOS_DEVSERVER=
CHROMEOS_RELEASE_APPID={F67500C1-C6D8-5287-E4EC-F9BBB4AEE5C5}
CHROMEOS_RELEASE_BOARD=samus-signed-mp-v3keys
CHROMEOS_RELEASE_BRANCH_NUMBER=0
CHROMEOS_RELEASE_BUILDER_PATH=samus-release/R64-10135.0.0
CHROMEOS_RELEASE_BUILD_NUMBER=10135
CHROMEOS_RELEASE_BUILD_TYPE=Official Build
CHROMEOS_RELEASE_CHROME_MILESTONE=64
CHROMEOS_RELEASE_DESCRIPTION=10135.0.0 (Official Build) dev-channel samus 
CHROMEOS_RELEASE_NAME=Chrome OS
CHROMEOS_RELEASE_PATCH_NUMBER=0
CHROMEOS_RELEASE_TRACK=dev-channel
CHROMEOS_RELEASE_VERSION=10135.0.0
DEVICETYPE=CHROMEBOOK
GOOGLE_RELEASE=10135.0.0

ChromeosChrootPostinst(10135.0.0)
Set boot target to /dev/sda5: Partition 5, Slot B
SetImage
KERNEL_CONFIG: console= loglevel=7 init=/sbin/init cros_secure oops=panic panic=-1 root=/dev/dm-0 rootwait ro dm_verity.error_behavior=3 dm_verity.max_bios=-1 dm_verity.dev_wait=1 dm="1 vroot none ro 1,0 3584000 verity payload=PARTUUID=%U/PARTNROFF=1 hashtree=PARTUUID=%U/PARTNROFF=1 hashstart=3584000 alg=sha1 root_hexdigest=5508dd04cad0cbd0a857a546cf81a6b7d780a843 salt=99f478e1e64265bbc029923f7e945efcf0443074472c65b04849f76a1e2b4f44" noinitrd vt.global_cursor_default=0 kern_guid=%U add_efi_memmap boot=local noresume noswap i915.modeset=1 tpm_tis.force=1 tpm_tis.interrupts=0 nmi_watchdog=panic,lapic i915.enable_psr=1  
Setting up verity.
Finished after 18 seconds.
Clearing network driver boot cache: /var/lib/preload-network-drivers.
Syncing filesystems before changing boot order...
Finished after 0 seconds.
Updating Partition Table Attributes using CgptManager...
Can you describe how did you performed the update? Is this on canary builds or manual update?

Also if possible, can you rollback the device and see what is the /etc/lsb-release?
Re c#2:

Can you describe how did you performed the update? 
Triggered autoupdate by giving 'autest' command from crosh terminal.

Is this on canary builds or manual update?
This is manual update triggered using autest command

Also if possible, can you rollback the device and see what is the /etc/lsb-release?
After download of updates, I did not reboot the device yet. So it is still on build 10134.0.0. /etc/lsb-release file has CHROMEOS_RELEASE_TRACK=dev-channel


Cc: vapier@chromium.org
Ok. I looked into this problem, it seems a problem with postinst. We probably need to mount /etc in addition to other points in chromeos-postinst. I'm running some tests to confirm this. If not, I have to investigate more to find the root cause. Adding vapier@ as he recently did some of the postinst changes.

Comment 5 by vapier@chromium.org, Nov 21 2017

postinst chroots into the new root, so there is no /etc to bind mount.  the bind mounts we list there are writable/pseudo mount points.
Doesn't bind mount work on simple folders too if /etc is not a writeable/pseudo mount point? 
I just tried exactly what postinst does and I'm able to bind mount /etc.
mount -n --bind "/etc" "./etc"
and then I chroot in there and /etc was the mounted one.

But even that is not the issue here. The issue is that we need /etc/lsb-release from both new and old partitions, but when postinst mounts those folders and chroot to the new image, we don't have access to /etc/lsb-release of the old image anymore. From chromeos_postinst.cc:466 we have:

  // If we can read in the lsb-release we are updating FROM, log it.
  string lsb_contents;
  if (ReadFileToString("/etc/lsb-release", &lsb_contents)) {
    printf("\nFROM (rootfs):\n%s", lsb_contents.c_str());
  }

...

  // If we can read the lsb-release we are updating TO, log it
  if (ReadFileToString(install_config.root.mount() + "/etc/lsb-release",
                       &lsb_contents)) {
    printf("\nTO:\n%s\n", lsb_contents.c_str());
  }

which needs both lsb-release files. We have to find a way to transfer the lsb-release to the new chroot. Is there any easy way to do it? We could for example copy the lsb-release into that tmp folder which we will chroot into and then read it in chromeos_postinst.cc:468:
  if (ReadFileToString("/etc/lsb-release", &lsb_contents)) { ->
  if (ReadFileToString("/lsb-release", &lsb_contents)) {

Does that sound reasonable or I'm doing something wrong?

Sorry, simple copy doesn't work since we mount the partition as read only. I'll think about it more. Any other ideas are welcomed.

Comment 8 by vapier@chromium.org, Nov 22 2017

yes, we could bind mount the old rootfs's /etc over the new rootfs's /etc ... my point is that's wrong and not what we want.  we already have access to the /etc that matches the new rootfs simply by virtue of chrooting into the new rootfs.

i think postinst making any assumptions about the state of / is bad.  any paths it operates on should only be ones explicitly passed into it on the cli.  it makes it a lot easier to keep track of and avoid future bugs like this slipping past.

if UE wants specific details, couldn't it just log them itself ?  let the postinst script log the /etc/lsb-release in the path it was told to run in only, and have UE itself log the existing /etc/lsb-release file.

the update code also implicitly looks in /mnt/stateful_partition/etc/lsb-release btw.
Owner: ahass...@chromium.org
Status: Started (was: Untriaged)
Ok. That sounds like a winner. I'll upload some patches to fix it. Thanks Mike.
Cc: dgarr...@chromium.org
I looked deeper into the code and it seems the postinst needs the source version in order to fix an old FS corruption problem which is for versions less than "0.10.156.2"

https://chromium.git.corp.google.com/chromiumos/platform2/+/619471e97b6197290448c268f9d51974a9ac193f/installer/chromeos_postinst.cc#251

Do we even support devices with this old version anymore? If not, can I get rid of that code? If yes, would passing the source version as a parameter to postinst be fine? I don't really prefer this because adding an input parameter to postinst will have no effect on the devices with UE that don't pass the parameter (older ones). This will potentially cause the postinst to fail. So this approach is probably no brainer.

I think at the end we have to have the source lsb-release somewhere for the postinst to read from.  As vapie@ mentioned, the UE reads from /mnt/stateful_partition/etc/lsb-release which is quite true, but I don't have enough background on this file. I cannot find an instance of it on my devices. Does anyone know if it has the exact same content as /etc/lsb-release? I don't think that is the case since there are places in UE that reads key HWID_OVERRIDE from it which doesn't belong in the system lsb-release.

Can we just copy the source lsb-release somewhere in the stateful partition and read it from there in postinst? and possibly delete it afterwards?

cc'ing dgarrett@ since he has worked on this in the past too.
there shouldn't be a need to support R10 versions anymore.  Lumpy is the oldest device we support currently and it was released on ~R17.  lets prune any such code.
Project Member

Comment 12 by bugdroid1@chromium.org, Nov 28 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/94e8bd508e9f6948c77654cb7003d7c8dc8a3324

commit 94e8bd508e9f6948c77654cb7003d7c8dc8a3324
Author: Amin Hassani <ahassani@google.com>
Date: Tue Nov 28 23:44:24 2017

installer: Remove an old FS corruption patch for version 10 and lower

We do not needs support for this patching anymore. Besides, this patching needs
the source image version and we are in the process of depricating passing
lsb-release of the source image to the postinst. So it is a good place to prune
this code.

BUG= chromium:786225 
TEST=unittest pass; precq pass;

Change-Id: I82726850d8d5eab3e4933723c009a7396fb64669
Reviewed-on: https://chromium-review.googlesource.com/791455
Commit-Ready: Amin Hassani <ahassani@chromium.org>
Tested-by: Amin Hassani <ahassani@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/94e8bd508e9f6948c77654cb7003d7c8dc8a3324/installer/inst_util_unittest.cc
[modify] https://crrev.com/94e8bd508e9f6948c77654cb7003d7c8dc8a3324/installer/inst_util.cc
[modify] https://crrev.com/94e8bd508e9f6948c77654cb7003d7c8dc8a3324/installer/inst_util.h
[modify] https://crrev.com/94e8bd508e9f6948c77654cb7003d7c8dc8a3324/installer/chromeos_postinst.cc

Project Member

Comment 13 by bugdroid1@chromium.org, Nov 29 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/cc8b451b70b61a762418c92ab0a3def6e74c26e7

commit cc8b451b70b61a762418c92ab0a3def6e74c26e7
Author: Amin Hassani <ahassani@google.com>
Date: Wed Nov 29 06:27:25 2017

installer: Stop printing source lsb-release in postinst

postinst should not try to print the source lsb-release because it does
not have access to it. We will show the source lsb-release in the
update_engine itself.

BUG= chromium:786225 
TEST=cros flash and update_engine.log does not show the source lsb-release.

Change-Id: Ic548b911b7a7ae6544352e6a2ef0808e13fb1e30
Reviewed-on: https://chromium-review.googlesource.com/791456
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Amin Hassani <ahassani@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/cc8b451b70b61a762418c92ab0a3def6e74c26e7/installer/chromeos_postinst.cc

Project Member

Comment 14 by bugdroid1@chromium.org, Nov 29 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/741eca3304a35ef55eac49cefe2ce6b5756ed401

commit 741eca3304a35ef55eac49cefe2ce6b5756ed401
Author: Amin Hassani <ahassani@google.com>
Date: Wed Nov 29 23:11:56 2017

installer: print uname when doing postinst

It is a good idea to know which kernel version is running the
postinst. So we print the uname before doing the main job of the
postinst for future debugging purposes.

BUG= chromium:786225 
TEST=cros flash and update_engine.log shows the uname.

Change-Id: I66ed1a1337fac0404d922e9c66fcd963ec2248c4
Reviewed-on: https://chromium-review.googlesource.com/794060
Commit-Ready: Amin Hassani <ahassani@chromium.org>
Tested-by: Amin Hassani <ahassani@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/741eca3304a35ef55eac49cefe2ce6b5756ed401/installer/inst_util_unittest.cc
[modify] https://crrev.com/741eca3304a35ef55eac49cefe2ce6b5756ed401/installer/inst_util.cc
[modify] https://crrev.com/741eca3304a35ef55eac49cefe2ce6b5756ed401/installer/inst_util.h
[modify] https://crrev.com/741eca3304a35ef55eac49cefe2ce6b5756ed401/installer/chromeos_postinst.cc

Project Member

Comment 15 by bugdroid1@chromium.org, Dec 19 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/aosp/platform/system/update_engine/+/afd8cea0060751f969f7d4d20b4c4100ff37626d

commit afd8cea0060751f969f7d4d20b4c4100ff37626d
Author: Amin Hassani <ahassani@google.com>
Date: Tue Dec 19 04:33:04 2017

update_engine: Log the rootfs and stateful partition's lsb-release

postinst cannot not do this anymore, so we need to log it in the UE itself.

BUG= chromium:786225 
TEST=cros_flash two times and made sure they are logged.

Change-Id: I2fbc593f55e6425127de283c78e4170460bac0d9
Reviewed-on: https://chromium-review.googlesource.com/791515
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Amin Hassani <ahassani@chromium.org>
Reviewed-by: Ben Chan <benchan@chromium.org>
Reviewed-by: Alex Deymo <deymo@google.com>

[modify] https://crrev.com/afd8cea0060751f969f7d4d20b4c4100ff37626d/image_properties_chromeos.cc
[modify] https://crrev.com/afd8cea0060751f969f7d4d20b4c4100ff37626d/image_properties_android.cc
[modify] https://crrev.com/afd8cea0060751f969f7d4d20b4c4100ff37626d/image_properties.h
[modify] https://crrev.com/afd8cea0060751f969f7d4d20b4c4100ff37626d/update_attempter.cc

Status: Fixed (was: Started)
This should be fixed by now. I just checked a canary build and it shows different lsb-release. Feel free to verify it.

I'm marking this as fixed.
Status: Verified (was: Fixed)
Verified lsb-release file on AU from 10257.0.0 => 10272.0.0

Sign in to add a comment