Project: chromium Issues People Development process History Sign in
New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 2 users
Status: Archived
Owner: ----
Closed: Jan 2017
Cc:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 3
Type: Bug



Sign in to add a comment
amd64-generic-goofy-release: SignerTest failure
Project Member Reported by drinkcat@chromium.org, Nov 21 2016 Back to list
amd64-generic-goofy-release SignerTest stage has been failing for the past 2 months at least:
https://uberchromegw.corp.google.com/i/chromeos/builders/amd64-generic-goofy-release?numbuilds=200

For example:
https://uberchromegw.corp.google.com/i/chromeos/builders/amd64-generic-goofy-release/builds/863/steps/SignerTest/logs/stdio

15:17:03: INFO: RunCommand: /b/cbuild/internal_master/chromite/bin/cros_sdk 'PARALLEL_EMERGE_STATUS_FILE=/tmp/tmpxrPcfq' -- ./security_test_image '--board=amd64-generic-goofy' in /b/cbuild/internal_master
/b/cbuild/internal_master/chromite/third_party/google/protobuf/__init__.py:37: UserWarning: Module simplejson was already imported from /b/build/third_party/simplejson/__init__.pyc, but /usr/local/lib/python2.7/dist-packages is being added to sys.path
  __import__('pkg_resources').declare_namespace(__name__)
INFO    security_test_image: Loading baselines from /mnt/host/source/cros-signing/security_test_baselines
INFO    security_test_image: Using /mnt/host/source/src/build/images/amd64-generic-goofy/R57-9008.0.0/recovery_image.bin
INFO    security_test_image: Using vboot_reference.git rev 42b74d2677786cf9e135873b574428bdd74bd3fb
INFO    security_test_image: Running ensure_no_nonrelease_files.sh
INFO    security_test_image: Running ensure_sane_lsb-release.sh
Loading config from /mnt/host/source/cros-signing/security_test_baselines/ensure_sane_lsb-release.config... Done.
INFO    security_test_image: Running ensure_secure_kernelparams.sh
Kernel dm= parameter does not match any expected values!
Actual:   1 vroot none ro 1,0 4096000 verity payload=PARTUUID=%U/PARTNROFF=1 hashtree=PARTUUID=%U/PARTNROFF=1 hashstart=4096000 alg=sha1 root_hexdigest=7261e7fd5bad3c124dc06bd2c3357b20be1de611 salt=1a4cec18be8e9da952b2389f2376c78465825faf70eec3f7deb6a200bd303e82
Expected: 
Expected (regex): 
Unexpected kernel parameters found:
 console= loglevel=7 init=/sbin/init cros_secure oops=panic panic=-1 root=/dev/dm-0 rootwait ro dm_verity.error_behavior=3 dm_verity.max_bios=-1 dm_verity.dev_wait=1 noinitrd vt.global_cursor_default=0 kern_guid=%U add_efi_memmap boot=local noresume noswap i915.modeset=1 tpm_tis.force=1 tpm_tis.interrupts=0 nmi_watchdog=panic,lapic 
Debug output:
required_kparams=(
	''
)
required_kparams_regex=(
	''
)
optional_kparams=(
	''
)
optional_kparams_regex=(
	''
)
required_dmparams=(
	''
)
required_dmparams_regex=(
	''
)

kparams='console= loglevel=7 init=/sbin/init cros_secure oops=panic panic=-1 root=/dev/dm-0 rootwait ro dm_verity.error_behavior=3 dm_verity.max_bios=-1 dm_verity.dev_wait=1 dm="1 vroot none ro 1,0 4096000 verity payload=PARTUUID=%U/PARTNROFF=1 hashtree=PARTUUID=%U/PARTNROFF=1 hashstart=4096000 alg=sha1 root_hexdigest=7261e7fd5bad3c124dc06bd2c3357b20be1de611 salt=1a4cec18be8e9da952b2389f2376c78465825faf70eec3f7deb6a200bd303e82" noinitrd vt.global_cursor_default=0 kern_guid=%U add_efi_memmap boot=local noresume noswap i915.modeset=1 tpm_tis.force=1 tpm_tis.interrupts=0 nmi_watchdog=panic,lapic '

dmparams='1 vroot none ro 1,0 4096000 verity payload=PARTUUID=%U/PARTNROFF=1 hashtree=PARTUUID=%U/PARTNROFF=1 hashstart=4096000 alg=sha1 root_hexdigest=7261e7fd5bad3c124dc06bd2c3357b20be1de611 salt=1a4cec18be8e9da952b2389f2376c78465825faf70eec3f7deb6a200bd303e82'

kparams_nodm='console= loglevel=7 init=/sbin/init cros_secure oops=panic panic=-1 root=/dev/dm-0 rootwait ro dm_verity.error_behavior=3 dm_verity.max_bios=-1 dm_verity.dev_wait=1  noinitrd vt.global_cursor_default=0 kern_guid=%U add_efi_memmap boot=local noresume noswap i915.modeset=1 tpm_tis.force=1 tpm_tis.interrupts=0 nmi_watchdog=panic,lapic '

mangled_dmparams='1 vroot none ro 1,0 4096000 verity payload=PARTUUID=%U/PARTNROFF=1 hashtree=PARTUUID=%U/PARTNROFF=1 hashstart=4096000 alg=sha1 root_hexdigest=MAGIC_HASH salt=MAGIC_SALT'

(actual error will be at the top of output)
ERROR   security_test_image: secure_kernelparams: test failed
INFO    security_test_image: Running ensure_not_ASAN.sh
ERROR   security_test_image: 1 tests failed
15:17:19: ERROR: 
return code: 1; command: /b/cbuild/internal_master/chromite/bin/cros_sdk 'PARALLEL_EMERGE_STATUS_FILE=/tmp/tmpxrPcfq' -- ./security_test_image '--board=amd64-generic-goofy'
cwd=/b/cbuild/internal_master, extra env={'PARALLEL_EMERGE_STATUS_FILE': '/tmp/tmpxrPcfq'}

This appears to be somewhat related to  Issue 605595 , but the failure message looks different.
 
Comment 1 by hungte@chromium.org, Nov 21 2016
I think it's almost the same.

vapier, have you pushed the change to signer?
It fails when run locally too, so I don't think anything needs to be pushed to the signer.

In  issue 605595 , the fix https://chromium-review.googlesource.com/#/c/371678 only modified ensure_sane_lsb-release.sh, but the test that is failing now is ensure_secure_kernelparams.sh . I'm not sure if we can safely change get_board_from_lsb_release in src/platform/vboot_reference/scripts/image_signing/common.sh.
Comment 3 by vapier@chromium.org, Nov 22 2016
here's the error:
Kernel dm= parameter does not match any expected values!

it's because you've defined a rootfs size that the signer doesn't permit.  look at the rootfs_sizes= size field in the ensure_secure_kernelparams.config file.  you're using:
  4096000
Yes, and in the few lines below, you see:
Kernel dm= parameter does not match any expected values!
Actual:   1 vroot none ro 1,0 4096000 verity payload=PARTUUID=%U/PARTNROFF=1 hashtree=PARTUUID=%U/PARTNROFF=1 hashstart=4096000 alg=sha1 root_hexdigest=7261e7fd5bad3c124dc06bd2c3357b20be1de611 salt=1a4cec18be8e9da952b2389f2376c78465825faf70eec3f7deb6a200bd303e82
Expected: 
Expected (regex): 

These 2 required_dmparams variables are empty:
required_dmparams=(
	''
)
required_dmparams_regex=(
	''
)

The reason being that ensure_secure_kernelparams.sh calls:
local board=$(get_board_from_lsb_release "${rootfs}")

Which returns "amd64_generic", which is what I tried to explain in #2.

But, yeah, once we add the amd64-generic in boardnames.config, the test starts failing because of the rootfs size.

Fix here: https://chrome-internal-review.googlesource.com/306755

But then presubmit hook is not happy:
error: board amd64-generic is missing an appid
Comment 5 by hungte@chromium.org, Nov 23 2016
I think we can change get_board_from_lsb_release to do same thing as CL:371678. Mike, do you agree?
Comment 6 by vapier@chromium.org, Nov 23 2016
that sounds fine
Project Member Comment 7 by bugdroid1@chromium.org, Nov 24 2016
The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/cros-signing/+/721c0d5c99084366cecab7c22bed75bfc005ef24

commit 721c0d5c99084366cecab7c22bed75bfc005ef24
Author: Nicolas Boichat <drinkcat@google.com>
Date: Wed Nov 23 00:42:14 2016

Project Member Comment 8 by bugdroid1@chromium.org, Nov 24 2016
The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/cros-signing/+/721c0d5c99084366cecab7c22bed75bfc005ef24

commit 721c0d5c99084366cecab7c22bed75bfc005ef24
Author: Nicolas Boichat <drinkcat@google.com>
Date: Wed Nov 23 00:42:14 2016

Project Member Comment 9 by bugdroid1@chromium.org, Nov 30 2016
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/vboot_reference/+/1e9245dfff914107ec06aac84f3b70c2df1f4a41

commit 1e9245dfff914107ec06aac84f3b70c2df1f4a41
Author: Mike Frysinger <vapier@chromium.org>
Date: Wed Nov 23 17:22:29 2016

image_signing: unify board extraction logic from lsb-release

We had two places extracting the board value from lsb-release and parsing
the output by hand.  Unify them to use the same parsing logic to avoid
desynchronized behavior.

We also create a new get_boardvar_from_lsb_release helper to unify the
board name -> variable name mangling logic.

BUG= chromium:667192 
TEST=`./security_test_image --board samus` still detects the correct board
BRANCH=None

Change-Id: If88a8ae59b9c9fd45ddd796653a0173ed0186d2d
Reviewed-on: https://chromium-review.googlesource.com/414224
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Nicolas Boichat <drinkcat@chromium.org>

[modify] https://crrev.com/1e9245dfff914107ec06aac84f3b70c2df1f4a41/scripts/image_signing/common.sh
[modify] https://crrev.com/1e9245dfff914107ec06aac84f3b70c2df1f4a41/scripts/image_signing/ensure_sane_lsb-release.sh
[modify] https://crrev.com/1e9245dfff914107ec06aac84f3b70c2df1f4a41/scripts/image_signing/ensure_no_nonrelease_files.sh
[modify] https://crrev.com/1e9245dfff914107ec06aac84f3b70c2df1f4a41/scripts/image_signing/ensure_secure_kernelparams.sh

Project Member Comment 10 by bugdroid1@chromium.org, Jan 3 2017
The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/cros-signing/+/e65620a4517f81c245a9e44d0387e9ad512c2d22

commit e65620a4517f81c245a9e44d0387e9ad512c2d22
Author: Nicolas Norvez <norvez@google.com>
Date: Thu Dec 22 19:59:08 2016

Project Member Comment 11 by bugdroid1@chromium.org, Jan 3 2017
The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/cros-signing/+/e65620a4517f81c245a9e44d0387e9ad512c2d22

commit e65620a4517f81c245a9e44d0387e9ad512c2d22
Author: Nicolas Norvez <norvez@google.com>
Date: Thu Dec 22 19:59:08 2016

Cc: norvez@chromium.org
Status: Fixed
Looks fixed to me. amd64-generic-goofy-release and amd64-generic-cheets-release are now green, the other canaries were already red before the merge as far as I can tell.
Comment 13 by dchan@google.com, Mar 4 2017
Labels: VerifyIn-58
Labels: VerifyIn-59
Labels: VerifyIn-60
Labels: VerifyIn-61
Status: Archived
Sign in to add a comment