starting testing:
guado_labstation-release/R59-9384.0.0 (build 948)
beaglebone_servo-release/R59-9384.0.0 (build 944)

Test of beaglebone failed. Requested logs and re-image

https://b.corp.google.com/u/0/issues/36458005

guado_labstation-release/R59-9384.0.0 (build 948) confirmed good for labstation.

Working with Richard. Looks like it's kernel problem.

From serial console
> Error: Kernel with LPAE support, but CPU does not support LPAE.

Can someone from the Kernel team comment?

Passing to this week's deputy.  But really, this needs action from
the kernel team.

Before we update the servo versions, can we do it during a low-peak time?  I've haven't updated the labstation version before and seeing as how there are >200 devices that will be impacted by the update and I'd like to monitor how those devices will do when it comes time to update and if I need to tweak the process (extend wait time, shorten, etc).

Seems we can't update servo versions since test is failed and it's a kernel problem.

@kevin, can you add some kernel team's experts to this bug to investigate that kernel errors?

> Before we update the servo versions, can we do it during a low-peak time?

"Low-peak" isn't as important as "observable", meaning we just need to
do it at a time when someone is on-duty and has time to watch what's
happening, and revert if necessary.

That said, it's moot until the beaglebone kernel issue is addressed.

There's a decent chance that this CL is our trouble:
    https://chromium-review.googlesource.com/#/c/415444/

We can test that one of two ways:
  * Build a beaglebone_servo image with that change reverted,
    and see if the problem goes away.
  * Test builds 9040.0.0 and 9041.0.0, and see if the problem
    was new in 9041.

I'm not clear how to do the tests mentioned in Comment #12.

Any doc or instructions I can refer to if I have a local beaglebone_servo (assuming it works well I don't use it for a long time) on my desk?

Ping.  xixuan@ - can you deal with this, or should we pass to the
next deputy?

I've tested 9040.0.0 and 9041.0.0.  Both builds appear to work.
So, whatever caused LPAE to be included in the beaglebone kernels,
it didn't happen until after that initial CL.

At this point, we still need help from someone who understands
the kernel2 eclass, so we can figure out what causes LPAE to be
turned on for beaglebone.  Our alternative is to bisect across
some 340 builds.

amstan, could you help take a look here?

TLDR: Looks like some attempts at speeding up chrome enabled a few kernel options for everything and affected beaglebone (which has an old cpu where those features (LPAE) don't exist). https://bugs.chromium.org/p/chromium/issues/detail?id=569963

Here's a fix: https://chromium-review.googlesource.com/c/469132/

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/30c4ffb7870c5b3e88a2873f1f842232f7d33e74

commit 30c4ffb7870c5b3e88a2873f1f842232f7d33e74
Author: Alexandru M Stan <amstan@chromium.org>
Date: Thu Apr 06 02:55:12 2017

Remove USE=transparent_hugepage for beaglebone

transparent_hugepage is a speed optimization, we really don't need the
beaglebone to be that fast (especially in chrome, given that it has no
UI). And the cpu doesn't support CONFIG_ARM_LPAE, which
transparent_hugepage now pulls in.

BUG= chromium:703361 ,  chromium:569963 
TEST=Build and boot a beaglebone

Change-Id: Idd1c8789f631efccada91b3a80057994b4586f2c
Signed-off-by: Alexandru M Stan <amstan@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/469132
Tested-by: Richard Barnette <jrbarnette@chromium.org>
Reviewed-by: Sonny Rao <sonnyrao@chromium.org>
Reviewed-by: Yunlian Jiang <yunlian@chromium.org>

[modify] https://crrev.com/30c4ffb7870c5b3e88a2873f1f842232f7d33e74/overlay-beaglebone/make.conf

OK.  Beaglebone images should work as of the latest build,
R59-9436.0.0.  So, let's test that, and see if we can move
the lab forward.

The primary deputy has too much going on, so the secondary
deputy gets this one.

OK, R59-9436.0.0 has passed basic sanity testing on
beaglebone and guado labstations.  So, we're go to
upgrade both of them simultaneously.

I note that this is the first update to affect labstations
in production, so we'll need to monitor the progress to
make sure that there's no unexpected disruption.

We're on our way!
    $ atest stable_version modify -b beaglebone_servo -i $BUILD
    Stable version for board beaglebone_servo is changed from R56-8999.0.0 to R59-9436.0.0.
    $ atest stable_version modify -b guado_labstation -i $BUILD
    Stable version for board guado_labstation is changed from R56-8964.0.0 to R59-9436.0.0.