New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 14 users

Issue metadata

Status: Verified
Last visit > 30 days ago
Closed: Aug 2012
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 0
Type: Bug

  • Only users with EditIssue permission may comment.

Sign in to add a comment

Issue 142440: Hang at Chrome logo after AU from Stable to earlier channel

Reported by, Aug 13 2012 Project Member

Issue description

Chrome Version     :  22.0.1229.0
OS Version         :  2723.0.0
Type of computer   :  CR-48

What steps will reproduce the problem?
1. System in dev mode
2. Get update to latest 2723.0.0 (R22)

What is the expected output?
System Boot

What do you see instead?
System stays in Chrome logo screen

Additional Information:
Seems to be happening on dev mode only

From /var/logs/messges I see a lot of these messages printed out:
crash_client: connect to server failed.
wlan0: detected beacon loss from AP - sending probe request
wlan0: cancelling probereq due to a received beacon

Seems like connectivity is working though so this may be UI issue

Comment 1 by, Aug 13 2012

Summary: CR-48 not booting in dev-mode - hangs in Chrome logo after AU to 22

Comment 2 by, Aug 13 2012

Labels: -Area-Network Area-UI
Status: Available
Debugged this issue with benchan@ and it looks like UI crashing/hanging

Comment 3 by, Aug 13 2012

Status: Assigned
+OOBE team

Perhaps some of the new OOBE work might be to blame? Josafat has the device if that would help for any of the MTV-based folks.

@derat - Not sure if you've got any ideas here? With @nkostylev out, I'm going to tentatively assign this to you for a quick look. If it looks OOBE-related, please pass on to @mlchan, who can find the right person in Moscow. Thanks!

Comment 4 by, Aug 13 2012

Labels: OS-Chrome

Comment 5 by, Aug 13 2012

If it's OOBE/sign in screen related (Chrome in crash loop?) then Ivan may be able to help.
Ideally would be to grab logs (generate_logs script) + client ID of that machine (assuming it has crash reporting enabled and they will show up on crash server).

Comment 6 by, Aug 13 2012

A possible lead - Nikita had fixed a P0 bug with WebUI not starting, which was merged a few hours ago:

Comment 7 by, Aug 13 2012

Some other folks also seeing this:!category-topic/chromebook-central/dev/_cMFiayhluY%5B1-25%5D

I don't see any files under /var/spool/crash

As mentioned, I have my system for anyone who wants to take a look

Comment 8 by, Aug 13 2012


Comment 10 by, Aug 13 2012

 Issue 142457  has been merged into this issue.

Comment 11 by, Aug 13 2012

Labels: ConOps-Issue
In case others don't go back to comment 10 - that's this bug occurring on an Alex (not a Cr-48).

Comment 13 by, Aug 14 2012

Dan is out too.  If we're confident this only happens in dev mode, is it really a P0?

I will take a look at the system logs, but the odds are good that the breakage is in Chrome, and I haven't landed a chrome change in about 18 months or more :-D

Comment 14 by, Aug 14 2012

Labels: -Pri-0 Pri-1
Setting as P1 since it appears to be in dev mode only

Comment 15 by, Aug 14 2012

The last couple chrome logs say this:

[] Blocking on UI thread in OwnershipService::GetStatus

and not much else.  Assigning to Mattias, since he's the most reasonable proxy for Julian, who I think knows that code best these days.

Comment 16 by, Aug 14 2012

Then again, that "Blocking on UI thread" line is present in every log file and harmless.

I'll still try to reproduce this. Can somebody answer these questions:
- I'm having trouble getting an AU upgrading a Mario from R20. Have we pulled the plug?
- Does the problem reproduce when I go through recovery with a downloaded R22 image?

Comment 17 by, Aug 14 2012

Mario updates are once again enabled, so it should now work.

Comment 18 by, Aug 14 2012

I just tried install 22 through a recovery image. Afterwards the system booted just fine to the login screen, and I could log in and take ownership etc.

I'll now reinstall a 21 recovery image and apply the update to see what happens.

Comment 19 by, Aug 14 2012

On a machine with dev switch flipped, I just installed R21 2465.121.0, logged in, owned the device, then went through AU and received R22 2723.0.0. The device booted just fine to the login screen.

I'm afraid I can't do anything if I you don't have accurate reproduction steps or get me access to the device (which is hard since I'm in Munich). Reassigning to cmasone@ to find a new owner in MTV.

Comment 20 by, Aug 14 2012

Owner: ----
Status: Available
This is pretty clearly a UI issue, and it would be great if someone on that team could take action on this issue.

Comment 21 by, Aug 14 2012

Labels: -ReleaseBlock-Dev ReleaseBlock-Beta
Status: Assigned
saintlou - could you please find an owner while Zel is ooo?

Comment 22 by, Aug 14 2012

FWIW, based on logs and the stateful dump, the encrypted partition correctly migrated disk contents and set up the encryption key, so it doesn't appear to be related to the encrypted partition code (i.e. it's not And the NVRAM was already finalized, so it's not either.

Comment 23 by, Aug 14 2012

Is the stateful dump available somewhere? I'd be interested in the contents of /var/lib/whitelist/ to make sure ownership and device settings are OK.

Comment 24 by, Aug 14 2012

If Josafat this has his machine in this state we can grab that data.

Comment 25 by, Aug 14 2012

I have asked Dave to help.

Comment 26 by, Aug 14 2012

FWIW, I believe I'm seeing the same symptom (machine hung at white background / chrome logo screen on bootup), on an Alex, which is *not* in dev mode.

Comment 27 by, Aug 14 2012

A problem like this has also been seen on a Lumpy on canary channel.
I was able to force the unit to roll back by power cycling it enough times.
On reboot, I obtained various logs.  I've attached the contents of
/var/log/messages from around the failure.

Key timestamps in the log:
  08-10T14:31 - reboot with updated image starts.
  08-10T14:43 - unit goes to sleep after being hung

Some logs for activity before the reboot are in the attached file:
more logs are available.
1.9 MB View Download

Comment 28 by, Aug 14 2012

Looking through the log file, I found all the reboots for the rollback.
In particular, I note the kernel banner for the last two boots:


2012-08-14T13:40:07.680957-07:00 localhost kernel: [    0.000000] Linux version 3.4.0 (chrome-bot@build30-m2) (gcc version 4.6.x-google 20120301 (prerelease) (gcc-4.6.3_cos_gg_2a32ae6) ) #1 SMP Thu Aug 9 23:09:30 PDT 2012
2012-08-14T13:40:33.721068-07:00 localhost kernel: [    0.000000] Linux version 3.4.0 (chrome-bot@build40-m2) (gcc version 4.6.x-google 20120301 (prerelease) (gcc-4.6.3_cos_gg_2a32ae6) ) #1 SMP Thu Jun 14 10:43:09 PDT 2012

Based on the time stamps, it appears that the working version was 2439.0.0,
and the failed update was to 2735.0.0.

Comment 29 by, Aug 14 2012

OK.  After a brief application of brainpower, I took a look at the Chrome
logs.  I found the following:

[] Blocking on UI thread in OwnershipService::GetStatus
[] Waiting to load statistics. Requested statistic: hardware_class
[] Unable to read statistics file: /tmp/machine-info
[] Unable to read statistics file: /var/cache/echo/vpd_echo.txt

The "Unable to read ..." error message is normal, based on other logs from successful
boots.  Looking at Chrome logs for successful boots, there should be much more.
So, something caused Chrome to stop partway through the boot.  I don't see evidence
of any crash, so I suspect Chrome is deadlocked.

I've attached sample Chrome logs from a successful and failed boot.
460 bytes View Download
4.1 KB View Download

Comment 30 by, Aug 15 2012

I took the failed lumpy, and walked it through a few more paces:
  * While I was debugging, the unit downloaded a new update.
  * When I finished debugging, I logged out, and the unit automatically
    rebooted to apply the new update.
  * After the reboot, the problem repeated.
  * I again forced the unit to roll back by power cycling it, then
    switched the unit to dev mode.  This wiped the stateful
  * After rebooting in dev mode, the unit downloaded and applied
    yet another update.
  * The final update was successful; the unit is now running 2754.0.0.

Conclusion:  Something in the stateful partition is contributing to the

Factors that may be required to reproduce the problem:
  * The unit was powered off for an extended time.
  * The unit was running a sufficiently old version.  Nothing
    older than (about) 2440.0.0 should be necessary.
  * The unit had been in use for some time before applying
    the update.
  * The update needs to be to something relatively recent.

This reinforces comment #19:  You can't reproduce this
problem by simply recovering to an old image and letting it

One possible option for reproducing the problem:
  * Recover a unit to stable channel
  * Use the unit
  * Switch to canary channel
  * Let the unit update and reboot

Comment 31 by, Aug 15 2012

+ bshe@ who worked on background animation

Brief looking to the code it looks like we might wait for background animation completion NOTIFICATION_WALLPAPER_ANIMATION_FINISHED (see chrome/browser/chromeos/login/ But as far as I can see we get this notification only in case of image background (see ash/desktop_background/ It looks like in case solid background we need to generate NOTIFICATION_WALLPAPER_ANIMATION_FINISHED too. But I might miss something.

Comment 32 by, Aug 15 2012

If the problem is the lack of the wallpaper animation notification,
what can a user do to create this condition?

Comment 33 by, Aug 15 2012

It looks like user can't do it in current UI. But we reproduced hang wit Ivan by editing "Local State" to set solid color background for the first user.

Comment 34 by, Aug 15 2012

Perhaps this is why it's affecting machines that are upgrading?  A device on which the first user never chose a background image, and then it upgrades to a build that has this animation feature?

Comment 35 by, Aug 16 2012

In what build was this wallpaper problem introduced?

Comment 36 by, Aug 16 2012

Prepared CL that send NOTIFICATION_WALLPAPER_ANIMATION_FINISHED in case of solid background. But I'm not 100% that it is exactly the same hang as in this issue.

Comment 37 by, Aug 16 2012

Labels: -Pri-1 -Sev-1 -ReleaseBlock-Beta Pri-0 Sev-0 ReleaseBlock-Dev
Summary: CR-48 not booting - hangs in Chrome logo after AU to 22
This is NOT restricted to dev mode, and potentially NOT restricted to Cr-48 (probably happening on all platforms). Jrbarnette has reliable repro of going from Stable -> Dev 22 (will be testing Stable -> R21 Beta).

Comment 38 by, Aug 16 2012

Summary: Hang at Chrome logo after AU to 22
I've got what looks like a reliable test case.  Here's what I did:
 1) Took a brand new Lumpy on stable channel, went through OOBE, and
    logged in.
 2) Switched to Dev channel.

The unit updated, and on reboot, the unit hung as described.

Given the symptoms to date, it looks like this problem will occur on all
platforms; I've updated the summary to reflect this.

Comment 39 by, Aug 16 2012

Labels: Mstone-21
Summary: Hang at Chrome logo after AU from Stable to earlier channel
He's also seen this Stable -> Beta.

Comment 40 by, Aug 16 2012

I've reproduced this using the same Lumpy device, but simply by switching from
Stable to Beta.  The original Stable build prior to the update was 2268.124.0,
Chrome 20.0.1132.59.

Comment 41 by, Aug 16 2012

What exactly do you see on the screen? If it is the hang that I'm fixing with solid background, you should see white (or almost white) screen without logo (i.e. logo disappears).

Comment 42 by, Aug 16 2012

@jrbarnette what wallpaper are you using in your repro case?  We tried from 21 and could not reproduce.  It sounds like it will only happen from 20.

Comment 43 by, Aug 16 2012

The hang I see happens with the Chrome logo from the boot splash screen is still
present, not a blank screen.

I just re-tested with the same unit in dev mode:  I updated from Stable to Beta,
and from Beta to Dev without seeing the hang.  I don't yet know whether this
means that the problem can't occur in dev mode, or if there's something clobbered
by the stateful wipe that I don't know about.

Comment 44 by, Aug 16 2012

Could someone in MTV get device in hang state and run debugger on it to check what actually happens?  Does Chrome process still alive? Create core dump for all Chrome processes, etc. In the logs I don't see anything suspicious. In M21 new OOBE and animated background code shouldn't work.

Comment 45 by, Aug 16 2012

Currently, the only reproducible test cases have involved MP-signed units
not in dev mode.  Attaching a debugger isn't possible.

My earlier investigation (see comments 27-30) showed that a) Chrome
started, b) Chrome didn't crash.

Given that we've seen this on M21, and given that the symptoms don't match
what we see for the wallpaper problem, I think the new wallpaper animation
isn't involved.

Comment 46 by, Aug 16 2012

I've reproduced the problem again.  For anyone who wants, here's a repeatable
 1) With the dev switch off, recover your test unit to Stable channel.
 2) After recovery, go through OOBE, log in, and switch to Dev channel.
 3) When the update to Dev channel is complete, log out and the system
    will reboot and hang.

Comment 47 by, Aug 16 2012

I've successfully reproduced the problem by running recovery in dev mode.
I did a preliminary check on VT2, and Chrome is in fact running.  `top` indicates
that the system is idle, so this looks like some sort of deadlock, or a
missing event.

Comment 48 by, Aug 16 2012

so, chrome renderer and gpu process never start here. I've made a dump of chrome browser process. rkc@ is now looking into that core dumps that we collected.

Comment 49 by, Aug 17 2012

Managed to get a stack trace of the issue, the problem is indeed with the wallpaper migration. So, this is the stack trace,

#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
#1  0x00007f5651d06215 in _L_lock_883 () from /lib64/
#2  0x00007f5651d0606a in __pthread_mutex_lock (mutex=0x7f56585efdb0) at pthread_mutex_lock.c:61
#3  0x00007f565301a903 in Acquire () at ./base/synchronization/lock.h:22
#4  AutoLock () at ./base/synchronization/lock.h:100
#5  get () at chrome/browser/chromeos/login/
#6  chromeos::UserManager::Get () at chrome/browser/chromeos/login/
#7  0x00007f5653022b96 in chromeos::WallpaperManager::SaveUserWallpaperProperties (this=<value optimized out>, email=..., type=chromeos::User::DEFAULT, 
    index=16) at chrome/browser/chromeos/login/
#8  0x00007f565301e05f in chromeos::UserManagerImpl::MigrateWallpaperData (this=<value optimized out>)
    at chrome/browser/chromeos/login/
#9  0x00007f5653020442 in chromeos::UserManagerImpl::UserManagerImpl (this=0x7f5658607400) at chrome/browser/chromeos/login/
#10 0x00007f565301a9bd in get () at chrome/browser/chromeos/login/
#11 chromeos::UserManager::Get () at chrome/browser/chromeos/login/
#12 0x00007f5653648599 in chrome::OpenAsh () at chrome/browser/ui/ash/
#13 0x00007f5653182975 in ChromeBrowserMainExtraPartsAsh::PreProfileInit (this=0x7f5658555e00)
    at chrome/browser/ui/views/ash/
#14 0x00007f565338d27e in ChromeBrowserMainParts::PreProfileInit (this=0x7f565857edc0) at chrome/browser/
#15 0x00007f5653391a27 in ChromeBrowserMainPartsLinux::PreProfileInit (this=0x7f565857edc0) at chrome/browser/
#16 0x00007f5652fc3400 in ChromeBrowserMainPartsChromeos::PreProfileInit (this=0x7f565857edc0) at chrome/browser/chromeos/
#17 0x00007f56533903c7 in ChromeBrowserMainParts::PreMainMessageLoopRunImpl (this=0x7f565857edc0) at chrome/browser/
#18 0x00007f565339160e in ChromeBrowserMainParts::PreMainMessageLoopRun (this=0x7f565857edc0) at chrome/browser/
#19 0x00007f56555b85dc in content::BrowserMainLoop::CreateThreads (this=0x7f565856bd80) at content/browser/
#20 0x00007f56555b8d63 in (anonymous namespace)::BrowserMainRunnerImpl::Initialize (this=0x7f5658553940, parameters=...)
    at content/browser/
#21 0x00007f56555b6ee7 in BrowserMain (parameters=...) at content/browser/
#22 0x00007f56539ada30 in content::ContentMainRunnerImpl::Run (this=0x7f5658560fa0) at content/app/
#23 0x00007f56539abf31 in content::ContentMain (argc=<value optimized out>, argv=0x7fffc78a3a18, delegate=0x7fffc78a38e0) at content/app/
#24 0x00007f5652f903c8 in ChromeMain (argc=43, argv=0x7fffc78a3a18) at chrome/app/
#25 0x00007f5650c6841d in __libc_start_main (main=0x7f5652f90380 <main(int, char const**)>, argc=43, ubp_av=0x7fffc78a3a18, init=<value optimized out>, 
    fini=<value optimized out>, rtld_fini=<value optimized out>, stack_end=0x7fffc78a3a08) at libc-start.c:234
#26 0x00007f5652f902a9 in _start ()

If you look at #10, you'll see that we're in UserManager::Get, where we've just done an Autolock 2 lines before it.

Then we call MigrateWallpaperData at #8, which eventually calls chromeos::WallpaperManager::SaveUserWallpaperProperties at #7, which does a check for Ephemeral users, which in turn does another UserManager::Get.

So on #5, we hit the lock again, creating a deadlock (since of course, we haven't finished constructing so we haven't released the lock yet).

This was probably introduced by this CL ( 3 weeks ago.

I'll speak with Zel about a fix and update.

Comment 50 by, Aug 17 2012

Labels: -Mstone-21
ok, M21 can safely sail now... this CL wasn't there to begin with.

Comment 51 by, Aug 17 2012

Assigning back to bshe@ to fix since I am not a 100% on how to fix it. Don't want to break something else trying to fix this problem :)

Comment 52 by Deleted ...@, Aug 17 2012

Comment 53 by, Aug 17 2012

Since it is night in MTV. I created quick patch that fixes this issue:

I'm not familiar with this code so please feel free to ignore it and propose different solution. This issue can be easily reproduced on TOT by removing UserWallpapersProperties section in Local State, it triggers wallpaper migration that hangs on recursive lock acquire inside UserManager::Get().

Comment 54 by, Aug 17 2012

Labels: Hotlist-Link

Comment 55 by, Aug 17 2012

Status: Started
just FYI: should also fix the issue.
Looks like the ephemeral user check logic is incorrect. It shouldn't be in SaveUserWallpaperProperties. I have removed it from the function.

Comment 56 by, Aug 17 2012

Labels: Iteration-62
Adding Iteration

Comment 57 by, Aug 17 2012


Comment 58 by, Aug 17 2012

Labels: AddAutomatedTest

Comment 59 by, Aug 17 2012

Project Member
The following revision refers to this bug:

r152136 | | 2012-08-17T19:49:11.171352Z

Changed paths:

Remove call to UserManager::Get when initializing UserManager

This CL removes UserManager::Get()(for ephemeral user check) out from
WallpaperManager::SaveUserWallpaperProperties(called when initializing UserManager).
It was creating a deadlock and make chrome hang on startup screen.

The ephemeral user check in SaveUserWallpaperProperties is not correct either.
We should not check if current logged in user is ephemeral user while we try
to save wallpaper properties for some other user specified by email.

BUG= 142440 

Review URL:

Comment 60 by, Aug 17 2012

FYI about how to reproduce the hang:

As described in #49 by rkc@, the hang happens when we try to migrate wallpaper data. And the condition to trigger wallpaper migration is when we have logged
 in users saved in LocalState but no corresponding new wallpaper properties saved.

For R20 and before, we used old wallpaper properties. And for R21 and newer, we will try to migrate the old wallpaper properties to new properties. So:
> 1. if update from R21 to R22, we would not migrate any data. Chrome should not hang.
> 2. if update from R20 to R21. The cl introduced the hang is landed at R22. We should also be fine here.
> 3. if update from R20 to R22. We need to migrate data which saved in R20 to R22. So migrate wallpaper data get called and cause
the hang.

Comment 61 by, Aug 17 2012

OK.  After further testing, I've confirmed that this problem cannot be reproduced
on R21.

The test case in R20 where I saw a failure after switching to R21 was actually
bug chromium-os:33635.  The upshot is because of a race in update_engine, I
requested Beta channel for R21, but got Dev channel despite the request.

Comment 62 by, Aug 18 2012

Labels: Merge-Requested

Comment 63 by, Aug 20 2012

Labels: Iteration-63

Comment 64 by, Aug 20 2012

Labels: -Merge-Requested Merge-Approved

Comment 65 by, Aug 20 2012

Project Member
Labels: -Merge-Approved merge-merged-1229
The following revision refers to this bug:

r152332 | | 2012-08-20T16:16:06.617114Z

Changed paths:

Merge 152136 - Remove call to UserManager::Get when initializing UserManager

This CL removes UserManager::Get()(for ephemeral user check) out from
WallpaperManager::SaveUserWallpaperProperties(called when initializing UserManager).
It was creating a deadlock and make chrome hang on startup screen.

The ephemeral user check in SaveUserWallpaperProperties is not correct either.
We should not check if current logged in user is ephemeral user while we try
to save wallpaper properties for some other user specified by email.

BUG= 142440 

Review URL:
Review URL:

Comment 66 by, Aug 20 2012

Status: Fixed
Looks like AddAutomatedTest is tracked here:

Mark this one as fixed.

Comment 67 by, Aug 21 2012

Status: Verified

Dev -Mode
Chrome Version: 22.0.1229.12
Chrome OS: 2723.25.0

Comment 68 by, Aug 21 2012

Can you please also verify that this is not happening in normal mode? The
problem was not restricted to dev mode.

Comment 69 by, Aug 22 2012

Status: Fixed
Marking as fixed to verify in normal mode.

Comment 70 by, Aug 22 2012

Status: Verified
Verified on 2723.27.0 dev.

Comment 71 by, Oct 13 2012

Project Member
Labels: Restrict-AddIssueComment-Commit
This issue has been closed for some time. No one will pay attention to new comments.
If you are seeing this bug or have new data, please click New Issue to start a new bug.

Comment 72 by, Mar 10 2013

Project Member
Labels: -Area-UI -Mstone-22 M-22 Cr-UI

Comment 73 by, Mar 14 2013

Project Member
Labels: -Restrict-AddIssueComment-Commit Restrict-AddIssueComment-EditIssue

Sign in to add a comment