Observed issue in Pit device.

Logs from elm device: https://storage.cloud.google.com/chromiumos-test-logs/bugfiles/cr/845219/debug-logs_20180521-111845-elm.tgz?_ga=2.44249830.-317153427.1526666211

Issue 845223 has been merged into this issue.

We can load pages after this, and this only means the user never sees the new tab page?

Can we confirm this only happens on ARM devices (e.g. it happens on Pit/Elm/Daisy, but not on Samus/Peppy/Caroline)?

Its not just with new tab page. We cannot load any webpages, even after sign-out and sign-in or rebooting device.

'Aw, snap' error is also seen on opening settings app. Unable to launch Get Help app.

Reproduced only on ARM devices so far. No repro on eve, caroline, peppy.

I tried three ARM devices ( elm, kevin and pi) but issue only reproduce on elm

No repro on veyron-minnie as well (ARM device)

Unable to reproduce the issue on 10701.0.0/68.0.3436.0 - Daisy, reks, kip.  I'm able to navigate any sites without the issue. 

So this only appears to happen on Elm and Pit (comment 1)?

If so we are fairly sure this is only on them, we can skip just these device families and go ahead with the rest. 

I would think this would trigger autotest failures, we can watch out for that on the build tonight.

+djkurtz for thoughts on what might be unique to MTK here (ignoring the Pit part for now)...

This is a Chrome crash. Please go to chrome://crashes, upload crashes, and then report back a crash ID.
Also, can you confirm that this does not happen on earlier Chrome / OS versions?

Crashes from the machine in #2 are at: https://crash.corp.google.com/browse?q=ClientID%3D%273f318319154a458592d2f137e8a1fcec%27&stbtiq=&reportid=&index=0

Lots of selinux crashes of various sorts.

https://crash.corp.google.com/browse?q=ClientID%3D%273f318319154a458592d2f137e8a1fcec%27+AND+product_name%3D%27Chrome_ChromeOS%27&stbtiq=&reportid=8a1eb16b261e9a3b&index=0 has the Chrome crashes.

Stacks look suspicious but point at ClearContentFrame

xdai@ can you or someone on your team dig into this?  I suspect the cause is way upstream of us (Blink?) but we'll need to do some initial investigation.

I also suggest that we do a new build for the affected devices. Crashes like this often just "go away"; usually because they're side effects of other issues upstream teams end up fixing that we don't find out about.

From the log in #2, there are lots of error message like:
[1256:1449:0521/111809.337809:ERROR:service_manager_context.cc(250)] Attempting to run unsupported native service: /opt/google/chrome/chrome_renderer.service

This error seems related to this switch --enable-service-binary-launcher, though it's not in production yet. +sky@, do you think it might related to the crash we are seeing here？ 

At the same time, I'll get an elm and see if I can repro the crash.

I didn't repro it on the elm device with version:
Chrome: 68.0.3433.0 && 68.0.3437.0
Platform: 10689.0.0 (Official Build) dev-channel elm test
ARC: 4787408

But looking at the GoldenEye, elm doesn't have a green build since 5/18 with Chrome OS version 10694.0.0 (Chrome version 68.0.3434.2). Since Chrome seems fine (see comment #17), I suspect something is wrong between Chrome OS version 10691 and 10694. 

We enabled thinLTO on elm release from 10672 and pit release from 10688. Is that possibly related?

Confirmed that for elm, Chrome OS version 10691 is the last known good version, and 10694 is the first known bad version. Chrome should be fine since on 10691, I flashed 68.0.3433.0 and also the latest Chrome, everything works fine.

I flashed 10694 (with Chrome version 68.0.34) to my elm device and could not even get to the login screen. The system keeps crashing. See the attached log. Kernel crash is observed in the log. 

Re#19: since it's enabled on elm from 10672, I don't think it's related. But someone from Chrome OS team better take a look at this issue. 

Selected kernel error message snippet:
2018-05-22T20:46:59.155334+00:00 INFO kernel: [   18.845105] chrome[1611]: unhandled level 2 translation fault (11) at 0x000effc6, esr 0x92000006
2018-05-22T20:46:59.155352+00:00 ALERT kernel: [   18.845139] pgd = ffffffc0f911f000
2018-05-22T20:46:59.155355+00:00 ALERT kernel: [   18.845150] [000effc6] *pgd=0000000139121003, *pud=0000000139121003, *pmd=0000000000000000
2018-05-22T20:46:59.155357+00:00 WARNING kernel: [   18.845162] 
2018-05-22T20:46:59.155359+00:00 WARNING kernel: [   18.845170] CPU: 0 PID: 1611 Comm: chrome Not tainted 3.18.0-17607-gd9c046a08d0c #1
2018-05-22T20:46:59.155361+00:00 WARNING kernel: [   18.845175] Hardware name: Mediatek Elm rev2, rev1 board (DT)
2018-05-22T20:46:59.155363+00:00 INFO kernel: [   18.845179] chrome[1647]: unhandled level 2 translation fault (11) at 0x000effc6, esr 0x92000006
2018-05-22T20:46:59.155365+00:00 ALERT kernel: [   18.845182] pgd = ffffffc0b6c6d000
2018-05-22T20:46:59.155367+00:00 ALERT kernel: [   18.845189] [000effc6] *pgd=00000000edece003, *pud=00000000edece003, *pmd=0000000000000000
2018-05-22T20:46:59.155369+00:00 WARNING kernel: [   18.845191] task: ffffffc0fa2ebd40 ti: ffffffc0f9140000 task.ti: ffffffc0f9140000
2018-05-22T20:46:59.155374+00:00 WARNING kernel: [   18.845199] PC is at 0x1d727fc
2018-05-22T20:46:59.155395+00:00 WARNING kernel: [   18.845201] LR is at 0x1adec49
2018-05-22T20:46:59.155397+00:00 WARNING kernel: [   18.845205] pc : [<0000000001d727fc>] lr : [<0000000001adec49>] pstate: 60010030
2018-05-22T20:46:59.155399+00:00 WARNING kernel: [   18.845207] sp : 00000000ffa63998
2018-05-22T20:46:59.155401+00:00 WARNING kernel: [   18.845211] x12: 0000000001db5737 
2018-05-22T20:46:59.155403+00:00 WARNING kernel: [   18.845214] 
2018-05-22T20:46:59.155405+00:00 WARNING kernel: [   18.845218] CPU: 1 PID: 1647 Comm: chrome Not tainted 3.18.0-17607-gd9c046a08d0c #1
2018-05-22T20:46:59.155407+00:00 WARNING kernel: [   18.845221] Hardware name: Mediatek Elm rev2, rev1 board (DT)
2018-05-22T20:46:59.155409+00:00 WARNING kernel: [   18.845224] task: ffffffc0f97a0c40 ti: ffffffc0aa928000 task.ti: ffffffc0aa928000
2018-05-22T20:46:59.155411+00:00 WARNING kernel: [   18.845227] PC is at 0x1d727fc
2018-05-22T20:46:59.155413+00:00 WARNING kernel: [   18.845230] LR is at 0x1adec49
2018-05-22T20:46:59.155415+00:00 WARNING kernel: [   18.845233] pc : [<0000000001d727fc>] lr : [<0000000001adec49>] pstate: 60010030
2018-05-22T20:46:59.155417+00:00 WARNING kernel: [   18.845236] sp : 00000000ffa63998
2018-05-22T20:46:59.155419+00:00 WARNING kernel: [   18.845239] x12: 0000000001db5737 
2018-05-22T20:46:59.155421+00:00 WARNING kernel: [   18.845244] x11: 0000000008513000 x10: 0000000000002e48 
2018-05-22T20:46:59.155423+00:00 WARNING kernel: [   18.845247] x11: 0000000008506000 x10: 0000000000002e48 
2018-05-22T20:46:59.155425+00:00 WARNING kernel: [   18.845251] x9 : 0000000000000000 x8 : 0000000008515620 
2018-05-22T20:46:59.155427+00:00 WARNING kernel: [   18.845257] x7 : 00000000000f0002 x6 : 0000000000000000 
2018-05-22T20:46:59.155429+00:00 WARNING kernel: [   18.845262] x5 : 0000000000000000 x4 : 0000000000000001 
2018-05-22T20:46:59.155431+00:00 WARNING kernel: [   18.845267] x3 : 0000000000000065 x2 : 0000000006c4c64c 
2018-05-22T20:46:59.155432+00:00 WARNING kernel: [   18.845272] x1 : 0000000000000000 x0 : 0000000000000000 
2018-05-22T20:46:59.155434+00:00 WARNING kernel: [   18.845277] 
2018-05-22T20:46:59.155436+00:00 WARNING kernel: [   18.845289] 
2018-05-22T20:46:59.155438+00:00 WARNING kernel: [   18.845294] x9 : 0000000000000000 x8 : 0000000008508620 
2018-05-22T20:46:59.155440+00:00 WARNING kernel: [   18.845302] x7 : 00000000000f0002 x6 : 0000000000000000 
2018-05-22T20:46:59.155442+00:00 WARNING kernel: [   18.845310] x5 : 0000000000000000 x4 : 0000000000000001 
2018-05-22T20:46:59.155444+00:00 WARNING kernel: [   18.845318] x3 : 000000000000003f x2 : 0000000006c4c64c 
2018-05-22T20:46:59.155446+00:00 WARNING kernel: [   18.845327] x1 : 0000000000000000 x0 : 0000000000000000 
2018-05-22T20:46:59.155448+00:00 WARNING kernel: [   18.845334] 

Deleted: messages 348 KB

messages 348 KB View Download

Sorry, didn't finish the sentence on my previous comment: On Chrome OS version 10694, both chrome version 68.0.3433.0 and latest chrome 68.0.3437.0 have crashes on login screen. 

This is the Chrome OS CLs between 10691 and 10694:
https://crosland.corp.google.com/log/10691.0.0..10694.0.0

yunlian@, are you a good owner of this issue? (maybe do a bisect on the Chrome OS CLs)?

It might be the problem from Chrome Source. Can we bisect the chrome code instead? (I am not a good owner for this one)

I don't think it's from Chrome, as on Chrome OS version 10691 (last known good Chrome OS version for elm), I flashed Chrome 68.0.3433.0 and 68.0.3437.0 (latest Chrome), everything works fine.

Hmm, this smells like a possible compiler/linker error (Chrome starts mysteriously crashing on different Chrome OS versions with very few diffs), so adding compiler team folks.  Elm is a bit unique since it is aarch64 on kernel-3.18.

Any chance there was a toolchain update recently?  Sometimes such updates cause issues that take a while to manifest due to old bin packages.

Can you try on hana, too?

The research in #20 pretty solidly convinces me that something changed on the Chrome OS side between 10691 and 10694 that causes the breakage.  The best path forward here is to figure out what and revert it.

Re#25: I don't think haha has the same error (though from GoldenEye its build has failed for a long time, probably more than 2 weeks). From the GoldenEye for build 10694, only elm and peach-pit has the same error: https://screenshot.googleplex.com/trD0X52dyi4. See the detailed error message: https://screenshot.googleplex.com/u3fgMU2Gm9b

On toolchain side, we cherry picked several upstream patches from llvm. The CL landed in 10690.0.0, and it should take effect one day after. So the timing matches, and I can verify on that part.

+this week's infra deputy, sheriff and gardener in case this error has affected some builders 

xdai, how do you know --enable-service-binary-launcher is being used here? --enable-service-binary-launcher is only for tests. There is no flag that turns the switch on. I looked at a handful of the crashes linked from #13 and didn't see the switch.

Re#30: sky@, Sorry I think I made a mistake. From code here, https://cs.chromium.org/chromium/src/content/browser/service_manager/service_manager_context.cc?rcl=450d4a7d8c0437fad9af4e591b1190b9103e40c5&l=348, NullServiceProcessLauncherFactory is only used when the switch is *NOT* specified. I thought it's the opposite. If it's the expected behavior, maybe we should consider to remove the error message?

From comment#20 and #21, it seems the problem is in Chrome OS. Chrome should be good. We're waiting for yunlian@ to verify if it's toolchain's problem, see comment#28.

It seems that the Chrome update between 10691 and 10694 makes thinlto unhappy. If I disable thinlto on elm, the chrome crash mentioned in #20 is gone.

I will try to see if I can find a quick workaround for that. If not, we need to disable thinlto on these two boards.

Issue 845822 has been merged into this issue.

Is this still happening on the latest R68 builds?

Issue still reproducible on latest M68 builds. Tested on elm with build 10718.4.0, 68.0.3440.4

I bisected the Chrome change list and found that the CL below in chrome makes chrome start without crash on kevin with thinlto. Can we cherry pick this CL to R68 branch?

commit 1d888f990fb31b46ea93b42ad49f645f5e627819
Author: v8-autoroll <v8-autoroll@chromium.org>
Date:   Sat May 26 07:34:10 2018 +0000

    Update V8 to version 6.9.1.
    
    Summary of changes available at:
    https://chromium.googlesource.com/v8/v8/+log/44d7d7d6..6415e8af

yangguo@, is it possible to backport the v8 update to R68?

 Issue 845429  has been merged into this issue.

As discussed offline, fast forwarding to V8 6.9 on the M68 branch is not an option. V8 needs to stay in lock step with Chrome. If you can however bisect which particular V8 commit fixed the issue (by manually changing DEPS entry for V8), we could maybe merge that commit back to V8 6.8.

It looks like we need the CL below from v8 to fix the issue reported in #20.

commit 45fa14f061b6bd869201facc956e01b57430799e
Author: Camillo Bruni <cbruni@chromium.org>
Date:   Wed May 23 13:11:52 2018 +0000

    Revert "Flush ICache on startup deserialization after marking memory executable"
    
    This reverts commit 05bcb12e554ea86e6bb03aef9db3c2fe220296ed.
    
    Reason for revert: Causes isolate startup regressions ( https://crbug.com/845508 )
    

Camillo, can you weigh in here whether this revert is the actual fix? Shall we merge it to 6.8?

05bcb12e554ea86e6bb03aef9db3c2fe220296ed, which introduced the crash, shipped first with V8 6.8.239. The revert, 45fa14f061b6bd869201facc956e01b57430799e, shipped first with 6.8.284, but we cut the M68 branch at 6.8.275. So the fix really ended up in M69.

So we definitely need to back merge the revert.

Who can backport the change to v8 and change the hash in DEPS?
I am not familiar with this process.

We need to get merge approval first. Then I can do the merge and a release manager will bump the DEPS for the branch.

Your change meets the bar and is auto-approved for M68. Please go ahead and merge the CL to branch 3440 manually. Please contact milestone owner if you have questions.
Owners: cmasso@(Android), kariahda@(iOS), bhthompson@(ChromeOS), abdulsyed@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

The following revision refers to this bug:
  https://chromium.googlesource.com/v8/v8.git/+/1262d17586f35fb8991c9bfa90893af3d427a8df

commit 1262d17586f35fb8991c9bfa90893af3d427a8df
Author: Yang Guo <yangguo@chromium.org>
Date: Mon Jun 04 07:55:23 2018

Merged: Revert "Flush ICache on startup deserialization after marking memory executable"

Revision: 45fa14f061b6bd869201facc956e01b57430799e

BUG=chromium:842862, chromium:845508 , chromium:845877 , chromium:845219 
LOG=N
NOTRY=true
NOPRESUBMIT=true
NOTREECHECKS=true
TBR=cbruni@chromium.org

Change-Id: I4c140554b96038de6b34c572a6a6f307300bf557
Reviewed-on: https://chromium-review.googlesource.com/1084477
Reviewed-by: Yang Guo <yangguo@chromium.org>
Cr-Commit-Position: refs/branch-heads/6.8@{#9}
Cr-Branched-From: 44d7d7d6b1041b57644400a00cb3fee35f6c51b2-refs/heads/6.8.275@{#1}
Cr-Branched-From: 5754f66f75136dc17b4c63fec84f31dfdb89186e-refs/heads/master@{#53286}
[modify] https://crrev.com/1262d17586f35fb8991c9bfa90893af3d427a8df/src/isolate.cc
[modify] https://crrev.com/1262d17586f35fb8991c9bfa90893af3d427a8df/src/snapshot/startup-deserializer.cc
[modify] https://crrev.com/1262d17586f35fb8991c9bfa90893af3d427a8df/src/snapshot/startup-deserializer.h

This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible!

If all merges have been completed, please remove any remaining Merge-Approved labels from this issue.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible!

If all merges have been completed, please remove any remaining Merge-Approved labels from this issue.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot