Also reproducible with v4.13.8 kernel. Looks like users with CONFIG_USER_NS enabled in kernel config are affected, but having it on 2 machines, one crashes, one doesn't (same versions apply).

Which linux distro are you using?  What is your glibc version?

Arch Linux here, glibc is 2.26-5.

pfactum, do I understand correctly that this is crashing both with CONFIG_USER_NS enabled and disabled?

For me it is crashing with CONFIG_USER_NS, but I haven't tried a kernel without it.

Though, Arch Linux had bug reported, but it got closed since ppl couldn't reproduce it without USER_NS.

Ah that makes more sense. I was expecting this to not crash on a machine without CONFIG_USER_NS, so your observations match my expectations.

Looks like our glibc TID caching tricks are failing, don't you think Thomas?

CONFIG_USER_NS seems to be the key. Without it enabled, things work fine. With it enabled, things crash. (Makes sense, since toggling the --disable-namespace-sandbox flag is basically testing the same thing, right?)

And I'm on Arch Linux with glibc 2.26 as well.

c#7 Yes it sounds like it.  Maybe the tid offset changed in glibc2.26?

You're running x86_64, right?

How did you build glibc?  Any patches or special options?

Right, x86_64.

Here is Arch glibc: https://git.archlinux.org/svntogit/packages.git/tree/trunk?h=packages/glibc

Adding 'RB-Stable' for next M62 re-spin. Please feel free to update the priority if required.

I can verify the same Issue for FrugalWare Linux. 

I noticed it only happens on the box using radeon GPU driver, 3 other using Intel don't seem to be affected.

CONFIG_USER_NS=y

glibc-2.25

Reverting 

https://chromium.googlesource.com/chromium/src/+/24df419517bb8ca8ac2b75eb752bd9a133267108

using the attached patch indeed fixes the issue.

Deleted: disable-tid-cache.patch 2.7 KB

disable-tid-cache.patch 2.7 KB Download

I tested with arch Linux, glibc 2.26-5 and was unable to repro.  For affected users, could you try building and running the attached program and attach the output?

Deleted: main.c 912 bytes

main.c 912 bytes View Download

@thomasanderson To repro on Arch you need a kernel with CONFIG_USER_NS=y, which the default Arch kernel doesn't have (the Arch maintainers think it's too risky). You could try my kernel if you want:

https://www.uplinklabs.net/repo/ec2/x86_64/linux-ec2-4.13.7-1-x86_64.pkg.tar.gz
https://www.uplinklabs.net/repo/ec2/x86_64/linux-ec2-4.13.7-1-x86_64.pkg.tar.gz.sig

I'll have to test the patch this evening when I'm home from work, unless someone else gets to it first.

I already built a kernel with CONFIG_USER_NS=y, and in chrome://sandbox, I see that the user namespace sandbox is enabled.

Oh, hm. Not sure what else the difference could be then. My kconfig is kind of special.

https://git.uplinklabs.net/steven/projects/archlinux/ec2/ec2-packages.git/tree/linux-ec2/config.x86_64

Here is main.c output from the machine where Chromium fails: https://gist.github.com/d9dec7c7934f911b08e8f8c8714f5625

Output from main.c is here: http://paste.opensuse.org/88098045

Thanks c#19 and c#20.  I believe the issue is this:

pid_t GetGlibcCachedTid() {
  pid_t tid;
  pthread_mutex_t lock;
  CHECK_EQ(0, pthread_mutex_init(&lock, nullptr));
  CHECK_EQ(0, pthread_mutex_lock(&lock));
  tid = lock.__data.__owner;
  CHECK_EQ(0, pthread_mutex_unlock(&lock));
  CHECK_EQ(0, pthread_mutex_destroy(&lock));
  return tid;
}

That's supposed to get the tid from glibc, but it's returning 0.  There must be something different about our libc's (maybe the pthread_mutex_t layout is different?).  Could you share your libc-2.xx.so?  This might be in eg. /lib/x86_64-linux-gnu/libc-2.26.so or /usr/lib/libc-2.26.so

Attached.

You can also find Arch's glibc here: http://mirror.rackspace.com/archlinux/core/os/x86_64/glibc-2.26-5-x86_64.pkg.tar.xz

Deleted: libc-2.26.so 2.0 MB

libc-2.26.so 2.0 MB Download

I have the same failure in opensuser 42.3, kernel 4.4.90-28-default and ldd (GNU libc) 2.22

+1 openSuSE 42.3, kernel 4.4.90-28-default, and glibc 2.22.

+Chris for visibility into sandbox maintainability.

Well, I have no idea what's causing this problem.  Tested with OpenSUSE 42.3, kernel 4.4.90-28-default, glibc 2.22, google-chrome-stable 62.0.3202.62.  Still launches fine with the namespace sandbox.

Also, the libc from c#22 had the same sha1sum as my own, so that couldn't be it either.

sandbox folks: For some reason, GetGlibcCachedTid() is returning 0.  In this case, could we skip the CHECK_EQ's in MaybeUpdateGlibcTidCache(), which would prevent the crash?

I'd be curious to see if initializing the lock with PTHREAD_RECURSIVE_MUTEX_INITIALIZER_NP and getting rid of the pthread_mutex_init call changes anything. Haven't read through the pthread mutex code carefully, but there do appear to be types that don't populate the owner.

That's worth trying.  Could anyone try a chromium build with the attached patch?

Deleted: speculative_patch.diff 616 bytes

speculative_patch.diff 616 bytes Download

I dont know if this could help by anyway I executed c#15 code in my system

Deleted: out.txt 17.1 KB

out.txt 17.1 KB View Download

@thomasanderson I built chromium with 'speculative_patch.diff', and it runs fine with or without the --disable-namespace-sandbox flag.

c#31: awesome!  Thanks to everyone who helped to diagnose the issue!

In that case, we'll go ahead and move forward with the fix.

vamshi.kommuri@techmahindra.com: Not sure, why you added me to Cc in #23 here.

I think 776356 should be merged here or closed.

 Issue 776356  has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e2fd7987b30fce6ba48c133b5503b7df9e1f888b

commit e2fd7987b30fce6ba48c133b5503b7df9e1f888b
Author: Tom Anderson <thomasanderson@chromium.org>
Date: Mon Oct 23 18:24:19 2017

Linux sandbox: Fix GetGlibcCachedTid returning 0 on some systems

BUG= 776359 
R=jorgelo@chromium.org

Change-Id: Ic792ffc039ce32e8199b6dc93ff6e5e6dac92fab
Reviewed-on: https://chromium-review.googlesource.com/732483
Reviewed-by: Jorge Lucangeli Obes <jorgelo@chromium.org>
Commit-Queue: Thomas Anderson <thomasanderson@chromium.org>
Cr-Commit-Position: refs/heads/master@{#510855}
[modify] https://crrev.com/e2fd7987b30fce6ba48c133b5503b7df9e1f888b/sandbox/linux/services/namespace_sandbox.cc

Requesting merge to 62 and 63.  The CL is small and should be relatively safe.

This bug requires manual review: Request affecting a post-stable build
Please contact the milestone owner if you have questions.
Owners: amineer@(Android), cmasso@(iOS), bhthompson@(ChromeOS), abdulsyed@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Approving this merge for M63. 

For M62, my recommendation is to wait until we can test this in one of the lower channels (beta or dev). How safe is this merge overall? Seems like a fairly small one line change. 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/db3f39cad73d06c4f5b0691302d4a3cc981ababa

commit db3f39cad73d06c4f5b0691302d4a3cc981ababa
Author: Tom Anderson <thomasanderson@chromium.org>
Date: Mon Oct 23 23:50:57 2017

[Merge to M63] Linux sandbox: Fix GetGlibcCachedTid returning 0 on some systems

> BUG= 776359 
> R=jorgelo@chromium.org
>
> Change-Id: Ic792ffc039ce32e8199b6dc93ff6e5e6dac92fab
> Reviewed-on: https://chromium-review.googlesource.com/732483
> Reviewed-by: Jorge Lucangeli Obes <jorgelo@chromium.org>
> Commit-Queue: Thomas Anderson <thomasanderson@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#510855}

BUG= 776359 
TBR=jorgelo@chromium.org
NOTRY=true
NOPRESUBMIT=true
NOTREECHECKS=true

Change-Id: Id534d32870368e7dd767516eebaf8326e76f545a
Reviewed-on: https://chromium-review.googlesource.com/734266
Reviewed-by: Thomas Anderson <thomasanderson@chromium.org>
Cr-Commit-Position: refs/branch-heads/3202@{#730}
Cr-Branched-From: fa6a5d87adff761bc16afc5498c3f5944c1daa68-refs/heads/master@{#499098}
[modify] https://crrev.com/db3f39cad73d06c4f5b0691302d4a3cc981ababa/sandbox/linux/services/namespace_sandbox.cc

CL listed at #41 got merged to M62 branch 3202 instead of M63 branch 3239. Per comment #40, merge approval is for M63 not for M62. abdulsyed@/thomasanderson@, pls double check. Thank you.

Oops, my bad.  I'll revert on the M62 branch and correctly merge to M63.  Thanks for noticing!

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/45404a51d39bd864027f52ca9f2938b7be78bf88

commit 45404a51d39bd864027f52ca9f2938b7be78bf88
Author: Tom Anderson <thomasanderson@chromium.org>
Date: Tue Oct 24 06:16:25 2017

Revert "[Merge to M63] Linux sandbox: Fix GetGlibcCachedTid returning 0 on some systems"

Reason for revert: Meant to merge to M63, but merged to M62.

> > BUG= 776359 
> > R=jorgelo@chromium.org
> >
> > Change-Id: Ic792ffc039ce32e8199b6dc93ff6e5e6dac92fab
> > Reviewed-on: https://chromium-review.googlesource.com/732483
> > Reviewed-by: Jorge Lucangeli Obes <jorgelo@chromium.org>
> > Commit-Queue: Thomas Anderson <thomasanderson@chromium.org>
> > Cr-Commit-Position: refs/heads/master@{#510855}
>
> BUG= 776359 
> TBR=jorgelo@chromium.org
> NOTRY=true
> NOPRESUBMIT=true
> NOTREECHECKS=true
>
> Change-Id: Id534d32870368e7dd767516eebaf8326e76f545a
> Reviewed-on: https://chromium-review.googlesource.com/734266
> Reviewed-by: Thomas Anderson <thomasanderson@chromium.org>
> Cr-Commit-Position: refs/branch-heads/3202@{#730}
> Cr-Branched-From: fa6a5d87adff761bc16afc5498c3f5944c1daa68-refs/heads/master@{#499098}

BUG= 776359 
TBR=jorgelo@chromium.org
NOTRY=true
NOPRESUBMIT=true
NOTREECHECKS=true

Change-Id: Ibcfea3d8cc5280677ce18ad6e6d30bcabffc5ce0
Reviewed-on: https://chromium-review.googlesource.com/735211
Reviewed-by: Thomas Anderson <thomasanderson@chromium.org>
Cr-Commit-Position: refs/branch-heads/3202@{#732}
Cr-Branched-From: fa6a5d87adff761bc16afc5498c3f5944c1daa68-refs/heads/master@{#499098}
[modify] https://crrev.com/45404a51d39bd864027f52ca9f2938b7be78bf88/sandbox/linux/services/namespace_sandbox.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5ec82ba92b487342ec92fb91873758ffdf76c118

commit 5ec82ba92b487342ec92fb91873758ffdf76c118
Author: Tom Anderson <thomasanderson@chromium.org>
Date: Tue Oct 24 06:19:57 2017

[Merge to M63] Linux sandbox: Fix GetGlibcCachedTid returning 0 on some systems

> BUG= 776359 
> R=jorgelo@chromium.org
>
> Change-Id: Ic792ffc039ce32e8199b6dc93ff6e5e6dac92fab
> Reviewed-on: https://chromium-review.googlesource.com/732483
> Reviewed-by: Jorge Lucangeli Obes <jorgelo@chromium.org>
> Commit-Queue: Thomas Anderson <thomasanderson@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#510855}

BUG= 776359 
TBR=jorgelo@chromium.org
NOTRY=true
NOPRESUBMIT=true
NOTREECHECKS=true

Change-Id: Ic792ffc039ce32e8199b6dc93ff6e5e6dac92fab
Reviewed-on: https://chromium-review.googlesource.com/732483
Reviewed-by: Jorge Lucangeli Obes <jorgelo@chromium.org>
Commit-Queue: Thomas Anderson <thomasanderson@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#510855}
Reviewed-on: https://chromium-review.googlesource.com/735212
Reviewed-by: Thomas Anderson <thomasanderson@chromium.org>
Cr-Commit-Position: refs/branch-heads/3239@{#172}
Cr-Branched-From: adb61db19020ed8ecee5e91b1a0ea4c924ae2988-refs/heads/master@{#508578}
[modify] https://crrev.com/5ec82ba92b487342ec92fb91873758ffdf76c118/sandbox/linux/services/namespace_sandbox.cc

Could some one please help us how to launch chrome without --disable-namespace-sandbox flag or configure "CONFIG_USER_NS" so that fix can be verified.

Thanks.!

c#47:  We were never able to reproduce the issue ourselves.  Fortunately, we had some very dedicated users that were willing to test out the patch for us :)

Since this will just have landed in Dev today, my recommendation is to wait until it has baked a few days in Dev. And if all looks good, we will consider this for a future M62 respin. We will not include this in our scheduled release tomorrow due to not enough baking time. 

FYI, just tested a dev snapshot and it seems to be working (as expected):

Chromium	64.0.3250.0 (Developer Build) (64-bit)
Revision	799b88bd8e0e99c06ae59d81e7b53524a63beafb-refs/heads/master@{#511487}

Sandbox Status
SUID Sandbox	No
Namespace Sandbox	Yes
PID namespaces	Yes
Network namespaces	Yes
Seccomp-BPF sandbox	Yes
Seccomp-BPF sandbox supports TSYNC	Yes
Yama LSM Enforcing	No

Thanks for the very fast response and turnaround on this, developers! Your efforts are greatly appreciated.

Let's mark as fixed given c#50. Thanks Ricky and Thomas!

 Issue 778410  has been merged into this issue.

 Issue 776240  has been merged into this issue.

 Issue 765599  has been merged into this issue.

abdulsyed@ could we get merge approval for M62?

Close this issue if you like.  I am running on Chromium or Developer's
version successfully.  Eventually, the fix will get to the stable version.
I just don't have time now to start a new issue with the developers and
work through to a solution..

Thanks,
pvz

Doyle Van Zandt
*Creative Support Systems, LLC*
225-324-7045 Mobile
225-275-7709 Preferred
225-366-7045
*dvanzandt@csssupports.com <dvanzandt@csssupports.com>*

Approving merge to M62. This has baked in Canary/Dev for over a week. We don't have an M62 Respin planned yet, but approving this merge to M62. Branch:3202

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e7c69232b1ab61d4121b69bb917d5e27da6f62a0

commit e7c69232b1ab61d4121b69bb917d5e27da6f62a0
Author: Tom Anderson <thomasanderson@chromium.org>
Date: Tue Oct 31 18:19:44 2017

[Merge to M63] Linux sandbox: Fix GetGlibcCachedTid returning 0 on some systems

> BUG= 776359 
> R=jorgelo@chromium.org
>
> Change-Id: Ic792ffc039ce32e8199b6dc93ff6e5e6dac92fab
> Reviewed-on: https://chromium-review.googlesource.com/732483
> Reviewed-by: Jorge Lucangeli Obes <jorgelo@chromium.org>
> Commit-Queue: Thomas Anderson <thomasanderson@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#510855}

BUG= 776359 
TBR=jorgelo@chromium.org
NOTRY=true
NOPRESUBMIT=true
NOTREECHECKS=true

Change-Id: I2f92944d20104f20011814e94cdd8043c70503c8
Reviewed-on: https://chromium-review.googlesource.com/747043
Reviewed-by: Thomas Anderson <thomasanderson@chromium.org>
Cr-Commit-Position: refs/branch-heads/3202@{#762}
Cr-Branched-From: fa6a5d87adff761bc16afc5498c3f5944c1daa68-refs/heads/master@{#499098}
[modify] https://crrev.com/e7c69232b1ab61d4121b69bb917d5e27da6f62a0/sandbox/linux/services/namespace_sandbox.cc

Had an offline chat with thomasanderson@, the fix has been thoroughly tested on trunk by various users (c#31) and it would be fine to go ahead with the merge to M62 Stable RC#62.0.3202.89 though we didn't find an exact repro from Dev & test teams.

Thank you!

*it would be fine to go ahead with the merge for the release of M62 Stable RC#62.0.3202.89.