Invalid file handle crash:
http://www.chromium.org/developers/crash-reports/crash-with-invalid-handle

Does this happen every time you upload a large file?

Not everytime, sometimes the file upload goes through. When I tried to upload more than one file at a time from NAS server to Dropbox, Chrome would crash almost all the time.

Thank you for providing more feedback. Adding requester "msrchandra@chromium.org" for another review and adding "Needs-Review" label for tracking.

For more details visit https://sites.google.com/a/chromium.org/dev/issue-tracking/autotriage - Your friendly Sheriffbot

Interesting. Do you always upload from a NAS?

The crash is occurring because the FD is not being closed safely. Networked file systems could be a possible culprit in that.

Thread 7 CRASHED [EXC_BREAKPOINT / 0x00000002 @ 0x0000000104d76be1 ] MAGIC SIGNATURE THREAD
0x0000000104d76be1	(Google Chrome Framework -debugger_posix.cc:257 )	base::debug::BreakDebugger()
0x0000000104d80f53	(Google Chrome Framework -scoped_generic.h:149 )	base::File::Close()
0x000000010510c518	(Google Chrome Framework -file_stream_context.cc:179 )	net::FileStream::Context::CloseFileImpl()
0x0000000104d77399	(Google Chrome Framework -callback.h:394 )	base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask const&)
0x0000000104d98de8	(Google Chrome Framework -message_loop.cc:486 )	base::MessageLoop::RunTask(base::PendingTask const&)
0x0000000104d99327	(Google Chrome Framework -message_loop.cc:495 )	base::MessageLoop::DoWork()
0x0000000104d6c290	(Google Chrome Framework -message_pump_libevent.cc:229 )	base::MessagePumpLibevent::Run(base::MessagePump::Delegate*)
0x0000000104daba42	(Google Chrome Framework -run_loop.cc:56 )	base::RunLoop::Run()
0x0000000104d9858c	(Google Chrome Framework -message_loop.cc:293 )	base::MessageLoop::Run()
0x0000000107e0e8e3	(Google Chrome Framework -browser_thread_impl.cc:188 )	content::BrowserThreadImpl::FileThreadRun(base::MessageLoop*)
0x0000000107e0ea22	(Google Chrome Framework -browser_thread_impl.cc:242 )	content::BrowserThreadImpl::Run(base::MessageLoop*)
0x0000000104dceb67	(Google Chrome Framework -thread.cc:252 )	base::Thread::ThreadMain()
0x0000000104dcaeab	(Google Chrome Framework -platform_thread_posix.cc:67 )	base::(anonymous namespace)::ThreadFunc(void*)
0x00007fff8db90898	(libsystem_pthread.dylib + 0x00001898 )	
0x00007fff8db90729	(libsystem_pthread.dylib + 0x00001729 )	
0x00007fff8db94fc8	(libsystem_pthread.dylib + 0x00005fc8 )	
0x0000000104dcae4f	(Google Chrome Framework + 0x005d7e4f )	

Yes, most of our uploads are via Synology NAS networked drive.

Thank you for providing more feedback. Adding requester "msrchandra@chromium.org" for another review and adding "Needs-Review" label for tracking.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Thanks. I think this is probably related to uploading large files from a network drive. I'm not sure why the file handle would be invalid, though.

What network protocol is the fileshare on (SMB, NFS, etc)?

Do the crash dumps have enough to extract errno from them? That might give us something handy. I know we've, in the past, had problems with Apple code double-closing fds.

No, unfortunately. The heavy inlining of ScopedFD has made that difficult, and because errno is in TLS rather than on the stack.

close() has limited errno values, though. I'd guess it would either be EBADF or EINTR, but knowing which of those it is would be good.

MDRawContextAMD64
  p1_home       = 0x0
  p2_home       = 0x0
  p3_home       = 0x0
  p4_home       = 0x0
  p5_home       = 0x0
  p6_home       = 0x0
  context_flags = 0x10001f
  mx_csr        = 0x1fa1
  cs            = 0x2b
  ds            = 0x0
  es            = 0x0
  fs            = 0x0
  gs            = 0x0
  ss            = 0x0
  eflags        = 0x206
  dr0           = 0x0
  dr1           = 0x0
  dr2           = 0x0
  dr3           = 0x0
  dr6           = 0x0
  dr7           = 0x0
  rax           = 0x113532050
  rcx           = 0x7fff961ebf0a
  rdx           = 0xffffffffffffffff
  rbx           = 0x610002a7a040
  rsp           = 0x113531a40
  rbp           = 0x113531a40
  rsi           = 0x113531af8
  rdi           = 0x10b3b1cec
  r8            = 0x1
  r9            = 0x113531c00
  r10           = 0x10510c510
  r11           = 0x203
  r12           = 0x7fa89350ab10
  r13           = 0x113531ca0
  r14           = 0x10945629f
  r15           = 0x10b3b3589
  rip           = 0x104d76be1

Stack
c819531301000000  ; 00000001135319c8
011a531301000000  ; 0000000113531a01
701f110080600000  ; 0000608000111f70
0100000000000000  ; 0x1
801f110080600000  ; 0000608000111f80
0000000000000000  ; 0x0
b019531301000000  ; 00000001135319b0
f338b98dff7f0000  ; 00007fff8db938f3
b019531301000000  ; 00000001135319b0
7f71d904
60200000
003c00dd
003c00dd
011a531301000000  ; 0000000113531a01
801f110080600000  ; 0000608000111f80
c819531301000000  ; 00000001135319c8
401a531301000000  ; 0000000113531a40
cc70d90401000000  ; 0000000104d970cc
e019531301000000  ; 00000001135319e0
1028dd0401000000  ; 0000000104dd2810
f019531301000000  ; 00000001135319f0
1028dd0401000000  ; 0000000104dd2810
0300000000000000  ; 0x3
0100000000000000  ; 0x1
101a531301000000  ; 0000000113531a10
2ab7da0401000000  ; 0000000104dab72a
501a531301000000  ; 0000000113531a50
b8ba2d983c000000  ; 0000003c982dbab8
301a531301000000  ; 0000000113531a30
f7fbdc0401000000  ; 0000000104dcfbf7
0100000000000000  ; 0x1
0000000000000000
401a531301000000  ; 0000000113531a40
e06bd70401000000  ; 0000000104d76be0  (#0 rip) base::debug::BreakDebugger()
701a531301000000  ; 0000000113531a70
540fd80401000000  ; 0x0000000104d80f54 (#1 rip) base::File::Close()
9f62450901000000
0000000000000000
801a531301000000
101b531301000000
801a531301000000
19c5100501000000  ; 0x000000010510c519 (#2 rip) net::FileStream::Context::CloseFileImpl()

Also this crash report is from 10.9.5. I wonder if a newer OS has the same problem.

The fileshare network protocol is AFP and SMB is also activated by I think all machines connect via AFP

There is at least one known double-close bug in OS X that was only fixed in 10.11.1 or later. But there are crashes here on versions newer than that, so there's probably more or these, be it our code or Apple's.

man close on my Mac says close can fail with EBADF, EINTR, or EIO. EBADF is pretty likely, if we or Apple have another double-close bug. EINTR should be ignored by IGNORE_EINTR. EIO is intriguing, especially since networked filesystems are involved. The man page says it only happens if "A previously-uncommitted write(2) encountered an input/output error" which I don't think would happen, but maybe something else is going on.

I'm thinking we should add a base::debug::Alias on ScopedFDCloseTraits::Free, just to confirm this is indeed EBADF.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3b6205579e92ceb47a07040a1c79d87923de7512

commit 3b6205579e92ceb47a07040a1c79d87923de7512
Author: davidben <davidben@chromium.org>
Date: Wed Jun 22 22:06:48 2016

Include close errnos in crash dumps.

We may be crashing here because close is failing with EBADF (signalling other
code having problems with fds) or because networked filesystems are buggy and
failing with some other errno. Alias a copy of errno on the stack so we may
distinguish the two.

BUG= 603354 

Review-Url: https://codereview.chromium.org/2090703002
Cr-Commit-Position: refs/heads/master@{#401427}

[modify] https://crrev.com/3b6205579e92ceb47a07040a1c79d87923de7512/base/files/scoped_file.cc

Next net bug triager (or perhaps the one after that): see if we've got any crash reports from a canary release after the revision above. Once we've figured out what the errno is, we can revert that change.

My thinking is:

If it's EBADF - Okay, we need to go hunt for more fd double-closers. That will be fun. The last time this happened, someone played interesting games with fd guards. This might be worth repeating.

If it's something else - AFP or SMB on Mac is dumb. Probably that would mean we need to loosen that check and only crash on EBADF, letting other errors silently go through?

(I'll make a note to check myself in a week.)

Note: did not see any new crashers here (note the version your patch went out is 53.0.2777.0).

dean: Sorry we let this sit for a while. Would you mind trying to reproduce this on Chrome Canary (https://www.google.com/chrome/browser/canary.html)? It sounds like you had a fairly reproducible case? Crash reports from that version should have slightly more information and help us determine why close() is failing. Thanks!

Note to net triager: if we time out on a response, before closing this bug, check if we got crash reports after 2777.0 on our own. If so, we should have the needed information anyway.

Hi David,

In reference to  Issue 603354 , I downloaded and installed the Canary browser as requested to see if I could replicate the issue but when I first tried to open the Canary browser on my iMac, the application crashed! See crash report attached. On my second attempt the application opened.

I will see if I can replicate the error and let you know.

Cheers




Dean Baker 
Web Developer & Technical Advisor

  

71-73 Lithgow Street
St Leonards NSW 2065
Phone: 61-2-8436 9957
Fax: 61-2-8436 9933
www.bpproductions.com.au

This e-mail may contain confidential information. If you are not the intended recipient, please notify the sender immediately and delete it from your system and do not disclose or use the email's content. Any opinions expressed in this email may not represent those of the Australian Dental Association (NSW Branch) Ltd (ADA). The ADA does not guarantee that e-mail transmission is secure or error or virus free and the ADA accepts no liability arising out of the transmission or receipt of this email. Personal information received by the ADA is handled in accordance with our privacy policy.


On 30 June 2016 at 6:53:48 am, david...@chromium.org via Monorail (monorail@chromium.org) wrote:

Thank you for providing more feedback. Adding requester "csharrison@chromium.org" for another review and adding "Needs-Review" label for tracking.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Overriding sheriff-bot. Thanks for trying (and trying again). I don't think the attachment went through. Can you post the crash ID found in chrome://crashes?

The new crash id is 90e8f2d600000000 as given over email.

Routing to davidben@ who added the stack aliasing and unmarking Needs-Feedback so the current net triager can look at this.

Ok, so I'm on bug triage.  Looking at the source of the crash report for 90e8f2d600000000, it seems like the local variable close_errno would contain the value of interest, but I don't know how to get at that.  Probably minidump, but I must confess I do not know how to do it, especially for an OSX minidump.

This is a terrible answer, but I usually build minidump_dump and look at the hexdump of the stack directly. The symbolizer will give you the addresses to look for around it and probably not much else will be on the stack, though be aware they're little-endian and off by an instruction.

I can take a look when I'm back from IETF, though that won't be until the week after next.

rsesek: Do you know a better way to get the value out of the stack? (Or cycles to poke at that crash, since I won't be able to get to it for a while?)

No, manually recovering data from the stack memory dump is what I did in #14. It's the only way to do that on Mac at the moment.

Okay, we have got to make better tools here. Everything about this manual process is a pain, from dealing with little-endian to go/crash correcting for things being one instruction off to minidump_dump not blocking the stack dump by words.

I went through every crash we've gotten on this stack trace since it went in. The results are:

01a94e5600000000: EBADF
022ed39600000000: EBADF
084b4af100000000: EBADF
0ba056d600000000: EBADF
1f5dcd8e00000000: EIO
41b8d19600000000: EBADF
4271ac3600000000: EBADF
4f6e9bfc00000000: ETIMEDOUT
838d592600000000: EBADF
889b1f3600000000: EBADF
90e8f2d600000000: EBADF
95d527f100000000: EBADF
995b5fd600000000: EBADF
a97796d600000000: EBADF
ae631e8200000000: EBADF
c80da17c00000000: EBADF
e048197c00000000: EBADF
e688eb0200000000: EBADF
fcb0ae1600000000: EBADF

So, there's two non EBADF errors, which might be funny networked filesystems. rsesek, do you know much of OS X's behavior there? EIO is cited in the man page, while ETIMEDOUT isn't. Should we silently ignore those errno values?

That said, the majority is EBADF, so someone's double-closing fds. Again. Previously, +sergeyu did something with close guards. I'm thinking we should do that again and see if we can find the culprit.
https://codereview.chromium.org/1053873003

Most of the crashes are happening in disk_cache::SimpleSynchronousEntry, so perhaps we should guard those fds. (+gavinp)

No, I'm not too familiar with macOS behavior here. But for a networked filesystem, ETIMEDOUT does seem like it's possible (from some casual searching in xnu/bsd/nfs).

Adding guards is probably a good next debugging step. The good news is that now that we support 10.9+ we shouldn't have to load the symbols dynamically.

In the ETIMEDOUT case, do you know if the fd would be closed? I know we had to do an IGNORE_EINTR since it wasn't safe to assume the fd was still valid.

I'll see how much of a nuisance it is to add a guard to the cache code.

I would not attempt to re-close an FD if it failed with ETIMEDOUT, because it's not documented as a return errno, so it's not guaranteed to be valid or invalid.

Issue 640472 has been merged into this issue.

I can't get access to the original crash report (?), and the crash report from 640472 isn't network related.  Why is this in Internals>Network?

The fd that blew up was within the net stack. Of course, since it's an fd double-close, it could be anywhere. We'll need to play the fd guard game to find out where it actually is.

It strikes me that this issue has been investigated to the point where we understand what we need to do in order to actually figure out what's going on, if not to the point where we actually understand what's going on, and the question is whether or not someone's actually going to be pu in the time.  Given that, is it reasonable to at least transition this issue to "Available" to get it off the triager's plate?

[+mmenke@ for triage process question.]

Yea, I think it's reasonable to change the label.

Users experienced this crash on the following builds:

Android Dev 54.0.2840.6 -  1.32 CPM, 4 reports, 3 clients (signature base::internal::ScopedFDCloseTraits::Free)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Users experienced this crash on the following builds:

Mac Canary 55.0.2851.0 -  0.67 CPM, 1 reports, 1 clients (signature base::internal::ScopedFDCloseTraits::Free)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Users experienced this crash on the following builds:

Mac Canary 56.0.2895.0 -  1.04 CPM, 1 reports, 1 clients (signature base::internal::ScopedFDCloseTraits::Free)
Android Dev 55.0.2883.9 -  0.24 CPM, 13 reports, 13 clients (signature base::internal::ScopedFDCloseTraits::Free)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

This issue has been Available for over a year. If it's no longer important or seems unlikely to be fixed, please consider closing it out. If it is important, please re-triage the issue.

Sorry for the inconvenience if the bug really should have been left as Available. If you change it back, also remove the "Hotlist-Recharge-Cold" label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Is this still occurring? We fixed some issues on macOS a while ago where this crash would trigger on network filesystems and the like.

Hello,

I have not had a repeat issue on Chrome.

Thanks
Dean



Dean Baker


*Web Developer & Technical Advisor *

*Best Practice Productions*

*A Division of the Australian Dental Association NSW Branch*

1 / 1 Atchison Street
St Leonards NSW 2065
Phone: 61-2-8436 9957
Fax: 61-2-8436 9933
www.bpproductions.com.au

This e-mail may contain confidential information. If you are not the
intended recipient, please notify the sender immediately and delete it from
your system and do not disclose or use the email's content. Any opinions
expressed in this email may not represent those of the Australian Dental
Association (NSW Branch) Ltd (ADA). The ADA does not guarantee that e-mail
transmission is secure or error or virus free and the ADA accepts no
liability arising out of the transmission or receipt of this email.
Personal information received by the ADA is handled in accordance with our
privacy policy.

On 31 October 2017 at 2:21:19 am, david… via monorail (
monorail+v2.294852074@chromium.org) wrote:

Thanks! I'm going to assume it was fixed by the macOS change then.