Actually, this isn't always.

I rebooted Chrome completely and was able to reinstall Secure Shell. It took a LONG time during checking and then installed.

I opened an SSH connection and 3 crosh windows. After a few minutes, the crosh windows stopped receiving inputs. Opening a new Secure Shell window opens it blank.

The SSH connection continued to work fine, but I know that if I close it, I'll have to reboot my Chromebook to connect back to it.

Some more info, onTerminalReady never gets called because it''s relying on a callback setProfile, which in turn, relies on a callback from pref._readStorage which never gets sent. 

This may explain why the options page is mostly blank. I would wager the stored preferences are fine on install, but something pollutes it during the course of usage and then prevents nassh from working. I'm trying to track it down now. 

It seems natives.StartRequest(functionName, request.args, hasCallback, optArgs.forIOThread, optArgs.preserveNullInObjects) doesn't return

here's what gets passed: 

{"functionName":"storage.get","request.args":["sync",["/hterm/profiles/default/alt-gr-mode","/hterm/profiles/default/alt-backspace-is-meta-backspace","/hterm/profiles/default/alt-is-meta","/hterm/profiles/default/alt-sends-what","/hterm/profiles/default/audible-bell-sound","/hterm/profiles/default/desktop-notification-bell","/hterm/profiles/default/background-color","/hterm/profiles/default/background-image","/hterm/profiles/default/background-size","/hterm/profiles/default/background-position","/hterm/profiles/default/backspace-sends-backspace","/hterm/profiles/default/character-map-overrides","/hterm/profiles/default/close-on-exit","/hterm/profiles/default/cursor-blink","/hterm/profiles/default/cursor-blink-cycle","/hterm/profiles/default/cursor-color","/hterm/profiles/default/color-palette-overrides","/hterm/profiles/default/copy-on-select","/hterm/profiles/default/use-default-window-copy","/hterm/profiles/default/clear-selection-after-copy","/hterm/profiles/default/ctrl-plus-minus-zero-zoom","/hterm/profiles/default/ctrl-c-copy","/hterm/profiles/default/ctrl-v-paste","/hterm/profiles/default/east-asian-ambiguous-as-two-column","/hterm/profiles/default/enable-8-bit-control","/hterm/profiles/default/enable-bold","/hterm/profiles/default/enable-bold-as-bright","/hterm/profiles/default/enable-clipboard-notice","/hterm/profiles/default/enable-clipboard-write","/hterm/profiles/default/enable-dec12","/hterm/profiles/default/environment","/hterm/profiles/default/font-family","/hterm/profiles/default/font-size","/hterm/profiles/default/font-smoothing","/hterm/profiles/default/foreground-color","/hterm/profiles/default/home-keys-scroll","/hterm/profiles/default/max-string-sequence","/hterm/profiles/default/media-keys-are-fkeys","/hterm/profiles/default/meta-sends-escape","/hterm/profiles/default/mouse-paste-button","/hterm/profiles/default/page-keys-scroll","/hterm/profiles/default/pass-alt-number","/hterm/profiles/default/pass-ctrl-number","/hterm/profiles/default/pass-meta-number","/hterm/profiles/default/pass-meta-v","/hterm/profiles/default/receive-encoding","/hterm/profiles/default/scroll-on-keystroke","/hterm/profiles/default/scroll-on-output","/hterm/profiles/default/scrollbar-visible","/hterm/profiles/default/scroll-wheel-move-multiplier","/hterm/profiles/default/send-encoding","/hterm/profiles/default/shift-insert-paste","/hterm/profiles/default/user-css"]],"preserveNullInObjects":true}"

Apparently, it's not related to crosh, it's all of ChromeOS. Any app that uses syncing has issues.

Disabling Android apps keeps it stable. In top, storage has medium CPU usage and Chrome_FileThread has 99% CPU

Since you mention android apps, is this flip, R11 or pixel ?
does it work if you switch back to stable channel ?

#1 mentioned samus.

I don't have a problem with my minnie or samus, I wonder what might trigger this.

Can you file feedback report with alt-shift-i referencing this bug?

I send feedback just now.

With ARC enabled, Crosh locks up with Chrome_FileThread hitting 99% CPU.

Crosh freezes, but VT-2 shows it clearly when using top.

The screenshot is when the lock is released, which I believe shows it. If I disable ARC, it never locks up. This isn't just Crosh Shell, but installing the Web Store locks up with "Installing..."

Sorry, I should clarify, by "Arc enabled" I mean "Enabled Apps on your Chromebook" checked in Settings. The only reason I added a screenshot of Crosh is because I can't take one of VT2.

Still happening on v54 that just released on dev branch.

Chrome_FileThread, sdcard and upto 6 instances of mount-passthrough make my CPU run at 99%.

Still happening on v54.0.2830.0 dev (64-bit)

I just enabled ARC, signed into Play Store, opened crosh, ran 'git status' on a repository and it hung.

Chrome_FileThread, sdcard and multiple instances of mount-passthrough eating up my CPU and halting Secure Shell from even loading.

I waited about 2 minutes and it came back.

Just now I tried to close Play Store and it crashed my system (black screen, but recovered).

Disabled Play Store.

Now top is showing just Chrome_FileThread at 99% (no more sdcard and mount-passthrough)

I tried to open Secure Shell and I get a blank screen again. Chrome Web Store apps won't install apps. Files.app won't open. I'm guessing anything that depends on Chrome_FileThread is hung.

I see this is chromeos in dev mode, you could have done anything to make it not work.
I cannot reproduce things that you describe in my environment so far.

what "git status" is that? Where is that directory located?

Let me see if I can get something more easily reproducible after a powerwash.

I updated to Version 54.0.2837.0 dev (64-bit) just now.

It seems overall better, but it's not completely gone. I'm trying to see if I can get the thread to lock up, but I was able to reproduce the stall by doing the following:

* Open crosh window (don't have to enter shell)

* Disable Play Store
* Enable Play Store
* Start sign in process

** DO THIS DURING THE PLEASE WAIT SCREEN **

* Go back to crosh
* Start typing on the keyboard (you don't have to run ANY linux commands, just keep typing)
* Keep typing and you'll see Crosh completely halts as the Play Store is setting up.
* After Play Store finishes setting up, all your typed characters will eventually show up in Crosh

You can also test thing by just doing 

ping 127.0.0.1

and it'll hang just as well, and it's pretty easy to notice.

This is just an example of what's going on. I haven't had enough time to play this new update, but I'll comment again but this was happening to me "randomly" when Play Store apps were enabled. I wouldn't be able to release the "lock" until I rebooted the OS (full reboot, Alt+Shift+Vol- didn't fix it). At this point, sdcard and mount-passthrough still have somewhat high CPU usage, but it's not locking up Crosh.

+dspaid 
I was testing with 53.0.2785.75 samus just because I grabbed my hands on that build it but doesn't seem to be an issue for me.

I've not been able to reproduce with either minnie or cyan on R54-8737.1.0.  Will continue to investigate.

I was not able to reproduce with the Dev version of Samus either...

If you are still able to reproduce this could you provide a profiler snapshot?
Once the device is in this slow/high-CPU usage state access chrome://profiler and choose to Save the results in the upper left corner.

@dspaid Thanks, I will do. I'm going to Powerwash before I do. I don't really do crazy modifications to my Chromebook. The most I do is modify some code and then push to git. I don't even run crouton. Hopefully the profiler will help narrow it down.

It might be related how many files are on the FST (file system table). My personal ~Downloads folder has over 100,000 files. This might be why it's hard to reproduce. Anyway, I'll powerwash it either today or tomorrow and see if I can figure out what's going on with chrome://profiler

The large Downloads directory may explain this, in particular during initial sign-in.
I tested with 100k files and found that it did indeed cause slowdown.  This primarily seems to be caused by arc_downloads_watcher_service.cc, which sends all of the existing files off to android's MediaProvider for indexing.  If you have anything writing lots of files to Downloads those writes will have extra overhead to scan and index as well, which in the case of my script to create 100k files is enough to completely lock up crosh even after the writing is done as its still trying to process the backlog of files that have been updated.

If this is the case you should see the problem go away entirely if you do a powerwash (or just remove everything from your Downloads directory).

Dan, thanks for investigation.

arc_downloads_watcher_service.cc is not efficient with so many files like 100K; it enumerates all files under Downloads directory when any change is made to any file under the directory.

I can think of three things to improve the situation:

1. Run DownloadsWatcher on a dedicated thread rather than FILE thread. Since it is not avoidable for DownloadsWatcher to run expensive file operations in some cases (e.g. on first Android boot), we should run operations in a thread so that it does not block other file operations by Chrome.

2. Bundle several unprocessed file watcher events into one filesystem scan. If file scan operation is expensive it is possible file watcher events pile up.

3. Avoid base::FilePathWatcher and write our own more efficient file watcher. It is capable of notifying that some files were modified, but not the filenames. Then we can avoid scanning the whole tree of Downloads directory whenever files are modified.

(1) and (2) are easy, and then we should be able to avoid crosh lockups. We may want to do (3) eventually, but it needs very careful treatment of inotify complications.

FWIW FILE thread was deprecated years ago. You are advised to use the blocking pool instead.
(https://groups.google.com/a/chromium.org/forum/#!msg/chromium-dev/OaA-9rnjOqI/H0cDC6UiGpAJ)

Well that explains it all. Unfortunately, I can't use another folder outside of ~/Downloads (like ~/myProjects) because Chrome Apps can only access Drive, Downloads.

I've tried adding a blank ".nomedia" file to my projects folder (that has 100K files) it still locks up. Having .nomedia support in ChromeOS would also help with this issue.

Just a quick note, I moved my projects to /home/chronos/user/projects and just did a symlink at /home/chronos/user/Downloads/projects, and it's still scanning all the files.

I guess when it says it'll skip symlink, it only means hard symlinks on files?

Seems like it is desirable to fix this behavior. nya@ are you on it?

I'm starting on it.  I think the initial quick win will be moving it to the blocking thread pool (may still cause general slowdown, but hopefully shouldn't freeze the system).
After that I'm planning on looking in to optimizing.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/842221d3c80f2afd6fe3f6723b0b06d56fc6e541

commit 842221d3c80f2afd6fe3f6723b0b06d56fc6e541
Author: dspaid <dspaid@chromium.org>
Date: Mon Sep 12 05:47:55 2016

Only update watches when directories changed.

Because updating all watches for a recursively watched directory is an
expensive operation (requires scanning the entire directory tree below
a recursive watch target) the code normally only performs this if a
directory is modified or some component of |target_| changes.

Because the target path is not present in |recursive_paths_by_watch_|,
the current implementation treats all inotify events in the target
directory as changes that require an expensive update to all recursive
watches, even if the event is only related to a non-directory file.

This patch addresses the issue by only forcing a global update if the
fired watch is both not in |recursive_paths_by_watch_| as well as not
the |target_| watch.

BUG= 628223 
TEST=base_unittests continues to pass
Manual testing confirms performance changes from O(n) to O(1) for
touching a file in a watched directory.

Review-Url: https://codereview.chromium.org/2308413002
Cr-Commit-Position: refs/heads/master@{#417872}

[modify] https://crrev.com/842221d3c80f2afd6fe3f6723b0b06d56fc6e541/base/files/file_path_watcher_linux.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d8b3cb93177c7223bc67bce7b823083a380b19c2

commit d8b3cb93177c7223bc67bce7b823083a380b19c2
Author: dspaid <dspaid@chromium.org>
Date: Thu Sep 15 04:24:10 2016

Don't block FILE thread for media scanning

Only perform media scanning if a scan has not been performed since the file update event.

Building up the timestamp map is also moved off the FILE thread and on to the blocking pool.

BUG= 628223 
TEST=create a large (~10k) set of files under /home/chronos/user/Downloads
Enable ARC++
create a profiler snapshot at chrome://profiler
create one more new file under Downloads
Create another snapshot in chrome://profiler
Verify that there are no long-running tasks on the FILE thread, and that DelayBuildTimestampMap is only called once.

Review-Url: https://codereview.chromium.org/2310833002
Cr-Commit-Position: refs/heads/master@{#418778}

[modify] https://crrev.com/d8b3cb93177c7223bc67bce7b823083a380b19c2/chrome/browser/chromeos/arc/arc_downloads_watcher_service.cc

Solutions #1 and #2 from comment 21 have been implemented, as well as some general improvements to base::FilePathWatcher that make #3 not as important.  These should be in the M-55 release.

Dan, could you cherry-pick this to M54?

Merge approval requested for the two relevant CLs
https://codereview.chromium.org/2308413002
https://codereview.chromium.org/2310833002

Your change meets the bar and is auto-approved for M54 (branch: 2840)

This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible!

If all merges have been completed, please remove any remaining Merge-Approved labels from this issue.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Ok, both changes have been merged to M54.