New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 826614 link

Starred by 40 users

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug

Blocked on:
issue 866189


Show other hotlists

Hotlists containing this issue:
Chromium-bugs-related-to-Crostini


Sign in to add a comment

Input Method support / integration with Linux app

Project Member Reported by mukai@chromium.org, Mar 28 2018

Issue description

Chrome version: 67.0.3376.0 (Official Build) dev (64-bit)
OS: Chrome

Repro steps:
1. set up crostini, install VSCode
2. start VSCode
3. switch input language to Japanese
4. type 'a' in VSCode window

Expected: 
- start the normal text input flow (for Japanese) within the VSCode

Actual: 
- simply 'a' appears in the app, input language is ignored
 

Comment 1 by vapier@chromium.org, Mar 28 2018

i guess the implication is that you're expecting the existing CrOS IME to be used ?  and, at least by default, not use an IME that the container has installed (e.g. anthy) ?  that's going to be tricky i think.

Comment 2 by mukai@chromium.org, Mar 28 2018

Cc: yukawa@chromium.org nona@chromium.org
yes, use of ChromeOS IME on linux apps in the container, and yes indeed it should be very tricky. However at least that works on Android apps in ARC++. nona@ and yukawa@ should know how ARC++ interacts with ChromeOS IMEs.

Installing linux IMEs into the container could be a choice, however an IME needs a bit more UI things which also complicates things on crostini apps, like current IME indicator (typically resides on a system tray on the linux desktop), global keyboard shortcut of IME switching, some additional windows like candidate windows.

An idea is to develop a "fake IME" in the container which implements something like gtkimmodule or fcitx and communicates with ChromeOS IME.
Cc: yhanada@chromium.org
FYI: For ARC++, we have a proxy of IME events in components/arc/ime. All IME events (for example, insertText) are sent to container via this proxy and the fake IME installed in container side handles the events.
Owner: shuchen@chromium.org
Status: Untriaged (was: Available)
 Issue 829166  has been merged into this issue.
Labels: -Pri-2 M-68 Pri-1
Owner: tbuck...@chromium.org
Status: Assigned (was: Untriaged)
+some crostini-ui folks

yukawa, nona -- we should meet up to discuss how to make IMEs work with Crostini.

Comment 7 by mukai@chromium.org, Apr 23 2018

Cc: kinaba@chromium.org
re #6: Sorry, correction; I've heard that yhanada@ and kinaba@ are responsible for IME support of ARC++ apps.
Cc: -nona@chromium.org -yukawa@chromium.org reve...@chromium.org jkardatzke@chromium.org
+jkardatzke, reveman -- it sounds like there might be a way to do this over Wayland?
Yes, we need to implement the upstream input method interface in chrome (https://cs.chromium.org/chromium/src/third_party/wayland-protocols/src/unstable/input-method/input-method-unstable-v1.xml). kinaba@ looked at this as part of implementing input methods for arc++ and would be the ideal person to work on crostini support.

Comment 10 by mukai@chromium.org, Apr 24 2018

It's possible to go through wayland, and actually there is already an unstable input-method protocol definition, but I think we may want to extend the API if we want the full feature (related to ARC++ protocol).

I've been writing some design doc / notes internally, I'll share it with you all.
For record, what I chatted with David a few weeks ago were:
* The structure/the conceptual design of Chrome IMF and the wayland protocol is basically similar. So the basic set of IME features should roughly map each other.
* There may be tough things in details. One thing is that Wayland uses UTF-8/byte-index and Chrome uses UTC-16/codepoint-index. Translating IME request like ("delete one char before the caret" may not simply work, because we don't know how many bytes to erase.)



What I actually don't know is the Crostini or wayland-client side.
Does the apps running inside is expected to implement the client-side IME protocol and adding the host-side implementation will make things work?
Or do we have to add some client IME service like fcitx or ibus to make the apps work?
Yes, it's expected that toolkits like GTK and QT implement support for that protocol. Not sure what the story is for Xwayland. We might need some XIM (https://www.x.org/releases/X11R7.6/doc/libX11/specs/XIM/xim.html) bridge for that to work.
Cc: mukai@chromium.org
Ignore my previous comment. mukai@ has been investigating this and created a proposal: https://docs.google.com/document/d/1TrDigvi7NwfwveBlmGxP7As0X1myv1qCM7UG-WhO3Po/edit?usp=sharing
Labels: Hotlist-Crostini-UI
Labels: -Restrict-View-Google
Owner: mukai@chromium.org
Started the implementation based on the document in #c13
Project Member

Comment 17 by bugdroid1@chromium.org, Jul 18

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/ed95ce6013afb44be94d45167c080d12e597a75e

commit ed95ce6013afb44be94d45167c080d12e597a75e
Author: Jun Mukai <mukai@google.com>
Date: Wed Jul 18 00:30:35 2018

vm_tools: sommelier: add text_input_unstable_v1 protocol

This is one of CLs for the integration between crostini apps and
ChromeOS input methods. This should work with the text-input server
within Chrome (https://chromium-review.googlesource.com/c/chromium/src/+/1132332/2)
and text-input clients within Linux apps. Plugins for such
Linux app are not yet publicly available -- will come in
other CLs.

BUG=chromium:826614
TEST=manually -- run with server in Chrome and Linux app with special plugin
Change-Id: I510ba7ee688001db529f6ea90522ba0ca7c59932
Reviewed-on: https://chromium-review.googlesource.com/1130504
Commit-Ready: Jun Mukai <mukai@chromium.org>
Tested-by: Jun Mukai <mukai@chromium.org>
Reviewed-by: David Reveman <reveman@chromium.org>

[modify] https://crrev.com/ed95ce6013afb44be94d45167c080d12e597a75e/vm_tools/sommelier/sommelier.c
[add] https://crrev.com/ed95ce6013afb44be94d45167c080d12e597a75e/vm_tools/sommelier/protocol/text-input-unstable-v1.xml
[add] https://crrev.com/ed95ce6013afb44be94d45167c080d12e597a75e/vm_tools/sommelier/sommelier-text-input.c
[modify] https://crrev.com/ed95ce6013afb44be94d45167c080d12e597a75e/vm_tools/sommelier/sommelier.h
[modify] https://crrev.com/ed95ce6013afb44be94d45167c080d12e597a75e/vm_tools/sommelier/sommelier.gyp

Project Member

Comment 18 by bugdroid1@chromium.org, Jul 20

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a446ee7f1fc1e9560d6d5289454ae7bc9e0c723e

commit a446ee7f1fc1e9560d6d5289454ae7bc9e0c723e
Author: Jun Mukai <mukai@chromium.org>
Date: Fri Jul 20 22:55:15 2018

initial support of text-input-unstable protocol

This CL introduces the support of text-input-unstable protocol
defined in wayland-protocols repository.

This CL will be the first one for a series of CLs for supporting
this protocol and lacks some parts surrounding text.

BUG=826614
TEST=manually, exo_unittests

Change-Id: I4f4e335ed26c51beea9f25479acc63e11f09f777
Reviewed-on: https://chromium-review.googlesource.com/1132332
Commit-Queue: Jun Mukai <mukai@chromium.org>
Reviewed-by: Antoine Labour <piman@chromium.org>
Reviewed-by: Mitsuru Oshima <oshima@chromium.org>
Reviewed-by: Yuichiro Hanada <yhanada@chromium.org>
Cr-Commit-Position: refs/heads/master@{#577022}
[modify] https://crrev.com/a446ee7f1fc1e9560d6d5289454ae7bc9e0c723e/components/exo/BUILD.gn
[add] https://crrev.com/a446ee7f1fc1e9560d6d5289454ae7bc9e0c723e/components/exo/text_input.cc
[add] https://crrev.com/a446ee7f1fc1e9560d6d5289454ae7bc9e0c723e/components/exo/text_input.h
[add] https://crrev.com/a446ee7f1fc1e9560d6d5289454ae7bc9e0c723e/components/exo/text_input_unittest.cc
[modify] https://crrev.com/a446ee7f1fc1e9560d6d5289454ae7bc9e0c723e/components/exo/wayland/BUILD.gn
[modify] https://crrev.com/a446ee7f1fc1e9560d6d5289454ae7bc9e0c723e/components/exo/wayland/server.cc
[modify] https://crrev.com/a446ee7f1fc1e9560d6d5289454ae7bc9e0c723e/third_party/wayland-protocols/BUILD.gn
[modify] https://crrev.com/a446ee7f1fc1e9560d6d5289454ae7bc9e0c723e/third_party/wayland-protocols/README.chromium
[add] https://crrev.com/a446ee7f1fc1e9560d6d5289454ae7bc9e0c723e/third_party/wayland-protocols/include/protocol/text-input-unstable-v1-client-protocol.h
[add] https://crrev.com/a446ee7f1fc1e9560d6d5289454ae7bc9e0c723e/third_party/wayland-protocols/include/protocol/text-input-unstable-v1-server-protocol.h
[add] https://crrev.com/a446ee7f1fc1e9560d6d5289454ae7bc9e0c723e/third_party/wayland-protocols/protocol/text-input-v1-protocol.c

Blockedon: 866189
Will it be possible to switch to another language in the linux app via the linux IME such as ibus? This will be helpful for languages that are not available in the chrome OS IME.

Currently, the shortcuts for switching via ibus do not appear to work - they are not passed through to the app. However, they can be manually switched via `ibus engine` command. Would be nice to be able to switch input methods within linux using the keyboard shortcuts as well.
#20, that is not a goal of this effort, sorry. So far the goal is to make ChromeOS IMEs work with Linux apps.

Also I think the existing Linux IME frameworks like IBus are already working, at least at some level. If some features (like IME switching) isn't in IBus, I am afraid that's a bug of IBus, not ChromeOS.
#21, thanks for the comment. I understand the distinction, I just wasn't sure how the ChromeOS IME interacts with the underlying Linux IME. I can get IBus to switch from the CLI, but what I haven't been able to figure out is how to switch via the keyboard shortcuts to the linux IME. It appears that the shortcuts are being "grabbed" by ChromeOS, though they don't appear to be reserved in the Ash accelerator table.
My current plan is a separate Gtk ImModule or an XIM server. This means you'll set like "export GTK_IM_MODULE=cros", and then the app will connect to the chromeOS IME instead of entities like IBus. With this scheme, shortcuts like IME switching will be handled at ChromeOS level, and that would be fine.

However, always stealing IME shortcuts isn't a good idea, as you pointed out. And I don't think that "grabbing" isn't the current behavior -- the IME switching shortcuts will come to the linux environment, I think.

By the way, what is the keyboard shortcut exactly you are referring?  I guess it's Ctrl-space or Alt-space?
Labels: -M-68 M-70
Maybe this isn't for M-68.
You can set the shortcut to anything you like, I have tried a whole bunch of combinations to see if any of them got through; nothing I tried worked. The default for iBus is ctrl-space.
I was asking since ctrl-space is also the IME switching shortcut key for ChromeOS, which could be intercepted by ChromeOS systems (although it shouldn't be). Things like alt-space wouldn't be used by ChromeOS IIRC, thus it should come to the Linux app as usual, so it's a mystery why it doesn't trigger IBus features -- let's track that on a separate issue. I filed issue 866681 for this; feel free to fill more details there. Thanks!
Great, thanks for the help!
 Issue 835718  has been merged into this issue.
I found some behavior that related with this issue it would help you fix easier. In a terminal, I can type another language but I have to minimize Terminal then switch layout and back to Terminal to type again but in other applications (e.g., Vscode) are not support this behavior, Even I go out to Chrome OS to switch language and back to VScode my keyboard language is still English 
I don't fully understand how input events work these days, but I believe there is a difference between X11 and Wayland applications. And that's quite possibly what you're seeing here
Sorry for the slow progress here. Basically, the Terminal is quite different from other Linux apps, from the text-input perspective. Normal Linux apps will communicate with ChromeOS window system through Wayland protocol, while Terminal isn't.

I've brought a wayland protocol support (zwp-text-input-v1) already (see https://cs.chromium.org/chromium/src/components/exo/wayland/zwp_text_input_manager.cc?g=0), so the Linux environment should be able to see this. The remaining part is the Linux plugins (i.e. GtkImModule or XIM) implementing this protocol.

https://chromium-review.googlesource.com/c/chromiumos/platform/cros-ime-bridge/+/1213614 is the initial CL for GtkImModule implementation, left work-in-progress for months but I've been busy on other things lately. I'll revisit soon.

Note that this GtkImModule is specific to Gtk3 and so not working with apps like VSCode unfortunately;  VSCode is on Gtk2, which lacks support of wayland directly. Same with Android Studio.

Comment 32 Deleted

Do you have any plan to make input method support Gtk2 or wait for Vscode update to gtk3? I'm very focused on this issue because of this in the last point for why I do not change to Chrome OS
Yes, I'll work on Gtk2 or XIM later, it was Gtk3 specific because it's simpler. Yet those supports need extra efforts and time, I can't promise when it's going to be done. Helps and contributions would be much appreciated.

Understand, I don't want a promise I only want to know your plan. I love to help, but I'm a web developer. My C++ skill of mine is zero. So sorry
Would this bug also cover support for on-screen keyboard input for Linux applications? As it stands, there appears to be no practical way to input text on tablet devices (or when devices are in tablet mode). Even forcing the on screen keyboard to open using accessibility settings doesn't rectify the issue as input from the software keyboard is not passed through.
#36: yes. The virtual keyboard (onscreen keyboard) of ChromeOS is also an input method, it works with the same protocol. Actually my demo on my local device works with the virtual keyboard.

However keep in mind that the virtual keyboard support won't be perfect. Yes it can input characters, though some neat features won't be there (for example, the text caret might be hidden by the virtual keyboard itself when it's on the bottom part of the screen -- the app won't scroll up automatically upon virtual keyboard appearance like mobile apps do). This is because the Linux apps do not care about tablet and virtual keyboard.
This is because the Linux apps do not care about tablet and virtual keyboard.

Ad far I can see, with the latest dev build 72.0.3609.3, when I recall chromeOs ime from the accessibility menu, do resize themself leaving all the necessary room for the on-screen keyboard. Such behaviour looks to be consistent across all applications, such as libreoffice, thunar, konsole... So, when your plugin is done, it should work in a neat way. I have had a look at it anyway and it looks that you are close to the point, apart from using C instead of C++. Good work, this will be a killer feature for tablet mode.
This P1 bug seems to have been stalled for a while.

mukai - do you have cycles to work on this for 73? If you don't maybe we should fine a new owner or update the priority.
Cc: benwells@chromium.org
Labels: -Pri-1 -M-70 M-74 Pri-2
I hope I could do something for M73, but I'm afraid that it is not likely to be done fully by M73. Since Crostini is still in beta, let me deprioritize this to P2 (tbuckley -- please reprerioritize if this is wrong).

benwells, if you or someone in your team is interested in working on this, I'm really happy. I can share any resources if so.
P2 sounds right to me, and 73 or 74 seems acceptable ... but please let us know if you think you won't get cycles to do that and we will try to figure something out.
Just want to say that I'll be happy to see this working for myself and all the new Pixel Slate users. I was bummed to breakout into Tablet mode and find the the virtual keyboard didn't popup when working in PyCharm. I'd hope to be able to do some code when travelling without the keyboard case.

Anyways - good work everyone! Just having Crostini is amazing and pushed to buy a Chrome device.

Comment 44 by dvallet@chromium.org, Dec 10 (2 days ago)

 Issue 913533  has been merged into this issue.

Comment 45 by mukai@chromium.org, Today (13 hours ago)

Owner: tbuck...@chromium.org
it seems it's hard for me to make time for this effort. tbuckley@, please find the owner.

Comment 46 by jkardatzke@chromium.org, Today (13 hours ago)

Owner: timzheng@chromium.org

Comment 47 by jkardatzke@chromium.org, Today (13 hours ago)

We will take care of this in the LA team.

Sign in to add a comment