New issue
Advanced search Search tips

Issue 826614 link

Starred by 29 users

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug

Blocked on:
issue 866189


Show other hotlists

Hotlists containing this issue:
Chromium-bugs-related-to-Crostini


Sign in to add a comment

Input Method support / integration with Linux app

Project Member Reported by mukai@chromium.org, Mar 28 2018

Issue description

Chrome version: 67.0.3376.0 (Official Build) dev (64-bit)
OS: Chrome

Repro steps:
1. set up crostini, install VSCode
2. start VSCode
3. switch input language to Japanese
4. type 'a' in VSCode window

Expected: 
- start the normal text input flow (for Japanese) within the VSCode

Actual: 
- simply 'a' appears in the app, input language is ignored
 

Comment 1 by vapier@chromium.org, Mar 28 2018

i guess the implication is that you're expecting the existing CrOS IME to be used ?  and, at least by default, not use an IME that the container has installed (e.g. anthy) ?  that's going to be tricky i think.

Comment 2 by mukai@chromium.org, Mar 28 2018

Cc: yukawa@chromium.org nona@chromium.org
yes, use of ChromeOS IME on linux apps in the container, and yes indeed it should be very tricky. However at least that works on Android apps in ARC++. nona@ and yukawa@ should know how ARC++ interacts with ChromeOS IMEs.

Installing linux IMEs into the container could be a choice, however an IME needs a bit more UI things which also complicates things on crostini apps, like current IME indicator (typically resides on a system tray on the linux desktop), global keyboard shortcut of IME switching, some additional windows like candidate windows.

An idea is to develop a "fake IME" in the container which implements something like gtkimmodule or fcitx and communicates with ChromeOS IME.
Cc: yhanada@chromium.org
FYI: For ARC++, we have a proxy of IME events in components/arc/ime. All IME events (for example, insertText) are sent to container via this proxy and the fake IME installed in container side handles the events.
Owner: shuchen@chromium.org
Status: Untriaged (was: Available)
 Issue 829166  has been merged into this issue.
Labels: -Pri-2 M-68 Pri-1
Owner: tbuck...@chromium.org
Status: Assigned (was: Untriaged)
+some crostini-ui folks

yukawa, nona -- we should meet up to discuss how to make IMEs work with Crostini.

Comment 7 by mukai@chromium.org, Apr 23 2018

Cc: kinaba@chromium.org
re #6: Sorry, correction; I've heard that yhanada@ and kinaba@ are responsible for IME support of ARC++ apps.
Cc: -nona@chromium.org -yukawa@chromium.org reve...@chromium.org jkardatzke@chromium.org
+jkardatzke, reveman -- it sounds like there might be a way to do this over Wayland?
Yes, we need to implement the upstream input method interface in chrome (https://cs.chromium.org/chromium/src/third_party/wayland-protocols/src/unstable/input-method/input-method-unstable-v1.xml). kinaba@ looked at this as part of implementing input methods for arc++ and would be the ideal person to work on crostini support.

Comment 10 by mukai@chromium.org, Apr 24 2018

It's possible to go through wayland, and actually there is already an unstable input-method protocol definition, but I think we may want to extend the API if we want the full feature (related to ARC++ protocol).

I've been writing some design doc / notes internally, I'll share it with you all.
For record, what I chatted with David a few weeks ago were:
* The structure/the conceptual design of Chrome IMF and the wayland protocol is basically similar. So the basic set of IME features should roughly map each other.
* There may be tough things in details. One thing is that Wayland uses UTF-8/byte-index and Chrome uses UTC-16/codepoint-index. Translating IME request like ("delete one char before the caret" may not simply work, because we don't know how many bytes to erase.)



What I actually don't know is the Crostini or wayland-client side.
Does the apps running inside is expected to implement the client-side IME protocol and adding the host-side implementation will make things work?
Or do we have to add some client IME service like fcitx or ibus to make the apps work?
Yes, it's expected that toolkits like GTK and QT implement support for that protocol. Not sure what the story is for Xwayland. We might need some XIM (https://www.x.org/releases/X11R7.6/doc/libX11/specs/XIM/xim.html) bridge for that to work.
Cc: mukai@chromium.org
Ignore my previous comment. mukai@ has been investigating this and created a proposal: https://docs.google.com/document/d/1TrDigvi7NwfwveBlmGxP7As0X1myv1qCM7UG-WhO3Po/edit?usp=sharing
Labels: Hotlist-Crostini-UI
Labels: -Restrict-View-Google
Owner: mukai@chromium.org
Started the implementation based on the document in #c13
Project Member

Comment 17 by bugdroid1@chromium.org, Jul 18

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/ed95ce6013afb44be94d45167c080d12e597a75e

commit ed95ce6013afb44be94d45167c080d12e597a75e
Author: Jun Mukai <mukai@google.com>
Date: Wed Jul 18 00:30:35 2018

vm_tools: sommelier: add text_input_unstable_v1 protocol

This is one of CLs for the integration between crostini apps and
ChromeOS input methods. This should work with the text-input server
within Chrome (https://chromium-review.googlesource.com/c/chromium/src/+/1132332/2)
and text-input clients within Linux apps. Plugins for such
Linux app are not yet publicly available -- will come in
other CLs.

BUG=chromium:826614
TEST=manually -- run with server in Chrome and Linux app with special plugin
Change-Id: I510ba7ee688001db529f6ea90522ba0ca7c59932
Reviewed-on: https://chromium-review.googlesource.com/1130504
Commit-Ready: Jun Mukai <mukai@chromium.org>
Tested-by: Jun Mukai <mukai@chromium.org>
Reviewed-by: David Reveman <reveman@chromium.org>

[modify] https://crrev.com/ed95ce6013afb44be94d45167c080d12e597a75e/vm_tools/sommelier/sommelier.c
[add] https://crrev.com/ed95ce6013afb44be94d45167c080d12e597a75e/vm_tools/sommelier/protocol/text-input-unstable-v1.xml
[add] https://crrev.com/ed95ce6013afb44be94d45167c080d12e597a75e/vm_tools/sommelier/sommelier-text-input.c
[modify] https://crrev.com/ed95ce6013afb44be94d45167c080d12e597a75e/vm_tools/sommelier/sommelier.h
[modify] https://crrev.com/ed95ce6013afb44be94d45167c080d12e597a75e/vm_tools/sommelier/sommelier.gyp

Project Member

Comment 18 by bugdroid1@chromium.org, Jul 20

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a446ee7f1fc1e9560d6d5289454ae7bc9e0c723e

commit a446ee7f1fc1e9560d6d5289454ae7bc9e0c723e
Author: Jun Mukai <mukai@chromium.org>
Date: Fri Jul 20 22:55:15 2018

initial support of text-input-unstable protocol

This CL introduces the support of text-input-unstable protocol
defined in wayland-protocols repository.

This CL will be the first one for a series of CLs for supporting
this protocol and lacks some parts surrounding text.

BUG=826614
TEST=manually, exo_unittests

Change-Id: I4f4e335ed26c51beea9f25479acc63e11f09f777
Reviewed-on: https://chromium-review.googlesource.com/1132332
Commit-Queue: Jun Mukai <mukai@chromium.org>
Reviewed-by: Antoine Labour <piman@chromium.org>
Reviewed-by: Mitsuru Oshima <oshima@chromium.org>
Reviewed-by: Yuichiro Hanada <yhanada@chromium.org>
Cr-Commit-Position: refs/heads/master@{#577022}
[modify] https://crrev.com/a446ee7f1fc1e9560d6d5289454ae7bc9e0c723e/components/exo/BUILD.gn
[add] https://crrev.com/a446ee7f1fc1e9560d6d5289454ae7bc9e0c723e/components/exo/text_input.cc
[add] https://crrev.com/a446ee7f1fc1e9560d6d5289454ae7bc9e0c723e/components/exo/text_input.h
[add] https://crrev.com/a446ee7f1fc1e9560d6d5289454ae7bc9e0c723e/components/exo/text_input_unittest.cc
[modify] https://crrev.com/a446ee7f1fc1e9560d6d5289454ae7bc9e0c723e/components/exo/wayland/BUILD.gn
[modify] https://crrev.com/a446ee7f1fc1e9560d6d5289454ae7bc9e0c723e/components/exo/wayland/server.cc
[modify] https://crrev.com/a446ee7f1fc1e9560d6d5289454ae7bc9e0c723e/third_party/wayland-protocols/BUILD.gn
[modify] https://crrev.com/a446ee7f1fc1e9560d6d5289454ae7bc9e0c723e/third_party/wayland-protocols/README.chromium
[add] https://crrev.com/a446ee7f1fc1e9560d6d5289454ae7bc9e0c723e/third_party/wayland-protocols/include/protocol/text-input-unstable-v1-client-protocol.h
[add] https://crrev.com/a446ee7f1fc1e9560d6d5289454ae7bc9e0c723e/third_party/wayland-protocols/include/protocol/text-input-unstable-v1-server-protocol.h
[add] https://crrev.com/a446ee7f1fc1e9560d6d5289454ae7bc9e0c723e/third_party/wayland-protocols/protocol/text-input-v1-protocol.c

Blockedon: 866189
Will it be possible to switch to another language in the linux app via the linux IME such as ibus? This will be helpful for languages that are not available in the chrome OS IME.

Currently, the shortcuts for switching via ibus do not appear to work - they are not passed through to the app. However, they can be manually switched via `ibus engine` command. Would be nice to be able to switch input methods within linux using the keyboard shortcuts as well.
#20, that is not a goal of this effort, sorry. So far the goal is to make ChromeOS IMEs work with Linux apps.

Also I think the existing Linux IME frameworks like IBus are already working, at least at some level. If some features (like IME switching) isn't in IBus, I am afraid that's a bug of IBus, not ChromeOS.
#21, thanks for the comment. I understand the distinction, I just wasn't sure how the ChromeOS IME interacts with the underlying Linux IME. I can get IBus to switch from the CLI, but what I haven't been able to figure out is how to switch via the keyboard shortcuts to the linux IME. It appears that the shortcuts are being "grabbed" by ChromeOS, though they don't appear to be reserved in the Ash accelerator table.
My current plan is a separate Gtk ImModule or an XIM server. This means you'll set like "export GTK_IM_MODULE=cros", and then the app will connect to the chromeOS IME instead of entities like IBus. With this scheme, shortcuts like IME switching will be handled at ChromeOS level, and that would be fine.

However, always stealing IME shortcuts isn't a good idea, as you pointed out. And I don't think that "grabbing" isn't the current behavior -- the IME switching shortcuts will come to the linux environment, I think.

By the way, what is the keyboard shortcut exactly you are referring?  I guess it's Ctrl-space or Alt-space?
Labels: -M-68 M-70
Maybe this isn't for M-68.
You can set the shortcut to anything you like, I have tried a whole bunch of combinations to see if any of them got through; nothing I tried worked. The default for iBus is ctrl-space.
I was asking since ctrl-space is also the IME switching shortcut key for ChromeOS, which could be intercepted by ChromeOS systems (although it shouldn't be). Things like alt-space wouldn't be used by ChromeOS IIRC, thus it should come to the Linux app as usual, so it's a mystery why it doesn't trigger IBus features -- let's track that on a separate issue. I filed issue 866681 for this; feel free to fill more details there. Thanks!
Great, thanks for the help!
 Issue 835718  has been merged into this issue.

Sign in to add a comment