New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 783732 link

Starred by 1 user

Issue metadata

Status: WontFix
Owner: ----
Closed: Mar 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Android
Pri: 2
Type: Bug



Sign in to add a comment

Smart Clip extraction does not return Clip Rect

Reported by mahesh...@samsung.com, Nov 10 2017

Issue description

Steps to reproduce the problem:
1. Use Device Note 8
2. Navigate to m.naver.com in Chrome
3. Use Stylus and Click on stylus button while hovering on screen.
4. Choose Translate icon.
5. Hover on any Korean text.

What is the expected behavior?
Translation is happened properly.

What went wrong?
Sometimes the Translate app is not able to recognize the text. Then it relies on Clip Rect being sent from browser engine to extract text on hovered area.

Did this work before? N/A 

Chrome version: 61.0.3163.100  Channel: n/a
OS Version: 8.0
Flash Version: 

This Clip Rect was removed in https://codereview.chromium.org/2667283007/ as there was no evidence that the OEM code used it. But, Translate app in Note 8 uses this clip rect and needs support from latest chrome engine.
 
Labels: Needs-triage-Mobile
Cc: msrchandra@chromium.org nyerramilli@chromium.org pnangunoori@chromium.org sandeepkumars@chromium.org
Labels: Triaged-Mobile Needs-Feedback
Tested the issue in Android and able to reproduce the issue only on one word. (But not completely sure whether it is a valid text or not)

Steps Followed:
1. Launched the Chrome Browser.
2. Navigate to m.naver.com 
3. Click on Stylus button and then tap on "Translate" button from Menu displayed.
4. Hover over the text displayed in Korean.
5. Observed that text is translated to English.

Chrome versions tested:
62.0.3202.84 (Stable)

OS:
Android 7.1.1

Android Devices:
Note 8 SM-N950F Build/NMF26X

@mahesh.ma -- Thanks for reporting this issue. Could you please provide screencast reproducing the issue. That would help us in further triaging the issue.

Thanks in advance.
It may not always reproduce to an issue as Samsung Framework makes the decision to use the clip Rect or not for Translating the text under Hover point. (I think they also have some other way of recognizing the screen text)

Samsung SPen Translate is the feature in consideration for us.

This feature is activated by clicking on SPen button on any page -> Selecting Translate.
Then Hover the spen on text to be translated.
The Hover point (making a Rect of 1x1 dp) is sent to WebContents.requestSmartClipExtract(int x, int y, int width, int height)

Samsung Framework has confirmed they need the Clip Rect, along with extracted text, which is returned from Blink as they use it for this feature. 
I have made a patch for this - PTAL
https://chromium-review.googlesource.com/c/chromium/src/+/762544
Project Member

Comment 4 by sheriffbot@chromium.org, Nov 14 2017

Labels: -Needs-Feedback
Thank you for providing more feedback. Adding requester "pnangunoori@chromium.org" to the cc list and removing "Needs-Feedback" label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Please find below the logs for the scenario in the video attached -- 

11-28 17:18:11.400 21156 21156 E smartclip: WebContentsImpl requestSmartClipExtract x,y,width,height=268,174,1,1,
11-28 17:18:11.473 21156 21156 E smartclip: WebContentsImpl onSmartClipDataExtracted clipRect=Rect(15, 165 - 331, 181)
11-28 17:18:11.473 21156 21156 E smartclip: WebContentsImpl onSmartClipDataExtracted text=
11-28 17:18:11.473 21156 21156 E smartclip: 
11-28 17:18:11.473 21156 21156 E smartclip: 중, 한국행 단체관광 금지 일부 해제…베이징·산둥성만


So, it can be seen that the request Rect is basically a 1x1 Rect at Hovered co-ordinates, but the clip Rect and extracted text actually returns the complete line being Hovered. Hence, passing the clipRect from blink so that it corresponds to the extracted text.

smart_clip_rect_2.mp4
4.0 MB View Download
Cc: dcheng@chromium.org
CC'ing reviewer for the further inputs from the CL: https://chromium-review.googlesource.com/c/chromium/src/+/762544

@dcheng - Could you please take a look.

Thanks in advance.
Status: WontFix (was: Unconfirmed)
***Bulk edit***

Since there is no valid updates, closing for now. Feel free to reopen if needed.
Project Member

Comment 8 by bugdroid1@chromium.org, May 24 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/346cd26654d5432be59abe7304d23300dfbf0499

commit 346cd26654d5432be59abe7304d23300dfbf0499
Author: Mahesh Machavolu <mahesh.ma@samsung.com>
Date: Thu May 24 12:40:11 2018

Return Clip Rect also during Smart clip extraction

Samsung smart select feature uses the extracted text returned from
SmartClipCallback->onSmartClipDataExtracted . This API earlier used
to return the Clip rect of the text that is being extracted. But,
this was removed in https://codereview.chromium.org/2667283007/ .

Now for Samsung Galazy Note 8, there is a new stylus feature called
"Translate" which also uses the same API for translating hovered text.
This API makes use of the Clip Rect for recognizing the hovered area
text and tries to translate it. So, adding this clip rect back again.

Bug:  783732 
Change-Id: I54e2f32f7862a2a4a2536b0113094d945160c714
Reviewed-on: https://chromium-review.googlesource.com/762544
Commit-Queue: AJITH KUMAR V <ajith.v@samsung.com>
Reviewed-by: Bo <boliu@chromium.org>
Reviewed-by: AJITH KUMAR V <ajith.v@samsung.com>
Reviewed-by: Jochen Eisinger <jochen@chromium.org>
Reviewed-by: Tom Sepez <tsepez@chromium.org>
Reviewed-by: David Trainor <dtrainor@chromium.org>
Cr-Commit-Position: refs/heads/master@{#561473}
[modify] https://crrev.com/346cd26654d5432be59abe7304d23300dfbf0499/chrome/android/javatests/src/org/chromium/chrome/browser/SmartClipProviderTest.java
[modify] https://crrev.com/346cd26654d5432be59abe7304d23300dfbf0499/content/browser/frame_host/render_frame_host_impl.cc
[modify] https://crrev.com/346cd26654d5432be59abe7304d23300dfbf0499/content/browser/frame_host/render_frame_host_impl.h
[modify] https://crrev.com/346cd26654d5432be59abe7304d23300dfbf0499/content/browser/web_contents/web_contents_android.cc
[modify] https://crrev.com/346cd26654d5432be59abe7304d23300dfbf0499/content/common/frame.mojom
[modify] https://crrev.com/346cd26654d5432be59abe7304d23300dfbf0499/content/public/android/java/src/org/chromium/content/browser/webcontents/WebContentsImpl.java
[modify] https://crrev.com/346cd26654d5432be59abe7304d23300dfbf0499/content/renderer/render_frame_impl.cc
[modify] https://crrev.com/346cd26654d5432be59abe7304d23300dfbf0499/third_party/blink/public/web/web_local_frame.h
[modify] https://crrev.com/346cd26654d5432be59abe7304d23300dfbf0499/third_party/blink/renderer/core/exported/web_frame_test.cc
[modify] https://crrev.com/346cd26654d5432be59abe7304d23300dfbf0499/third_party/blink/renderer/core/exported/web_view_test.cc
[modify] https://crrev.com/346cd26654d5432be59abe7304d23300dfbf0499/third_party/blink/renderer/core/frame/web_local_frame_impl.cc
[modify] https://crrev.com/346cd26654d5432be59abe7304d23300dfbf0499/third_party/blink/renderer/core/frame/web_local_frame_impl.h

This bug is fixed. 

Sign in to add a comment