Smart Clip extraction does not return Clip Rect
Reported by
mahesh...@samsung.com,
Nov 10 2017
|
|||||
Issue descriptionSteps to reproduce the problem: 1. Use Device Note 8 2. Navigate to m.naver.com in Chrome 3. Use Stylus and Click on stylus button while hovering on screen. 4. Choose Translate icon. 5. Hover on any Korean text. What is the expected behavior? Translation is happened properly. What went wrong? Sometimes the Translate app is not able to recognize the text. Then it relies on Clip Rect being sent from browser engine to extract text on hovered area. Did this work before? N/A Chrome version: 61.0.3163.100 Channel: n/a OS Version: 8.0 Flash Version: This Clip Rect was removed in https://codereview.chromium.org/2667283007/ as there was no evidence that the OEM code used it. But, Translate app in Note 8 uses this clip rect and needs support from latest chrome engine.
,
Nov 14 2017
Tested the issue in Android and able to reproduce the issue only on one word. (But not completely sure whether it is a valid text or not) Steps Followed: 1. Launched the Chrome Browser. 2. Navigate to m.naver.com 3. Click on Stylus button and then tap on "Translate" button from Menu displayed. 4. Hover over the text displayed in Korean. 5. Observed that text is translated to English. Chrome versions tested: 62.0.3202.84 (Stable) OS: Android 7.1.1 Android Devices: Note 8 SM-N950F Build/NMF26X @mahesh.ma -- Thanks for reporting this issue. Could you please provide screencast reproducing the issue. That would help us in further triaging the issue. Thanks in advance.
,
Nov 14 2017
It may not always reproduce to an issue as Samsung Framework makes the decision to use the clip Rect or not for Translating the text under Hover point. (I think they also have some other way of recognizing the screen text) Samsung SPen Translate is the feature in consideration for us. This feature is activated by clicking on SPen button on any page -> Selecting Translate. Then Hover the spen on text to be translated. The Hover point (making a Rect of 1x1 dp) is sent to WebContents.requestSmartClipExtract(int x, int y, int width, int height) Samsung Framework has confirmed they need the Clip Rect, along with extracted text, which is returned from Blink as they use it for this feature. I have made a patch for this - PTAL https://chromium-review.googlesource.com/c/chromium/src/+/762544
,
Nov 14 2017
Thank you for providing more feedback. Adding requester "pnangunoori@chromium.org" to the cc list and removing "Needs-Feedback" label. For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Nov 28 2017
Please find below the logs for the scenario in the video attached -- 11-28 17:18:11.400 21156 21156 E smartclip: WebContentsImpl requestSmartClipExtract x,y,width,height=268,174,1,1, 11-28 17:18:11.473 21156 21156 E smartclip: WebContentsImpl onSmartClipDataExtracted clipRect=Rect(15, 165 - 331, 181) 11-28 17:18:11.473 21156 21156 E smartclip: WebContentsImpl onSmartClipDataExtracted text= 11-28 17:18:11.473 21156 21156 E smartclip: 11-28 17:18:11.473 21156 21156 E smartclip: 중, 한국행 단체관광 금지 일부 해제…베이징·산둥성만 So, it can be seen that the request Rect is basically a 1x1 Rect at Hovered co-ordinates, but the clip Rect and extracted text actually returns the complete line being Hovered. Hence, passing the clipRect from blink so that it corresponds to the extracted text.
,
Nov 29 2017
CC'ing reviewer for the further inputs from the CL: https://chromium-review.googlesource.com/c/chromium/src/+/762544 @dcheng - Could you please take a look. Thanks in advance.
,
Mar 9 2018
***Bulk edit*** Since there is no valid updates, closing for now. Feel free to reopen if needed.
,
May 24 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/346cd26654d5432be59abe7304d23300dfbf0499 commit 346cd26654d5432be59abe7304d23300dfbf0499 Author: Mahesh Machavolu <mahesh.ma@samsung.com> Date: Thu May 24 12:40:11 2018 Return Clip Rect also during Smart clip extraction Samsung smart select feature uses the extracted text returned from SmartClipCallback->onSmartClipDataExtracted . This API earlier used to return the Clip rect of the text that is being extracted. But, this was removed in https://codereview.chromium.org/2667283007/ . Now for Samsung Galazy Note 8, there is a new stylus feature called "Translate" which also uses the same API for translating hovered text. This API makes use of the Clip Rect for recognizing the hovered area text and tries to translate it. So, adding this clip rect back again. Bug: 783732 Change-Id: I54e2f32f7862a2a4a2536b0113094d945160c714 Reviewed-on: https://chromium-review.googlesource.com/762544 Commit-Queue: AJITH KUMAR V <ajith.v@samsung.com> Reviewed-by: Bo <boliu@chromium.org> Reviewed-by: AJITH KUMAR V <ajith.v@samsung.com> Reviewed-by: Jochen Eisinger <jochen@chromium.org> Reviewed-by: Tom Sepez <tsepez@chromium.org> Reviewed-by: David Trainor <dtrainor@chromium.org> Cr-Commit-Position: refs/heads/master@{#561473} [modify] https://crrev.com/346cd26654d5432be59abe7304d23300dfbf0499/chrome/android/javatests/src/org/chromium/chrome/browser/SmartClipProviderTest.java [modify] https://crrev.com/346cd26654d5432be59abe7304d23300dfbf0499/content/browser/frame_host/render_frame_host_impl.cc [modify] https://crrev.com/346cd26654d5432be59abe7304d23300dfbf0499/content/browser/frame_host/render_frame_host_impl.h [modify] https://crrev.com/346cd26654d5432be59abe7304d23300dfbf0499/content/browser/web_contents/web_contents_android.cc [modify] https://crrev.com/346cd26654d5432be59abe7304d23300dfbf0499/content/common/frame.mojom [modify] https://crrev.com/346cd26654d5432be59abe7304d23300dfbf0499/content/public/android/java/src/org/chromium/content/browser/webcontents/WebContentsImpl.java [modify] https://crrev.com/346cd26654d5432be59abe7304d23300dfbf0499/content/renderer/render_frame_impl.cc [modify] https://crrev.com/346cd26654d5432be59abe7304d23300dfbf0499/third_party/blink/public/web/web_local_frame.h [modify] https://crrev.com/346cd26654d5432be59abe7304d23300dfbf0499/third_party/blink/renderer/core/exported/web_frame_test.cc [modify] https://crrev.com/346cd26654d5432be59abe7304d23300dfbf0499/third_party/blink/renderer/core/exported/web_view_test.cc [modify] https://crrev.com/346cd26654d5432be59abe7304d23300dfbf0499/third_party/blink/renderer/core/frame/web_local_frame_impl.cc [modify] https://crrev.com/346cd26654d5432be59abe7304d23300dfbf0499/third_party/blink/renderer/core/frame/web_local_frame_impl.h
,
May 25 2018
This bug is fixed. |
|||||
►
Sign in to add a comment |
|||||
Comment 1 by pnangunoori@chromium.org
, Nov 13 2017