Reference Issue 705657 (main launch bug for translate event logging).

Jon/Renjie - roughly how much work would it be to change how we're doing the event logging to log an event when the translate bar is shown, so that we can infer IGNOREs on Mac?

I think we need to do the following:
1) modify metrics logging to add an additional event to log:https://cs.chromium.org/chromium/src/components/metrics/proto/translate_event.proto?type=cs&q=translate+metrics&sq=package:chromium&l=71 (this includes google3 change as well, we need to double the current translate log storage)
2) modify translate/language preferences logging

+hamelphi@ do you have any suggestions?

+rogerm@
Good catch. It would really improve the quality of TranslateRankerModel to know about these ignores. Note that this also affects other platforms use the infobar (e.g. Android, iOS, ...).

In these cases, the TranslateEventProto (TEP) is initialized, but never sent to UMA because it is not complete. My suggestion would be to work at the TranslateManager level to make sure we don't drop them. Maybe we could force the recording of incomplete event when the TranslateManager is destroyed?

I am not 100% this would work, but here what I suggest:
- Add a new bool (or maybe enum if we want info on what type of UI was shown (e.g. infobar, bubbleUI, compactUI) in TranslateEventProto that logs if the UI was shown. This is not mandatory, but could be helpful anyway to debug our logs.
- At TranslateManager destruction time check if the TEP is initialized but incomplete, and record it. We could log it as either EventType USER_IGNORE, UNKNOWN, or some new type like USER_NO_FEEDBACK that would allow us to discriminate these events from other explicit USER_IGNORE.

On Android and iOS, doesn't the infobar disappear automatically if you don't interact with it? Is comment #3 saying that we currently do not record IGNORE events for those platforms anyways?

Doubling the current translate log storage sounds like a potentially long-haul change.

Actually, after digging a bit more in the logs, I believe the USER_IGNORE events that we thought were missing are logged as USER_DECLINE.

If you take a look at this dashboard:
https://dashboards.corp.google.com/google::_fda05998_0de3_4a9c_8d63_fa9b1c19d16c

You'll see that the ratio of USER_ACCEPT is relatively the same accross platforms (12-14%), but the distribution of USER_IGNORE and USER_DECLINE is very different between infobar (Mac, Android) and bubbleUI (Windows, Linux) platforms. So, my guess is that we only log USER_DECLINE events with the infobar. There is a tiny amount of USER_IGNORE for Mac that I can't really explain.

I can't really figure out the codepath of the infobar, I suppose  TranslateUIDelegate::TranslationDeclined (https://cs.chromium.org/chromium/src/components/translate/core/browser/translate_ui_delegate.cc?rcl=b31e727530f2d633fff54d1c2b7a3e9a866065fe&l=234) is called somewhere, and I guess the explicitly_closed bool is probably always true for infobar.
rogerm@ may have better insights on this.

For TranslateRanker, we don't discrimnate between ignores and declines for training, so we are not affected by this.

My apologies, my filters were not correctly set on the dashboard, so I did not analyze the situation correctly.

We see that USER_ACCEPT events are abnormally high for Mac (30%) compared to other platforms (10-14%) which does point to missing events.

For Android and iOS, we only get USER_DECLINE events, as stated above. For Mac, we do get a tiny amount of USER_IGNORE events (probably when the user clicks the 'x' on the right of the infobar?), but I believe that we are missing the bulk of it, since the ratio of USER_ACCEPT doesn't match.

Thanks Philippe for the suggestions. I will try if we can force logging in the destruction phase.

Since I don't have access to the dashboard, I don't have many insights about what's going on. But from my personal experiment, if we enable the New Translate UX on mac (chrome://flags -> New Translate UX), we can trigger IGNORE event by just going to a different tab. Maybe that's why there's a small portion of IGNORE on mac?

I haven't tested on Android, iOS, Windows, or Linux. But on Mac, whenever I click the "x" or "Nope," (with "New Translate UX" turned off, i.e. the default state), it records a DECLINE user event. IGNORE is never logged.

I'd like to clarify what's going on on Android specifically, since that could dramatically change the impact of this bug. Android is an inforbar (like Mac). Can it be dismissed automatically if you ignore it for long enough (or navigate within the tab somewhere, e.g. by tapping a link)? Or is it like Mac, where it stays around until you explicitly interact with it.

If it's the former, then shouldn't we still get IGNORE events when the user doesn't click anything on the infobar?

I agree with you that when the user doesn't click on the infobar on android and it goes away, it would be logical to log it as IGNORE. However, in the current state, as I understand it, we log them as DECLINE (and I am pretty convinced that we DO log all these events for Android).

In the current state, the infobar code does not discriminate between explicit declines and other ways of dismissing the UI. It always sends back an explicitly_closed to the UI delegate:
https://cs.chromium.org/chromium/src/components/translate/core/browser/translate_infobar_delegate.cc?rcl=7be9a5b4591d062524aa41b77ed03f14c474bbe3&l=160
which is then logged as USER_DECLINE.

Got it, okay, thanks for the clarification. This actually may have a significant impact on our analysis for the event logger then.

Renjie/Jon/Yana - how much will it affect our ability to do the proper modeling if we can't differentiate between ignoring the infobar, clicking "x," and clicking "Nope"?

Since we don't have a baseline model yet, I guess it is hard to tell how large is the impact if we can't differentiate between ignoring and clicking "x" or clicking "Nope" at this point.

+hamelphi@ how's the impact for the translate ranker model?

Meanwhile, I'm aiming to find a proper way to log the ignore event for info-bars.

For training TranslateRanker, we treat all 'negative' interactions the same. We don't discriminate between USER_DECLINE, USER_IGNORE, etc. As long as the UI was shown to the user and the outcome is not USER_ACCEPT, it is considered as a negative example.

With more accurate logging, we could potentially refine this and maybe have different weights on the negative events, but it was not necessary to do so to get a decent model.

However, the bug described here where we are missing translate events for Mac does affect our training. The amount of logs coming from MacOS users is small, so it does not affect strongly our global model, but it does affect our metrics for Mac users, and models trained on MacOS users only. We eventually plan to roll out different models for each OS, and this would probably be a blocker for releasing a MacOS only model.

Since we're switching to MacView Mac issue *should* no longer be relevant. Flagging to Anthony as FYI for when we do use Translate event logs.