Unable to view currently active field trials
Reported by
dec...@blizzard.com,
Oct 6 2016
|
||||||
Issue descriptionUserAgent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36 Steps to reproduce the problem: There's not a clear way to list which field trials are active in Chrome. This causes problems in the following scenarios: - when developers are trying to diagnose why different instances of Chrome are behaving differently, - when developers are trying to gather info from end users to reproduce issues reported in the wild, - when trying to set up a test Chrome instance that reflects the most common state of each trial in the wild, - when trying to determine why an experiment under chrome://flags appears to have no effect (for trials linked to experiments), - when trying to ensure consistent behavior between instances, versions, and launches of chrome (for SESSION_RANDOM variations) for QA purposes, - when trying to learn what the command line switch is to configure a trial that is suspected to be causing problems (requires looking through chromium source today) What is the expected behavior? What went wrong? I would expect a page like chrome://fieldtrials to list all trials and the current state, and chrome://flags to reflect the state of field trials (ideally indicating that the state of a flag is really (trial || flag), and what the defaults are). Exposing the probability of the trials would be ideal as well. I would expect chrome://fieldtrials to have a checkbox to disable field trials (like --disable-field-trial-config). It didn't look like field trial info was exposed via platform APIs for extension use, otherwise that could be a stop-gap option. Did this work before? N/A Chrome version: 53.0.2785.143 Channel: stable OS Version: 10.0 Flash Version: Shockwave Flash 23.0 r0
,
Oct 6 2016
chrome://version displays all currently active field trials (under the name "Variations:"). Apologies that these are hashes. We've having a discussion on e-mail among us metrics folks whether we should leave them unhashed. (There are plusses and minuses to both; e.g., having plaintext names might bias field trial results for people who look at chrome://version.) If you want to disable all field trials, you can use the flag --enable-benchmarking on your developer build. I agree that there is currently no friendly way to interact with and investigate field trials.
,
Oct 6 2016
Also, Google Chrome developers have the ability to map those hashes from chrome://version to names - so if you file a bug that you think corresponds to a specific hash - or that is different between two Chrome installs (that still repros in Incognito or in a new profile) - then we can help debug on the crbug.
,
Oct 6 2016
Understood that the variant hashes offers *some* info, but full transparency could save a lot time for folks interested in Chrome bugs and those just interested in bugs in their webapps. The quality of Chromium bug reports could potentially improve. Another way to look at it is that someone could take the time to write a tool that scrapes experiment info from chromium source, and attempts to force each of them on command line to identify which variants are actually active. So why not expose details that are already discernible with some effort? If you're worried about the exposure of trial info affecting results, maybe you could mark results as tainted if the user has actually opened the field trials list since the last randomization. That'd also prove whether or not exposing field trial info actually affects results. Maybe do a field trial on determining whether or not to expose field trials? :-)
,
Oct 6 2016
FWIW, this is based on a real experience deckel@ dealt with in issue 580770 . He did a lot of hard work to track down some problems for us, and it was made more difficult because he didn't have insight into who was in what trial. I do see the pluses and minuses, though, so I'm not personally voting either way. I do really like the idea to field trial to determine whether or not to expose field trials.
,
Oct 9 2017
Issue has not been modified or commented on in the last 365 days, please re-open or file a new bug if this is still an issue. For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Oct 9 2017
Still an issue. Needs re-opening. Not clear how to do that via this UI.
,
Oct 9 2017
Marking this as 'Available' as I think it's still a valid feature request. This was also somewhat recently discussed on chromium-dev [1]. We are not currently prioritizing work on this issue, as it seems like a relatively high effort-to-reward ratio... but if anyone wants to pick up work on this, I think these are the necessary ingredients: - We want an explicit signal that a user is interested in viewing which field trials are currently active. Visiting chrome://version is not sufficient, as there are other reasons to visit that page. We could perhaps add a link on chrome://version that deciphers the field trials, or we could implement a separate page (e.g. chrome://fieldtrials) or tool that deciphers the field trials. This is motivated primarily by the second requirement: - We want to annotate uploads from users who have viewed their own field trial state, to exclude them from default analysis in those trials whose states they have viewed. This is to avoid (potential) bias from users changing their behavior based on knowing what experiments are active. Note: We do not want to change the behavior that these users experience simply because they viewed their own state; we just want to make it easy to exclude their data from analysis. - We want to implement a monitoring system to validate that no experiment has a nontrivial percentage of users being excluded from it due to viewing their own field trial state. If a single experiment has an unusually large number of users being excluded due to this reason, that suggests that something might be going wrong with the experiment. If many experiments simultaneously see an uptick in users being excluded due to this reason, then there might be an external factor at work – e.g. a popular blog post – that we might need to compensate for in some way. - Optionally, it would be nice to make it easy for users who do view their field trial state to adjust it, e.g. by cross-linking to the appropriate chrome:flags. If we implement this, we'd again want to add some monitoring logic to understand how often this functionality is being used. FWIW, I do not think that it's sufficient to run a field trial to understand whether we can just always show experiment state. It's likely that for many experiments, there is no harm to users seeing their own state, in terms of introducing bias. Rather, I'd expect that there are a handful of experiments where user behavior would change in response to seeing their state; and that this would be a minority of experiments. We could, in theory, dedicate a lot of resources to trying to understand precisely what sorts of experiments are susceptible, and treat them in some special way... but I don't think this is a promising approach, compared to my suggestion above. If anyone outside of the Chrome Metrics team wants to take on this bug, please reach out to one of us first. What I've written above are my personal thoughts; and while I think the broader team would likely agree with the general spirit, there might be some refinements that we'd want to make based on a team discussion. [1] https://groups.google.com/a/chromium.org/forum/#!msg/chromium-dev/328Zta3bDKY/Z3Qnr76QAwAJ
,
Oct 10 2017
,
Oct 10
This issue has been Available for over a year. If it's no longer important or seems unlikely to be fixed, please consider closing it out. If it is important, please re-triage the issue. Sorry for the inconvenience if the bug really should have been left as Available. For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Oct 15
Back to available, but Metrics team can chime in with actionable next steps at this point. |
||||||
►
Sign in to add a comment |
||||||
Comment 1 by mmenke@chromium.org
, Oct 6 2016