If I recall correctly, there maybe some flag size limit to Android Chrome? The field trial flag that Chrome Android is using on bot:
field-trials: ["853359fa-f7de5af1",
                "241fff6c-ca7d8d80",
                "1e528f0f-15305a2",
                "236d5d9e-347fc86a",
                "e036a196-3f4a17df",
                "77207729-3f4a17df",
                "4bf94d2-dae7879e",
                "2a33b90e-3f4a17df",
                "ba3f87da-92cc81ec",
                "3fbe5359-3f4a17df",
                "684d1cdf-51126808",
                "b22b3d54-3f4a17df",
                "93731dca-3f4a17df",
                "9e5c75f1-431fdfeb",
                "64cbdfc2-3f4a17df",
                "b7786474-d93a0620",
                "9e46b3e5-e47679e6",
                "2182dcae-69de46be",
                "4ea303a6-3f4a17df",
                "826d6cab-96e3ef3c",
                "8b4d89aa-3f4a17df",
                "db5efd10-88e62167",
                "dd139bd7-48702b26",
                "fe05be5f-4ad60575",
                "828a5926-d8f52f32"]

Flag Chrome is using:
command_line: "_ --disable-fre --no-default-browser-check --disable-features=AndroidSpellChecker,LazyParseCSS,DisplayPersistenceToggleInPermissionPrompts --user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2194.2 Safari/537.36 --ignore-certificate-errors --disable-default-apps --enable-remote-debugging --disable-external-intent-requests --proxy-server=socks://localhost:39510 --metrics-recording-only --enable-features=AlternateComponentUrls,AndroidAIAFetching,AndroidHistoryManager,AndroidSpellCheckerNonLowEnd,AutofillCreditCardLastUsedDateDisplay,AutofillCreditCardPopupLayout,AutofillCreditCardSigninPromo,AutofillProfileCleanup,BackgroundVideoTrackOptimization,ContextualSearchUrlActions,DocumentWriteEvaluator,DownloadsUi,EnableSyncClientToServerCompression,EnableSyncUSSDeviceInfo,ExpectCTReporting,ExpensiveBackgroundTimerThrottling,FasterLocationReload,HttpFormWarning,ImportantSitesInCBD,MidiManagerAndroid,MidiManagerDynamicInstantiation,NTPArticleSuggestions,NTPBookmarkSuggestions,ContentSuggestionsCategoryRanker,NTPPreferAmpUrls,NTPForeignSessionsSuggestions,PrioritySupportedRequestsDelayable,NetworkSchedulerYielding,NetworkTimeServiceQuerying,NoStatePrefetch,NTPOfflinePages,OfflinePagesAsyncDownload,OfflinePagesSvelte,PassiveDocumentEventListeners,PassiveEventListenersDueToFling,PasswordMetadataFilling,ModalPermissionPrompts,PersistentHistograms,PhysicalWeb,PointerEvent,PreconnectMore,progress-bar-completion-resources-before-domContentLoaded,RafAlignedMouseInput,RafAlignedTouchInput,CanShowScoutOptIn,ThreatDomDetailsTagAttributes,ConsistentOmniboxGeolocation,SeccompSandboxAndroid,SpeculativeLaunchServiceWorker,SyncUSSAutocomplete,NegotiateTLS13,token-binding,TranslateRankerLogging,V8Future,VrShell,RequestUnbufferedDispatch,VsyncAlignedInput,WebRTC-EnableWebRtcEcdsa,use-new-media-cache --force-fieldtrials=AlternateComponentUrls/AlternateComponentUrls/AndroidAIAFetching/Enabled/AndroidHistoryManager/Enabled/AndroidSpellChecker/Enabled_NonLowEnd/AsyncDns/AsyncDnsA/AutodetectEncoding/Enabled/AutofillClassifier/Enabled/AutofillCreditCardDropdownVariations/LastUsedDateAndExpDate_IncreaseDropdownItemHeight_Experiment/AutofillCreditCardSigninPromo/EnabledThree/AutofillFieldMetadata/Enabled/AutofillProfileCleanup/Enabled/AutofillProfileOrderByFrecency/EnabledLimitTo3/BackgroundVideoOptimizations/BackgroundOptimizationEnabled5sOrLess/BrowserScheduler/Default/CSSExternalScanner/Enabled_ScanAndPreload/ChromotingQUIC/Enabled/ContextualSearch/Expected/DataCompressionProxyDevRollout/Control/DataCompressionProxyLoFi/Enabled/DataCompressionProxyPromoVisibility/Enabled/DataReductionProxyFREPromo/Enabled_Warmup/DataReductionProxyServerExperiments/Brotli/DataReductionProxyUseQuic/Enabled/DisallowFetchForDocWrittenScriptsInMainFrame/DocumentWriteEvaluatorGroup/DownloadsUi/Enabled/EnableMediaThreadForMediaPlayback/Enabled/EnableSyncClientToServerCompression/Enabled/EnableSyncUSSDeviceInfo/Enabled/ExpectCTReporting/ExpectCTReportingEnabled/ExpensiveBackgroundTimerThrottling/Enabled_30sMaxThrottilingDelay/FasterLocationReload/Enabled/HttpFormWarning/Enabled/ImportantSitesInCBD/DialogEnabled/InstantApps/InstantAppsEnabled/InvalidationsGCMUpstream/Enabled/LazyParseCSS/Control/LimitSyncPassphrasePrompt/Enabled/LowPriorityIFrames/Enabled/MarkNonSecureAs/show-non-secure-passwords-cc-ui/MidiManagerAndroid/Enabled/MidiManagerDynamicInstantiation/Enabled/MojoChannel/Enabled/NTPArticleSuggestions/EnabledSoftFetches/NTPBookmarkSuggestions/Phase2Enabled3wWithDesktop/NTPPersonalizedSectionRanking/Enabled-ClickBasedRanker/NTPPopularSites/EnabledFull/NTPPreferAmpUrls/Enabled/NTPRecentForeignTabs/Enabled-ThreeHour/NTPSnippets/ContentSuggestionsBackend-FCS/NetDelayableH2AndQuicRequests/Enabled/NetProxyPreconnects/Enabled/NetworkQualityEstimator/Enabled/NetworkSchedulerYielding/Enabled/NetworkTimeQueries/NetworkTimeQueriesEnabledBackgroundAndOnDemand/NoCreditCardAbort/Disabled/NoStatePrefetchValidation/Enabled/OfferUploadCreditCards/Enabled/OfflinePages/EnabledAsBookmarks/OfflinePagesAsyncLoading/Enabled/OmniboxBundledExperimentV1/Beta_Android_PhysWeb_QuerySuggest_Experiment/PageRevisitInstrumentation/Enabled/PassiveDocumentEventListeners/Enabled/PassiveEventListenersDueToFling/Enabled/PasswordBranding/SmartLockBrandingSavePromptOnly/PasswordManagerSettingsMigration/Enable/PasswordMetadataFilling/Enabled/PermissionPromptUIAndroid/ModalGestureEnabled/PersistentHistograms/EnabledOnDiskNoStability3/PhysicalWeb/Enabled/PointerEvent/Enabled/Precache/EnabledCGRUV20b/PreconnectMore/Enabled/ProgressBarAnimationAndroid/SmoothIndeterminateAnimation/ProgressBarCompletion/progress-bar-completion-resources-before-domContentLoaded/QUIC/Enabled/RafAlignedMouseInput/Enabled/RafAlignedTouchInput/Enabled/ReaderModeUI/AdaBoost/ReaderModeUIFeedback/Show/ReportCertificateErrors/ShowAndPossiblySend/SafeBrowsingAndroid/Enabled/SafeBrowsingScoutTransitionStudy/CanShowScoutOptInGroup2/SafeBrowsingThreatDomDetailsTagAttributes/Enabled/SafeBrowsingUseLocalBlacklist/Use3PAPI/SchedulerExpensiveTaskBlocking/Enabled/SdchPersistence/Enabled/SearchGeolocationConsistency/EnableNewShow3Each1Day/SeccompSandboxAndroid/SandboxEnabled/SpeculativeLaunchServiceWorker/Enabled/StrictSecureCookies/Enabled/SyncUSSAutocomplete/Enabled/TLS13Negotiation/Enabled/TabManagementExperiment/Elderberry/TabSyncByRecency/Enabled/TokenBinding/TokenBinding/TranslateRankerLogging/TranslateRankerLogging/TranslateServerStudy/SmartRendering/UMA_EnableCellularLogUpload/Enabled_wo_sampling/UnifiedMediaPipelineTrial/Enabled/UpdateMenuItem/EnabledWithNewerVersionAvailableSummary/V8CacheStrategiesForCacheStorage/default/V8Ignition/Future/VrShell/Enabled/VsyncAlignedInput/Enable/WebBluetoothBlocklist/TestGroup/WebFontsInterventionV2/Enabled-slow2g/WebRTC-EnableWebRtcEcdsa/Enabled/WebRTC-LocalIPPermissionCheck/Enabled/use-new-media-cache/Enabled --enable-gpu-benchmarking --enable-net-benchmarking --disable-component-extensions-with-background-pages --disable-background-networking --disable-search-geolocation-disclosure --no-first-run --touch-events=enabled --force-fieldtrial-params=AutofillCreditCardDropdownVariations.LastUsedDateAndExpDate_IncreaseDropdownItemHeight_Experiment:dropdown_item_height/56/show_expiration_date/true,AutofillCreditCardSigninPromo.EnabledThree:impression_limit/3,AutofillProfileOrderByFrecency.EnabledLimitTo3:limit/3,BackgroundVideoOptimizations.BackgroundOptimizationEnabled5sOrLess:max_keyframe_distance_ms/5000,BrowserScheduler.Default:RedirectSequencedWorkerPools/true,CSSExternalScanner.Enabled_ScanAndPreload:cssExternalScannerNoPreload/false/cssExternalScannerPreload/true,DataCompressionProxyLoFi.Enabled:hysteresis_period_seconds/120/rtt_msec/2000,DataReductionProxyFREPromo.Enabled_Warmup:enable_warmup/true,DataReductionProxyServerExperiments.Brotli:exp/allow_brotli,DisallowFetchForDocWrittenScriptsInMainFrame.DocumentWriteEvaluatorGroup:disallowFetchForDocWrittenScriptsInMainFrame/false/disallowFetchForDocWrittenScriptsInMainFrameOnSlowConnections/false,ExpensiveBackgroundTimerThrottling.Enabled_30sMaxThrottilingDelay:cpu_budget/0%2E01/initial_budget/1%2E0/max_budget/3%2E0/max_delay/-1,LowPriorityIFrames.Enabled:lowPriorityIframes/true,NTPArticleSuggestions.EnabledSoftFetches:fetching_interval_hours-wifi-active_ntp_user/8/scheduler_trigger_types/persistent_scheduler_wake_up%2Cntp_opened/send_top_languages/true/soft_fetching_interval_hours-active-active_ntp_user/2/soft_fetching_interval_hours-active-active_suggestions_consumer/1/soft_fetching_interval_hours-active-rare_ntp_user/4,NTPBookmarkSuggestions.Phase2Enabled3wWithDesktop:bookmarks_consider_desktop_visits/true/bookmarks_max_age_in_days/21,NTPPersonalizedSectionRanking.Enabled-ClickBasedRanker:category_ranker/click_based,NTPRecentForeignTabs.Enabled-ThreeHour:max_foreign_tabs_age_in_minutes/180,NetProxyPreconnects.Enabled:restrict_to_one_preconnect_for_proxies/true,NetworkQualityEstimator.Enabled:2G%2EDefaultMedianKbps/81/2G%2EDefaultMedianRTTMsec/1337/3G%2EDefaultMedianKbps/658/3G%2EDefaultMedianRTTMsec/297/4G%2EDefaultMedianKbps/1756/4G%2EDefaultMedianRTTMsec/159/Bluetooth%2EDefaultMedianKbps/449/Bluetooth%2EDefaultMedianRTTMsec/135/Ethernet%2EDefaultMedianKbps/3263/Ethernet%2EDefaultMedianRTTMsec/106/HalfLifeSeconds/60/None%2EDefaultMedianKbps/567/None%2EDefaultMedianRTTMsec/272/Slow2G%2EThresholdMedianHttpRTTMsec/2000/Unknown%2EDefaultMedianKbps/1916/Unknown%2EDefaultMedianRTTMsec/121/WiFi%2EDefaultMedianKbps/2736/WiFi%2EDefaultMedianRTTMsec/168/correlation_logging_probability/0%2E001,NetworkTimeQueries.NetworkTimeQueriesEnabledBackgroundAndOnDemand:FetchBehavior/background-and-on-demand,NoStatePrefetchValidation.Enabled:mode/no_state_prefetch,OmniboxBundledExperimentV1.Beta_Android_PhysWeb_QuerySuggest_Experiment:PhysicalWebAfterTyping/true/PhysicalWebZeroSuggest/true,PermissionPromptUIAndroid.ModalGestureEnabled:require_gesture/true,PersistentHistograms.EnabledOnDiskNoStability3:send_unreported_metrics/no/storage/MappedFile,Precache.EnabledCGRUV20b:config_url/https%3A%2F%2Fwww%2Egstatic%2Ecom%2Fchrome%2Fwifiprefetch%2Fprecache_config_g20,ProgressBarAnimationAndroid.SmoothIndeterminateAnimation:progress-bar-animation/smooth-indeterminate,ReportCertificateErrors.ShowAndPossiblySend:sendingThreshold/1%2E0,SafeBrowsingAndroid.Enabled:enabled/true/types_to_check/0%2C1%2C3%2C6%2C7%2C8%2C9%2C10%2C11%2C13%2C14%2C15,SafeBrowsingThreatDomDetailsTagAttributes.Enabled:tag_attribute_csv/div%2Cid%2Ciframe%2Cid,SafeBrowsingUseLocalBlacklist.Use3PAPI:check_local_blacklist/false,TranslateServerStudy.SmartRendering:server_params/smrd,UMA_EnableCellularLogUpload.Enabled_wo_sampling:Enabled/true/Optimize/true/Sample_Probability/100/Uma_Quota/204800/Uma_Ratio/0%2E05,UpdateMenuItem.EnabledWithNewerVersionAvailableSummary:enable_update_badge/true/enable_update_menu_item/true/show_summary/true,WebBluetoothBlocklist.TestGroup:blocklist_additions/00060000%3Ae%2Cfffd%3Ae%2Ced5f25a4%3Ae --use-mobile-user-agent --top-controls-show-threshold=0.5 --top-controls-hide-threshold=0.5 --use-mobile-user-agent --enable-pinch --enable-viewport --enable-overlay-scrollbar --validate-input-event-stream --enable-longpress-drag-selection --touch-selection-strategy=direction --disable-gpu-process-crash-limit --main-frame-resizes-are-orientation-changes --disable-composited-antialiasing --ui-prioritize-in-gpu-process --profiler-timing=0 --prerender-from-omnibox=enabled --enable-dom-distiller --flag-switches-begin --flag-switches-end",


(from clicking "M" in the top right of https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/trace-file-id_2-2017-03-27_06-24-28-75457.html)

Those hashes decode to:

AsyncDns-AsyncDnsA,
BrowserScheduler-Default,
CSSExternalScanner-Enabled_ScanAndPreload,
ContextualSearch-Expected,
DataCompressionProxyLoFi-Enabled,
DataCompressionProxyPromoVisibility-Enabled,
DataReductionProxyServerExperiments-dae7879e,
DataReductionProxyUseQuic-Enabled,
DisallowFetchForDocWrittenScriptsInMainFrame-DocumentWriteEvaluatorGroup,
LowPriorityIFrames-Enabled,
MarkNonSecureAs-show-non-secure-passwords-cc-ui,
NetProxyPreconnects-Enabled,
NetworkQualityEstimator-Enabled,
OmniboxBundledExperimentV1-Beta_Android_PhysWeb_QuerySuggest_Experiment,
PageRevisitInstrumentation-Enabled,
PasswordManagerSettingsMigration-Enable,
Precache-EnabledCGRUV20b,
ProgressBarAnimationAndroid-SmoothIndeterminateAnimation,
QUIC-Enabled,
ReaderModeUI-AdaBoost,
SafeBrowsingAndroid-Enabled,
TabManagementExperiment-Elderberry,
UpdateMenuItem-EnabledWithNewerVersionAvailableSummary,
V8CacheStrategiesForCacheStorage-default,
WebFontsInterventionV2-Enabled-slow2g


So it seems it is picking up some things - but e.g. PersistentHistograms is not in the list.

When did the change land? It's possible that perf bot are staying behind.

The two linked bugs are from a while ago... only got around to following up on them now.

But in both cases, a client default change landed - that should have no effect due to field trial testing config.

For example, from  crbug.com/694565 , it identified this CL:

Suspected Commit
  Author : bcwhite
  Commit : 8cf7e79b5bb95b8108861d8d7b910afdbc1db94d
  Date   : Thu Feb 16 20:44:17 2017
  Subject: Move to MappedFile as default for persistent histogra

Which changed client default. But at the time, the field trial testing config had an entry for PersistentHistograms thus:

    "PersistentHistograms": [
        {
            "platforms": [
                "android",
                "chromeos",
                "ios",
                "linux",
                "mac",
                "win"
            ],
            "experiments": [
                {
                    "name": "EnabledInMemory",
                    "params": {
                        "storage": "LocalMemory"
                    },
                    "enable_features": [
                        "PersistentHistograms"
                    ]
                },
                {
                    "name": "EnabledOnDisk",
                    "params": {
                        "storage": "MappedFile"
                    },
                    "enable_features": [
                        "PersistentHistograms"
                    ]
                }
            ]
        }
    ],

https://chromium.googlesource.com/chromium/src/+/8cf7e79b5bb95b8108861d8d7b910afdbc1db94d/testing/variations/fieldtrial_testing_config.json

That entry specified "storage" param as "LocalMemory" meaning the client change to treat empty string value of that param as defaulting to MappedFile *shouldn't* have had any effect on the bots if they were using that config.

Ned, do you have similar logs you posted above from historical runs? e.g. from this run:

chromium@451093      19.3348 +- 0.670567      6      bad       <--

(Or perhaps if old logs are lost, we could re-trigger the bisect to get the logs again?)

Seems to not be specific to Android. Using above command line from Ned I was able to reproduce the issue locally on my Mac.

So it's something related to parsing the configuration from the commandline. Will debug.

(Ned: You can ignore my request for more info above, looks like I have all I need to debug this now.)

Ok, this is caused by missing support in fieldtrial_util.py for producing associations between an --enable-feature param and its associated field trial. Which means any params being queried by feature (rather than by trial) wouldn't have been getting picked up through the testing config. :(

Will prepare a fix.

This does unfortunately mean that when that fix lands, we might seed perf benchmarks move as a result of params that were unfortunately not receiving the proper test coverage before.

Thanks for digging into this!

Things we normally do to help with the perf alerts:
1) Add a line in the CL description: PERF SHEERIFFS: This change is expected to...
2) Send a note to perf-sheriffs@chromium.org after landing

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d47c7180be6c7531200d034d36c470fb5980f0e4

commit d47c7180be6c7531200d034d36c470fb5980f0e4
Author: asvitkine <asvitkine@chromium.org>
Date: Wed Mar 29 19:44:58 2017

Fix perfbotnot feature param setting from trial testing config.

Previously, params that were queried via their associated features
would not get correctly picked by perfbots using the field trial
testing config.

This was caused by the generated command-line not using the
--enable-features=FeatureName<TrialName syntax. This syntax is
needed in order to estable the link between a feature and its
associated trial, so that when a field-trial param is looked up
by feature, it's able to find it from the associated trial.

Adds quotations to these params since "<" characters would
otherwise get interpreted by some shells.

PERF SHERIFFS: This CL may cause perf bots to move - if so, it
means that one of the entries in the field trial testing config
that uses both params and features is causing the change but was
previously not being correctly tested. If so, we will need to
narrow down which one.

BUG= 706414 

Review-Url: https://codereview.chromium.org/2778313003
Cr-Commit-Position: refs/heads/master@{#460498}

[modify] https://crrev.com/d47c7180be6c7531200d034d36c470fb5980f0e4/tools/variations/fieldtrial_util.py
[modify] https://crrev.com/d47c7180be6c7531200d034d36c470fb5980f0e4/tools/variations/fieldtrial_util_unittest.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e376f4717780e3248c359ea4fd5e47976130cc0d

commit e376f4717780e3248c359ea4fd5e47976130cc0d
Author: asvitkine <asvitkine@chromium.org>
Date: Thu Mar 30 15:58:09 2017

Don't add quotes for testing config feature command line flags.

I added these in https://codereview.chromium.org/2778313003/, but
turns out perfbot infra doesn't use the shell for the command line
used, so they're not necessary and would in fact cause problems as
the starts/ends of the feature lists would have leading and trailing
quotes and would not be interpreted correctly by Chrome.

PERF SHERIFFS: This CL may move metrics as it fixes a bug in the
evaluation of field trial testing config.

BUG= 706414 

Review-Url: https://codereview.chromium.org/2789583002
Cr-Commit-Position: refs/heads/master@{#460779}

[modify] https://crrev.com/e376f4717780e3248c359ea4fd5e47976130cc0d/tools/variations/fieldtrial_util.py
[modify] https://crrev.com/e376f4717780e3248c359ea4fd5e47976130cc0d/tools/variations/fieldtrial_util_unittest.py

I think we should also add some unittest in Telemetry that verify the field trial of browser under Telemetry is matching what we expect.

Example test start up a browser, collect the trace & test the trace model in Telemetry:  https://cs.chromium.org/chromium/src/tools/perf/measurements/clock_domain_test.py?rcl=79580105235657014da58d4a36ca170fd4e63ff9&l=20

Field trials only get activated in the browser when they're used. If we don't hit a codepath of a trial, it won't show up as active in chrome://version.

Of course, we could still verify that any active trials match the corresponding command-line that was passed in. Suggest filing a bug if this is something that's interesting - although I'm not sure we have cycles on metrics team to pursue this.

Marking this bug as Fixed. There's bug about a memory regression as a result of this where some further discussion is happening: https://bugs.chromium.org/p/chromium/issues/detail?id=706941

Can you file the bug about adding Telemetry test & provide guidance on testing? Benchmarking team can work on the implementation.

Done - filed crbug.com/707310.