Thanks for filing. I have a sneaking suspicion the failures might be due to the browser hitting DCHECKs during the course of the tests. (I think telemetry exercises parts of the browser that the sanity test doesn't touch?) So there's a chance it's not as simple as bumping timeouts :(

I recall the suite being somewhat green until recently, so something must have changed. I'll try to look into it more in the next couple days.

That's definitely a possibility. I'll run unittests with DCHECKS and see what happens.

I ran telemetry_unittests with DCHECKS on.

1032 tests passed in 3977.4s, 136 skipped, 0 failures. (3977.4s is 66 min)

There are no crashing stacks in /var/log/chrome/

Sorry for the delay, finally got around to looking into this. There's a couple problems, one for each test suite:

telemetry_perf_unittests:
Mostly passes, but occasionally flakes due to two different errors:

- rsync not found in VM:
https://chromium-swarm.appspot.com/task?id=3fd6aa34f0883010&refresh=10&show_raw=1&wide_logs=true
https://chromium-swarm.appspot.com/task?id=3fddcac0dce36c10&refresh=10&show_raw=1&wide_logs=true

- stuck at oobe ("Timed out going through login screen. OOBE not dismissed."):
https://chromium-swarm.appspot.com/task?id=3fd964829f156b10&refresh=10&show_raw=1&wide_logs=true
https://chromium-swarm.appspot.com/task?id=3fd49942a7116d10&refresh=10&show_raw=1&wide_logs=true

I'll try to get a local repro going for either of those... maybe we need to add a retry somewhere?


telemetry_unittests:
It fails consistently on the bot due to the "--extra-browser-args=--disable-gpu" argument. (See below for crash trace.) The arg was added in https://chromium-review.googlesource.com/c/chromium/src/+/1167894. The CL's descriptions sounds like it would be applicable here (we're in VMs after all), but I can't say for sure. I'll work on removing that arg for this bot so it can go back to passing like it does locally.

[19800:19800:0910/154725.752978:FATAL:gpu_process_transport_factory.cc(453)] Check failed: use_gpu_compositing. 
#0 0x58854a4ba26c base::debug::StackTrace::StackTrace()
#1 0x58854a4391a0 logging::LogMessage::~LogMessage()
#2 0x588548f01ccc content::GpuProcessTransportFactory::EstablishedGpuChannel()
#3 0x588548f0152d content::GpuProcessTransportFactory::CreateLayerTreeFrameSink()
#4 0x58854b87708e ui::Compositor::RequestNewLayerTreeFrameSink()
#5 0x58854a4d0379 base::debug::TaskAnnotator::RunTask()
#6 0x58854a43f8ce base::MessageLoop::RunTask()
#7 0x58854a43fc92 base::MessageLoop::DoWork()
#8 0x58854a4cce19 base::MessagePumpLibevent::Run()
#9 0x58854a45f755 base::RunLoop::Run()
#10 0x58854a11390a ChromeBrowserMainParts::MainMessageLoopRun()
#11 0x588548a822b4 content::BrowserMainLoop::RunMainMessageLoopParts()
#12 0x588548a84d52 content::BrowserMainRunnerImpl::Run()
#13 0x588548a7e888 content::BrowserMain()
#14 0x58854a103908 content::ContentMainRunnerImpl::Run()
#15 0x58854a10a392 service_manager::Main()
#16 0x58854a101c41 content::ContentMain()
#17 0x588547e6b81f ChromeMain
#18 0x7aeb7dc01736 __libc_start_main
#19 0x588547e6b649 _start

ChromeOS might, like Android, require a GPU process to be present. Perhaps --use-gl=swiftshader on these bots would work?

The tests appear to pass with "--use-gl=swiftshader", but the logs are filled with quite a bit of "ERROR :GL_INVALID_FRAMEBUFFER_OPERATION : glDrawElements: framebuffer incomplete". I'll take that as generally-not-a-good-sign and stick to just removing the arg altogether unless someone tells me otherwise.

As for the missing "rsync" binary, I managed to find a bot exhibiting the same failure, and its "/usr/local/" dir (under which rsync is usually found) is suspiciously empty:

$ ls -al /usr/local/
total 12
drwxr-xr-x  3 root    root    4096 Sep 10 11:32 .
drwxr-xr-x 10 root    root    4096 Sep  6 18:52 ..
drwxr-xr-x 11 chronos chronos 4096 Sep 10 13:40 vm_test

Something happened in there... something bad. I'll try to find out what.

Re #7: Looks like you were right about cros being similar to android:
https://codesearch.chromium.org/chromium/src/content/test/gpu/gpu_tests/gpu_integration_test.py?rcl=6e18cb2b9fff68df28bfc270a6fe27153f982840&l=71

Uploaded https://chromium-review.googlesource.com/c/chromium/src/+/1217913 to remove it.

Check out this failing run:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel/19833

03:57:08: INFO: Copying Chrome to /opt/google/chrome on device...
rsync is not found on the device.
Run dev_install on the device to get rsync installed
cros_run_vm_test: Unhandled exception:
Traceback (most recent call last):
  File "/b/s/w/ir/third_party/chromite/bin/cros_run_vm_test", line 169, in <module>
    DoMain()
  File "/b/s/w/ir/third_party/chromite/bin/cros_run_vm_test", line 165, in DoMain
    commandline.ScriptWrapperMain(FindTarget)
  File "/b/s/w/ir/third_party/chromite/lib/commandline.py", line 912, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/b/s/w/ir/third_party/chromite/scripts/cros_run_vm_test.py", line 407, in main
    return VMTest(ParseCommandLine(argv)).Run()
  File "/b/s/w/ir/third_party/chromite/scripts/cros_run_vm_test.py", line 72, in Run
    self._Deploy()
  File "/b/s/w/ir/third_party/chromite/scripts/cros_run_vm_test.py", line 124, in _Deploy
    cros_build_lib.RunCommand(deploy_cmd)
  File "/b/s/w/ir/third_party/chromite/lib/cros_build_lib.py", line 647, in RunCommand
    raise RunCommandError(msg, cmd_result)
chromite.lib.cros_build_lib.RunCommandError: return code: 1; command: deploy_chrome --force --build-dir out_amd64-generic/Release --process-timeout 180 --to localhost --port 9222 --board amd64-generic --cache-dir /b/s/w/ir/build/cros_cache
cmd=['deploy_chrome', '--force', '--build-dir', 'out_amd64-generic/Release', '--process-timeout', '180', '--to', 'localhost', '--port', '9222', '--board', 'amd64-generic', '--cache-dir', '/b/s/w/ir/build/cros_cache']

Nice find... the bot in that build only ran 16 tests before it was recycled:
https://chromium-swarm.appspot.com/bot?id=gce-trusty-02248837-us-west1-c-b5zt

Assuming the VM image was healthy when we first downloaded it (I'm 99% sure it is), then one of those tests likely got it into the broken state. I'll start looking through them.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/865856edea0838da41c48e89ae363f0c99c7e47a

commit 865856edea0838da41c48e89ae363f0c99c7e47a
Author: Ben Pastene <bpastene@chromium.org>
Date: Tue Sep 11 16:06:35 2018

Don't disable GPU process when running telemetry_unittests on cros VMs.

That arg has been shown to cause the browser to crash on cros VMs.

Bug:  876494 
Change-Id: I217451ce65a78b62a947a8b2b77618875e83dd7f
Reviewed-on: https://chromium-review.googlesource.com/1217913
Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#590335}
[modify] https://crrev.com/865856edea0838da41c48e89ae363f0c99c7e47a/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/865856edea0838da41c48e89ae363f0c99c7e47a/testing/buildbot/test_suites.pyl

The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/57e0c4898edce7bbbe2fb13358b1606da1dbfd82

commit 57e0c4898edce7bbbe2fb13358b1606da1dbfd82
Author: Achuith Bhandarkar <achuith@chromium.org>
Date: Wed Sep 12 23:28:34 2018

Disable AndroidAppTest.testWebView on chromeos.

Bug= chromium:876494 
Test=This is the test.
TBR=nednguyen

Change-Id: I2ef7f3be7c3bccd10b5084a2bb37daa2d8eb10e1
Reviewed-on: https://chromium-review.googlesource.com/1222369
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Commit-Queue: Achuith Bhandarkar <achuith@chromium.org>

[modify] https://crrev.com/57e0c4898edce7bbbe2fb13358b1606da1dbfd82/telemetry/telemetry/internal/app/android_app_unittest.py

telemetry_unittests is back to flaking pretty aggressively on the fyi bot >.<
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel-vm-tests?limit=200

It's almost always due to a timeout (a shard times out at 60min, when it should take 10-20 min). Looks like the #1 cause is "Timed out going through login screen. Cryptohome not mounted. Browser didn't launch."

Managed to get a local repro, and the logs tell a similar story:
[2663:2663:0918/112700.738490:ERROR:object_proxy.cc(615)] Failed to call method: org.chromium.CryptohomeInterface.MountEx: object_path= /org/chromium/Cryptohome: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
[2663:2663:0918/112700.739047:ERROR:device_event_log_impl.cc(159)] [11:27:00.738] Login: cryptohome_util.cc:25 Cryptohome call failed with empty reply.
[2663:2663:0918/112700.739165:ERROR:device_event_log_impl.cc(159)] [11:27:00.739] Login: cryptohome_authenticator.cc:147 MountEx failed. Error: 1
[2663:2663:0918/112700.739283:ERROR:device_event_log_impl.cc(159)] [11:27:00.739] Login: cryptohome_authenticator.cc:971 Cryptohome failure: state(AuthState)=2, code(cryptohome::MountError)=1
[2663:2663:0918/112700.739951:ERROR:device_event_log_impl.cc(159)] [11:27:00.739] Login: cryptohome_authenticator.cc:745 Login failed: Could not mount cryptohome.
[2663:2663:0918/112700.740141:ERROR:login_performer.cc(63)] Login failure, reason=1, error.state=0

@achuith: WDYT about adding wait-for-cryptohome logic to the vm launcher? If it doesn't start up soon enough, we tear the VM down and try again. Prob somewhere around here:
https://codesearch.chromium.org/chromium/src/third_party/chromite/scripts/cros_vm.py?rcl=151945a7459ce7d5cecda7b05fb8dff7f8a7b1a2&l=397

That said, I don't actually know what cryptohome is, but telemetry sure likes to wait for it to start up before running any tests.

Hmm, I looked at this one:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel-vm-tests/5224

And there are about 50 failures, only 2 of which are the cryptohome ones.

Before we get a browser window on chromeos, we log in as a fake user. Telemetry determines that the login has succeeded if 3 conditions are met - the oobe/login view is dismissed, the cryptohome (the encrypted user directory) is mounted, and a new browser process launches. The cryptohome mount is failing here for some reason. This is a pretty basic startup operation in dozens of autotests, including vm_sanity. 

https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel-vm-tests/5220

This run had 41 failures, 3 of which were the cryptohome problem:
telemetry.core.memory_cache_http_server_unittest.MemoryCacheHTTPServerTest.testAbsoluteAndRelativePathsYieldSameURL
telemetry.core.tracing_controller_unittest.StartupTracingTest.testCloseBrowserBeforeTracingIsStopped
telemetry.internal.actions.action_runner_unittest.ActionRunnerTest.testScrollToElement
telemetry.internal.actions.key_event_unittest.KeyPressActionTest.testTextEntry
telemetry.internal.actions.play_unittest.PlayActionTest.testPlayWithVideoSelector
telemetry.internal.actions.scroll_unittest.ScrollActionTest.testTouchScrollDistanceWhileZoomed
telemetry.internal.backends.chrome.crx_id_unittest.CrxIdUnittest.testPackedHashAppId
telemetry.internal.backends.chrome.possible_desktop_browser_unittest.PossibleDesktopBrowserTest.testCopyProfileFilesRecursive
telemetry.internal.backends.chrome_inspector.inspector_console_unittest.TabConsoleTest.testConsoleOutputStream
telemetry.internal.backends.chrome_inspector.tracing_backend_unittest.TracingBackendUnittest.testCollectTracingDataFromStreamNoContainer
telemetry.internal.browser.browser_unittest.CommandLineBrowserTest.testCommandLineOverriding
telemetry.internal.browser.extension_unittest.NonExistentExtensionTest.testExtensionBasic
telemetry.internal.browser.tab_unittest.TabTest.testActivateTab
telemetry.internal.platform.linux_based_platform_backend_unittest.LinuxBasedPlatformBackendTest.testGetSystemTotalPhysicalMemory
telemetry.internal.platform.tracing_controller_backend_unittest.TracingControllerBackendTest.testFlush
telemetry.internal.results.chart_json_output_formatter_unittest.ChartJsonTest.testAsChartDictPageSpecificValuesSamePageWithoutInteractionRecord
telemetry.internal.results.gtest_progress_reporter_unittest.GTestProgressReporterTest.testSingleFailedPage
telemetry.internal.results.json_3_output_formatter_unittest.Json3OutputFormatterTest.testAsDictWithTwoPages
telemetry.internal.results.page_test_results_unittest.PageTestResultsTest.testAddSharedDiagnostic
telemetry.internal.results.page_test_results_unittest.PageTestResultsTest.testGetSuccessfulPageValuesMergedNoFailures
telemetry.internal.results.page_test_results_unittest.PageTestResultsTest.testTypeChange
telemetry.internal.snap_page_util_unittest.SnapPageTest.testSnappingToInvalidSnapshotPath
telemetry.internal.story_runner_unittest.StoryRunnerTest.testPagesetRepeat
telemetry.internal.story_runner_unittest.StoryRunnerTest.testRunBenchmark_AddsOwners_NoComponent
telemetry.internal.story_runner_unittest.StoryRunnerTest.testRunStoryAndProcessErrorIfNeeded_tryError_finallyException
telemetry.internal.story_runner_unittest.StoryRunnerTest.testRunStoryWithLongName
telemetry.internal.util.path_set_unittest.PathSetTest.testAdd
telemetry.page.page_run_end_to_end_unittest.ActualPageRunEndToEndTests.testOneTab
telemetry.page.page_unittest.TestPage.testDisplayUrlForFile
telemetry.testing.run_tests_unittest.RunTestsUnitTest.testDoesntHaveTabs
telemetry.testing.system_stub_unittest.CloudStorageTest.testOpenRestoresCorrectly
telemetry.util.image_util_unittest.ImageUtilTest.testWritePngWithoutPngSuffixThrows
telemetry.util.screenshot_unittest.ScreenshotUtilTests.testScreenShotTakenSupportedPlatform
telemetry.value.list_of_scalar_values_unittest.ValueTest.testListWithNoneReasonMustHaveNoneValue
telemetry.value.merge_values_unittest.MergeValueTest.testSamePageMergeNonstandardKeyFunc
telemetry.value.scalar_unittest.ValueTest.testScalarDifferentPageMerging
telemetry.value.summary_unittest.SummaryTest.testBasicSummary
telemetry.value.trace_unittest.ValueTest.testAsDictWhenTraceIsNotSerializedAndUploaded
telemetry.value.value_unittest.ValueTest.testDescriptionMustBeStringOrNone
telemetry.value.value_unittest.ValueTest.testImportantMustBeBool
telemetry.wpr.archive_info_unittest.WprArchiveInfoTest.testDownloadArchivesIfNeededWithStoryNames

I suspect the problem is not with the actual failing tests. The interesting thing is that all 40+ failures happened on the same shard. And as you note, the shard timeout of 60 min is quite a bit higher than the average shard execution time of 10-15 min. 

The cryptohome check succeeds on hundreds of tests on other shards. I suspect trying to make sense of individual test failures and their stacks will not solve the problem.

I suspect there's a problem with the VM - is it being shared by the shards somehow?

Looking again at:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel-vm-tests/5224

Failing shard:
https://chromium-swarm.appspot.com/task?id=4009d9bc2af1c310&refresh=10&show_raw=1

This seems suspicious:
  OSError: cryptohome-path failed: Warning: Identity file /b/s/w/ir/third_party/catapult/telemetry/third_party/chromite/ssh_keys/testing_rsa not accessible: No such file or directory.

Also these:
[22/99] telemetry.internal.backends.browser_backend_unittest.BrowserBackendLogsUploadingUnittest.testUploadingToCLoudStorage failed unexpectedly 0.0004s:
  Traceback (most recent call last):
    File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/browser_backend_unittest.py", line 17, in testUploadingToCLoudStorage
    File "/b/s/w/ir/.swarming_module/lib/python2.7/tempfile.py", line 475, in NamedTemporaryFile
    File "/b/s/w/ir/.swarming_module/lib/python2.7/tempfile.py", line 244, in _mkstemp_inner
  OSError: [Errno 2] No such file or directory: '/b/s/w/itnujslK/tmpn9VE2v'

[24/99] telemetry.internal.backends.chrome.desktop_browser_finder_unittest.FindSystemTest.testFindLocalAppData queuedsh: 0: getcwd() failed: No such file or directory

[33/99] telemetry.internal.browser.extension_unittest.ExtensionTest.testExtensionBasic failed unexpectedly 0.0010s:
  Traceback (most recent call last):
    File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/browser/extension_unittest.py", line 54, in testExtensionBasic
    File "/b/s/w/ir/.swarming_module/lib/python2.7/contextlib.py", line 17, in __enter__
    File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/browser/extension_unittest.py", line 45, in CreateBrowserWithExtension
    File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/browser/extension_unittest.py", line 25, in GetExtensionToLoad
    File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/browser/extension_to_load.py", line 19, in __init__
  ExtensionPathNonExistentException: Extension path not a directory /b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/testing/simple_extension

44/99] telemetry.internal.results.chart_json_output_formatter_unittest.ChartJsonTest.testAsChartDictWithTraceValuesThatHasTirLabel failed unexpectedly 0.0003s:
  Traceback (most recent call last):
    File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/results/chart_json_output_formatter_unittest.py", line 33, in setUp
    File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/results/chart_json_output_formatter_unittest.py", line 23, in _MakeStorySet
    File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/story/story_set.py", line 42, in __init__
  ValueError: Invalid directory path of base_dir: /b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/results

There seems to be something screwy with the filesystem on the shard - all these errors are directory/file errors. (I initially wondered if the deploy might be screwing something up, but I don't think this is the problem.) Is it possible to preserve the shard state to log in and inspect it?

> I suspect the problem is not with the actual failing tests.

Agreed. Like you point out, it's always confined to a single shard, and it's pretty much every test in that shard that fails. One VM gets spun-up per shard, so if that VM is busted in someway, all the tests in that shard will have a tough time. I think that's what we're seeing here.

As for the "/b/s/w/ir/third_party/catapult/telemetry/third_party/chromite/ssh_keys/testing_rsa not accessible" error, I think that's a red herring. There's an unfortunate race-condition in our tests when they reach the timeout (1 hour in this case): the test harness kills the test process then starts removing the test's files. If the test doesn't shut down in time, or ignores the kill signal, it'll keep running even while its files are disappearing from underneath it. Notice how the "missing-ssh-keys" error only shows up at the very end... I think that's a result of the race-condition.

I do still think the root cause is the OOBE/cryptohome error. All failing shards complain about it, and you're right that they only complain 3 times in each shard despite ~50 failing tests. But look how long those 3 tests take in https://chromium-swarm.appspot.com/task?id=4009d9bc2af1c310&refresh=10&show_raw=1&wide_logs=true

[17/99] telemetry.internal.actions.action_runner_unittest.ActionRunnerMeasureMemoryTest.testWithFailedDump failed unexpectedly 1225.8311s:
[18/99] telemetry.internal.actions.action_runner_unittest.ActionRunnerTest.testWaitForJavaScriptCondition_returnsValue failed unexpectedly 1533.6399s:
[19/99] telemetry.internal.actions.page_action_unittest.PageActionTest.testEvaluateCallbackWithElementWithConflictingParams failed unexpectedly 748.5388s:

That's about 58 minutes the shard spent on those three tests alone. It would have likely kept failing with the cryptohome errors, except it hit the 60min timeout shortly after, and started failing much faster with the missing ssh-keys error (due to the dumb race-condition above).

That said, the suite is now suddenly much less flakey:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel-vm-tests?limit=200

Past 15 or so runs have been green despite it failing consistently beforehand. What the heck happened....

The suite was green... and then suddenly started failing every run on the FYI bot:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel-vm-tests?limit=200

Given that we don't seem to see this on the amd64-generic-telemetry informational bot, this might be due to how we reuse the same boot images over and over. I'm willing to bet the copy-on-write option I'm adding is going to help a lot here:
https://chromium-review.googlesource.com/c/chromiumos/chromite/+/1239577

I'll let that land, enable it for the suite, then see if things improve.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/02984296a6429d35f9eb152239263b748ed182ad

commit 02984296a6429d35f9eb152239263b748ed182ad
Author: Ben Pastene <bpastene@chromium.org>
Date: Wed Oct 10 17:12:50 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f364f9bb7b6428f2eb145b261cc5c2d42c050631

commit f364f9bb7b6428f2eb145b261cc5c2d42c050631
Author: Ben Pastene <bpastene@chromium.org>
Date: Wed Oct 10 21:00:27 2018

Move telemetry_perf_unittests from cros VM FYI bot to CQ bot.

The suite's looking good on the fyi bot:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel-vm-tests?limit=200

Needs http://crrev.com/i/694912 before this can land.

Bug:  876494 
Change-Id: Id6907736bcd7d700865d47081f52c8ec53503091
Reviewed-on: https://chromium-review.googlesource.com/c/1272237
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>
Cr-Commit-Position: refs/heads/master@{#598497}
[modify] https://crrev.com/f364f9bb7b6428f2eb145b261cc5c2d42c050631/testing/buildbot/chromium.chromiumos.json
[modify] https://crrev.com/f364f9bb7b6428f2eb145b261cc5c2d42c050631/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/f364f9bb7b6428f2eb145b261cc5c2d42c050631/testing/buildbot/test_suites.pyl
[modify] https://crrev.com/f364f9bb7b6428f2eb145b261cc5c2d42c050631/testing/buildbot/waterfalls.pyl

telemetry_perf_unittests has been added to the CQ, and it looks like it's catching breakages now:
https://chromium-review.googlesource.com/c/chromium/src/+/1278287/1 yay!

As for telemetry_unittests, looks like it's back to perma-failing on the FYI bot
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel-vm-tests

Always timing out. And always only on shard #2. Hmm....

Looks like it's failing due to https://chromium-review.googlesource.com/c/chromium/src/+/1256570

The browser crashes during "telemetry.internal.browser.browser_unittest.TestBrowserCreation.testCreateBrowserTwice" w/ this trace:

[3717:3717:1012/140905.085909:FATAL:thread_restrictions.cc(32)] Check failed: !g_blocking_disallowed.Get().Get(). Function marked as blocking was called from a scope that disallows b
locking! If this task is running inside the TaskScheduler, it needs to have MayBlock() in its TaskTraits. Otherwise, consider making this blocking work asynchronous or, as a last res
ort, you may use ScopedAllowBlocking (see its documentation for best practices).
#0 0x58e28f1f6fef base::debug::StackTrace::StackTrace()
#1 0x58e28f1507eb logging::LogMessage::~LogMessage()
#2 0x58e28f1cc0f6 base::AssertBlockingAllowed()
#3 0x58e28f1fd0c9 base::PathExists()
#4 0x58e28f431bc8 ui::PathProvider()
#5 0x58e28f17972d base::PathService::Get()
#6 0x58e28f42e163 ui::ResourceBundle::GetLocaleFilePath()
#7 0x58e28f42e0c5 ui::ResourceBundle::LocaleDataPakExists()
#8 0x58e28f420e52 (anonymous namespace)::IsLocaleAvailable()
#9 0x58e28f42061c l10n_util::CheckAndResolveLocale()
#10 0x58e28d8dc1f3 chromeos::locale_util::IsNativeUILanguage()
#11 0x58e28da692ee chromeos::UserSessionManager::RespectLocalePreference()
#12 0x58e28da6da7f chromeos::UserSessionManager::RespectLocalePreferenceWrapper()
#13 0x58e28ed8c2c5 ProfileImpl::OnPrefsLoaded()
#14 0x58e28fffd18d PrefNotifierImpl::OnInitializationCompleted()
#15 0x58e28fff79e4 JsonPrefStore::FinalizeFileRead()
#16 0x58e28fff6519 JsonPrefStore::OnFileRead()
#17 0x58e28fff847f _ZN4base8internal12InvokeHelperILb1EvE8MakeItSoIRKM13JsonPrefStoreFvNSt3__110unique_ptrINS4_10ReadResultENS5_14default_deleteIS7_EEEEERKNS_7WeakPtrIS4_EEJSA_EEEvOT
_OT0_DpOT1_
#18 0x58e28fff88d6 base::internal::ReplyAdapter<>()
#19 0x58e28bc1a7d7 _ZN4base8internal7InvokerINS0_9BindStateIPFvNS_12OnceCallbackIFvbEEEPNSt3__110unique_ptrIbNS6_14default_deleteIbEEEEEJS5_NS0_12OwnedWrapperISA_EEEEEFvvEE7RunOnceEP
NS0_13BindStateBaseE
#20 0x58e28f1c7610 base::(anonymous namespace)::PostTaskAndReplyRelay::RunReply()
#21 0x58e28f1c76cb _ZN4base8internal7InvokerINS0_9BindStateIPFvNS_12_GLOBAL__N_121PostTaskAndReplyRelayEEJS4_EEEFvvEE7RunOnceEPNS0_13BindStateBaseE
#22 0x58e28f21c510 base::debug::TaskAnnotator::RunTask()
#23 0x58e28f159a7f base::MessageLoop::RunTask()
#24 0x58e28f159e52 base::MessageLoop::DoWork()
#25 0x58e28f218249 base::MessagePumpLibevent::Run()
#26 0x58e28f159551 base::MessageLoop::Run()
#27 0x58e28f1850a6 base::RunLoop::Run()
#28 0x58e28eca73ea ChromeBrowserMainParts::MainMessageLoopRun()
#29 0x58e28cace824 content::BrowserMainLoop::RunMainMessageLoopParts()
#30 0x58e28cad15c3 content::BrowserMainRunnerImpl::Run()
#31 0x58e28cacaa26 content::BrowserMain()
#32 0x58e28ec95150 content::ContentMainRunnerImpl::Run()
#33 0x58e28ec9cce9 service_manager::Main()
#34 0x58e28ec93221 content::ContentMain()
#35 0x58e28b877abf ChromeMain
#36 0x7d1f67a9f736 __libc_start_main
#37 0x58e28b8778e9 _start


Not user what we want to do here. Revert? Fix? Disable the test?

That's unfortunate. I'd say revert the change, except that it landed 4 days ago and it's unclear whether a revert will work.

voit@: can you own this bug? It's caused breakage on a ChromeOS bot, which is exactly the platform which your fix targeted. Thanks.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ca7c8b17ae5f868bc878c61e1d3018c9c9f08e13

commit ca7c8b17ae5f868bc878c61e1d3018c9c9f08e13
Author: Alexander Hendrich <hendrich@chromium.org>
Date: Sat Oct 13 09:52:08 2018

Revert "Updated AllowedUILocales to handle preferred languages."

This reverts commit 75d0995fa069994cee37b7ba3c940c6927e7d91b.

Reason for revert: Causes DCHECK crashes on sign-in

Original change's description:
> Updated AllowedUILocales to handle preferred languages.
> 
> Updated AllowedUILocales to limit not only the ui locales but also the
> available preferred languages. Renamed AllowedUILocales to AllowedLanguages to
> better indicate that.
> 
> Bug:  579660 ,  876494 ,  894406 
> Cq-Include-Trybots: luci.chromium.try:closure_compilation
> Change-Id: Iae929216e164fe941e5e335a3ce2e7ff47dce0e3
> Reviewed-on: https://chromium-review.googlesource.com/c/1256570
> Commit-Queue: Zakhar Voit <voit@google.com>
> Reviewed-by: Sergey Poromov <poromov@chromium.org>
> Reviewed-by: David Roger <droger@chromium.org>
> Reviewed-by: Steven Bennetts <stevenjb@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#597559}

TBR=stevenjb@chromium.org,droger@chromium.org,poromov@chromium.org,hendrich@chromium.org,voit@google.com

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug:  579660 
Change-Id: I6107a6668abde445d972a97e63dca93446b0ab0d
Cq-Include-Trybots: luci.chromium.try:closure_compilation
Reviewed-on: https://chromium-review.googlesource.com/c/1278761
Reviewed-by: Alexander Hendrich <hendrich@chromium.org>
Commit-Queue: Alexander Hendrich <hendrich@chromium.org>
Cr-Commit-Position: refs/heads/master@{#599503}
[modify] https://crrev.com/ca7c8b17ae5f868bc878c61e1d3018c9c9f08e13/chrome/browser/chromeos/base/locale_util.cc
[modify] https://crrev.com/ca7c8b17ae5f868bc878c61e1d3018c9c9f08e13/chrome/browser/chromeos/base/locale_util.h
[modify] https://crrev.com/ca7c8b17ae5f868bc878c61e1d3018c9c9f08e13/chrome/browser/chromeos/locale_change_guard.cc
[modify] https://crrev.com/ca7c8b17ae5f868bc878c61e1d3018c9c9f08e13/chrome/browser/chromeos/login/session/user_session_manager.cc
[modify] https://crrev.com/ca7c8b17ae5f868bc878c61e1d3018c9c9f08e13/chrome/browser/chromeos/preferences.cc
[modify] https://crrev.com/ca7c8b17ae5f868bc878c61e1d3018c9c9f08e13/chrome/browser/chromeos/preferences.h
[modify] https://crrev.com/ca7c8b17ae5f868bc878c61e1d3018c9c9f08e13/chrome/browser/extensions/api/language_settings_private/language_settings_private_api.cc
[modify] https://crrev.com/ca7c8b17ae5f868bc878c61e1d3018c9c9f08e13/chrome/browser/policy/configuration_policy_handler_list_factory.cc
[modify] https://crrev.com/ca7c8b17ae5f868bc878c61e1d3018c9c9f08e13/chrome/browser/policy/policy_browsertest.cc
[modify] https://crrev.com/ca7c8b17ae5f868bc878c61e1d3018c9c9f08e13/chrome/browser/profiles/profile.cc
[modify] https://crrev.com/ca7c8b17ae5f868bc878c61e1d3018c9c9f08e13/chrome/browser/profiles/profile.h
[modify] https://crrev.com/ca7c8b17ae5f868bc878c61e1d3018c9c9f08e13/chrome/browser/resources/settings/languages_page/add_languages_dialog.js
[modify] https://crrev.com/ca7c8b17ae5f868bc878c61e1d3018c9c9f08e13/chrome/browser/resources/settings/languages_page/languages.js
[modify] https://crrev.com/ca7c8b17ae5f868bc878c61e1d3018c9c9f08e13/chrome/browser/resources/settings/languages_page/languages_page.html
[modify] https://crrev.com/ca7c8b17ae5f868bc878c61e1d3018c9c9f08e13/chrome/browser/resources/settings/languages_page/languages_page.js
[modify] https://crrev.com/ca7c8b17ae5f868bc878c61e1d3018c9c9f08e13/chrome/browser/resources/settings/languages_page/languages_types.js
[modify] https://crrev.com/ca7c8b17ae5f868bc878c61e1d3018c9c9f08e13/chrome/browser/ui/webui/settings/languages_handler.cc
[modify] https://crrev.com/ca7c8b17ae5f868bc878c61e1d3018c9c9f08e13/chrome/common/extensions/api/language_settings_private.idl
[modify] https://crrev.com/ca7c8b17ae5f868bc878c61e1d3018c9c9f08e13/chrome/common/pref_names.cc
[modify] https://crrev.com/ca7c8b17ae5f868bc878c61e1d3018c9c9f08e13/chrome/common/pref_names.h
[modify] https://crrev.com/ca7c8b17ae5f868bc878c61e1d3018c9c9f08e13/chrome/test/data/policy/policy_test_cases.json
[modify] https://crrev.com/ca7c8b17ae5f868bc878c61e1d3018c9c9f08e13/components/policy/resources/policy_templates.json
[modify] https://crrev.com/ca7c8b17ae5f868bc878c61e1d3018c9c9f08e13/testing/buildbot/filters/mojo.fyi.chromeos.network_browser_tests.filter
[modify] https://crrev.com/ca7c8b17ae5f868bc878c61e1d3018c9c9f08e13/third_party/closure_compiler/externs/language_settings_private.js
[modify] https://crrev.com/ca7c8b17ae5f868bc878c61e1d3018c9c9f08e13/tools/metrics/histograms/enums.xml

#26 got it green again, tyvm!
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel-vm-tests?limit=50

I'm going to add it to the CQ now, then we can finally be done w/ these suites.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/49bfeebd653e16045a60bd3d2cd97ade13d5cace

commit 49bfeebd653e16045a60bd3d2cd97ade13d5cace
Author: Ben Pastene <bpastene@chromium.org>
Date: Mon Oct 15 20:00:21 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7f0c13c716d21f1ef1460372bb99eb4e196fc53f

commit 7f0c13c716d21f1ef1460372bb99eb4e196fc53f
Author: Ben Pastene <bpastene@chromium.org>
Date: Tue Oct 16 04:41:37 2018

Move telemetry_unittests from cros VM fyi bot to CQ bot.

Bug:  876494 
Change-Id: Icfa8631e1278745a50e439d78b4373fb1f0f1f3a
Reviewed-on: https://chromium-review.googlesource.com/c/1281230
Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#599851}
[modify] https://crrev.com/7f0c13c716d21f1ef1460372bb99eb4e196fc53f/testing/buildbot/chromium.chromiumos.json
[modify] https://crrev.com/7f0c13c716d21f1ef1460372bb99eb4e196fc53f/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/7f0c13c716d21f1ef1460372bb99eb4e196fc53f/testing/buildbot/test_suites.pyl
[modify] https://crrev.com/7f0c13c716d21f1ef1460372bb99eb4e196fc53f/testing/buildbot/waterfalls.pyl

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/17f06f3bd2197ef8c78ba99513790eface88c329

commit 17f06f3bd2197ef8c78ba99513790eface88c329
Author: Ben Pastene <bpastene@chromium.org>
Date: Tue Oct 16 17:08:24 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d002614c03444c51d0b84480acf961f3096b5b08

commit d002614c03444c51d0b84480acf961f3096b5b08
Author: Ben Pastene <bpastene@chromium.org>
Date: Tue Oct 16 18:03:13 2018

Revert "Move telemetry_unittests from cros VM fyi bot to CQ bot."

This reverts commit 7f0c13c716d21f1ef1460372bb99eb4e196fc53f.

Reason for revert: we don't have the capacity :(
https://luci-milo.appspot.com/p/chromium/builders/luci.chromium.try/chromeos-amd64-generic-rel

Original change's description:
> Move telemetry_unittests from cros VM fyi bot to CQ bot.
> 
> Bug:  876494 
> Change-Id: Icfa8631e1278745a50e439d78b4373fb1f0f1f3a
> Reviewed-on: https://chromium-review.googlesource.com/c/1281230
> Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>
> Commit-Queue: Ben Pastene <bpastene@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#599851}

TBR=achuith@chromium.org,bpastene@chromium.org,jbudorick@chromium.org

Change-Id: I8b9981920ec5078da9147e0ef36d2c96028501b4
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  876494 
Reviewed-on: https://chromium-review.googlesource.com/c/1283932
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#600042}
[modify] https://crrev.com/d002614c03444c51d0b84480acf961f3096b5b08/testing/buildbot/chromium.chromiumos.json
[modify] https://crrev.com/d002614c03444c51d0b84480acf961f3096b5b08/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/d002614c03444c51d0b84480acf961f3096b5b08/testing/buildbot/test_suites.pyl
[modify] https://crrev.com/d002614c03444c51d0b84480acf961f3096b5b08/testing/buildbot/waterfalls.pyl

Had to revert #29 since there's a large pending queue building up on the VM bot:
https://luci-milo.appspot.com/p/chromium/builders/luci.chromium.try/chromeos-amd64-generic-rel

I think adding the suite bumped us past our test-bot capacity despite increases in #28 and #30. I'm going run some numbers on how much capacity we really need to successfully add the suite, then reland it if we can.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/infra_internal/+/6f2d7b4baedee07274ca214a33a85c24804e24fa

commit 6f2d7b4baedee07274ca214a33a85c24804e24fa
Author: Ben Pastene <bpastene@chromium.org>
Date: Tue Oct 16 20:53:41 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/82431e273d2f347205266ff4363205d6de25e0ad

commit 82431e273d2f347205266ff4363205d6de25e0ad
Author: Ben Pastene <bpastene@chromium.org>
Date: Tue Oct 16 21:01:17 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/ee48a1b1e817029c1fd03b7a113854a17dc1a301

commit ee48a1b1e817029c1fd03b7a113854a17dc1a301
Author: Ben Pastene <bpastene@chromium.org>
Date: Wed Oct 17 17:05:12 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/fcaf2e59138d695e582af17e0606712bce4363e8

commit fcaf2e59138d695e582af17e0606712bce4363e8
Author: Ben Pastene <bpastene@chromium.org>
Date: Wed Oct 17 20:00:26 2018

Reland "Move telemetry_unittests from cros VM fyi bot to CQ bot."

This is a reland of 7f0c13c716d21f1ef1460372bb99eb4e196fc53f

We've bumped capacity in both the builder pool and tester pool.
Hopefully that'll be enough!

Original change's description:
> Move telemetry_unittests from cros VM fyi bot to CQ bot.
>
> Bug:  876494 
> Change-Id: Icfa8631e1278745a50e439d78b4373fb1f0f1f3a
> Reviewed-on: https://chromium-review.googlesource.com/c/1281230
> Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>
> Commit-Queue: Ben Pastene <bpastene@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#599851}

Bug:  876494 
Change-Id: Id83dc5f3e0063c130ce7b80b8a72e28b9f72bd83
Reviewed-on: https://chromium-review.googlesource.com/c/1286900
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#600527}
[modify] https://crrev.com/fcaf2e59138d695e582af17e0606712bce4363e8/testing/buildbot/chromium.chromiumos.json
[modify] https://crrev.com/fcaf2e59138d695e582af17e0606712bce4363e8/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/fcaf2e59138d695e582af17e0606712bce4363e8/testing/buildbot/test_suites.pyl
[modify] https://crrev.com/fcaf2e59138d695e582af17e0606712bce4363e8/testing/buildbot/waterfalls.pyl

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/b88caebdc493b50e266ada85cce82466aaae7cd9

commit b88caebdc493b50e266ada85cce82466aaae7cd9
Author: Ben Pastene <bpastene@chromium.org>
Date: Fri Oct 19 16:10:48 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0986b5c19969d52fcb465e8333942c70e3471b90

commit 0986b5c19969d52fcb465e8333942c70e3471b90
Author: Ben Pastene <bpastene@chromium.org>
Date: Fri Oct 19 18:04:12 2018

Bump telemetry_unittests to 100% experiment on cros VM CQ bot.

Hopefully after crrev.com/i/701289 there's enough capacity this time.

TBR=jbudorick, achuith

Bug:  876494 
Change-Id: I10e8fa31851c8ebd5f38bb233d66c5b0b47bbf29
Reviewed-on: https://chromium-review.googlesource.com/c/1291709
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#601223}
[modify] https://crrev.com/0986b5c19969d52fcb465e8333942c70e3471b90/testing/buildbot/chromium.chromiumos.json
[modify] https://crrev.com/0986b5c19969d52fcb465e8333942c70e3471b90/testing/buildbot/test_suites.pyl

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f24d2e318f99b4eea30ada95a37122e91fddc116

commit f24d2e318f99b4eea30ada95a37122e91fddc116
Author: Alexander Hendrich <hendrich@chromium.org>
Date: Mon Oct 22 16:01:55 2018

[Merge M71] Revert "Updated AllowedUILocales to handle preferred languages."

This reverts commit 75d0995fa069994cee37b7ba3c940c6927e7d91b.

Reason for revert: Causes DCHECK crashes on sign-in

Original change's description:
> Updated AllowedUILocales to handle preferred languages.
>
> Updated AllowedUILocales to limit not only the ui locales but also the
> available preferred languages. Renamed AllowedUILocales to AllowedLanguages to
> better indicate that.
>
> Bug:  579660 ,  876494 ,  894406 
> Cq-Include-Trybots: luci.chromium.try:closure_compilation
> Change-Id: Iae929216e164fe941e5e335a3ce2e7ff47dce0e3
> Reviewed-on: https://chromium-review.googlesource.com/c/1256570
> Commit-Queue: Zakhar Voit <voit@google.com>
> Reviewed-by: Sergey Poromov <poromov@chromium.org>
> Reviewed-by: David Roger <droger@chromium.org>
> Reviewed-by: Steven Bennetts <stevenjb@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#597559}

TBR=stevenjb@chromium.org,droger@chromium.org,poromov@chromium.org,hendrich@chromium.org,voit@google.com

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug:  579660 
Change-Id: I6107a6668abde445d972a97e63dca93446b0ab0d
Cq-Include-Trybots: luci.chromium.try:closure_compilation
Reviewed-on: https://chromium-review.googlesource.com/c/1278761
Reviewed-by: Alexander Hendrich <hendrich@chromium.org>
Commit-Queue: Alexander Hendrich <hendrich@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#599503}(cherry picked from commit ca7c8b17ae5f868bc878c61e1d3018c9c9f08e13)
Reviewed-on: https://chromium-review.googlesource.com/c/1293909
Reviewed-by: Maksim Ivanov <emaxx@chromium.org>
Cr-Commit-Position: refs/branch-heads/3578@{#210}
Cr-Branched-From: 4226ddf99103e493d7afb23a4c7902ee496108b6-refs/heads/master@{#599034}
[modify] https://crrev.com/f24d2e318f99b4eea30ada95a37122e91fddc116/chrome/browser/chromeos/base/locale_util.cc
[modify] https://crrev.com/f24d2e318f99b4eea30ada95a37122e91fddc116/chrome/browser/chromeos/base/locale_util.h
[modify] https://crrev.com/f24d2e318f99b4eea30ada95a37122e91fddc116/chrome/browser/chromeos/locale_change_guard.cc
[modify] https://crrev.com/f24d2e318f99b4eea30ada95a37122e91fddc116/chrome/browser/chromeos/login/session/user_session_manager.cc
[modify] https://crrev.com/f24d2e318f99b4eea30ada95a37122e91fddc116/chrome/browser/chromeos/preferences.cc
[modify] https://crrev.com/f24d2e318f99b4eea30ada95a37122e91fddc116/chrome/browser/chromeos/preferences.h
[modify] https://crrev.com/f24d2e318f99b4eea30ada95a37122e91fddc116/chrome/browser/extensions/api/language_settings_private/language_settings_private_api.cc
[modify] https://crrev.com/f24d2e318f99b4eea30ada95a37122e91fddc116/chrome/browser/policy/configuration_policy_handler_list_factory.cc
[modify] https://crrev.com/f24d2e318f99b4eea30ada95a37122e91fddc116/chrome/browser/policy/policy_browsertest.cc
[modify] https://crrev.com/f24d2e318f99b4eea30ada95a37122e91fddc116/chrome/browser/profiles/profile.cc
[modify] https://crrev.com/f24d2e318f99b4eea30ada95a37122e91fddc116/chrome/browser/profiles/profile.h
[modify] https://crrev.com/f24d2e318f99b4eea30ada95a37122e91fddc116/chrome/browser/resources/settings/languages_page/add_languages_dialog.js
[modify] https://crrev.com/f24d2e318f99b4eea30ada95a37122e91fddc116/chrome/browser/resources/settings/languages_page/languages.js
[modify] https://crrev.com/f24d2e318f99b4eea30ada95a37122e91fddc116/chrome/browser/resources/settings/languages_page/languages_page.html
[modify] https://crrev.com/f24d2e318f99b4eea30ada95a37122e91fddc116/chrome/browser/resources/settings/languages_page/languages_page.js
[modify] https://crrev.com/f24d2e318f99b4eea30ada95a37122e91fddc116/chrome/browser/resources/settings/languages_page/languages_types.js
[modify] https://crrev.com/f24d2e318f99b4eea30ada95a37122e91fddc116/chrome/browser/ui/webui/settings/languages_handler.cc
[modify] https://crrev.com/f24d2e318f99b4eea30ada95a37122e91fddc116/chrome/common/extensions/api/language_settings_private.idl
[modify] https://crrev.com/f24d2e318f99b4eea30ada95a37122e91fddc116/chrome/common/pref_names.cc
[modify] https://crrev.com/f24d2e318f99b4eea30ada95a37122e91fddc116/chrome/common/pref_names.h
[modify] https://crrev.com/f24d2e318f99b4eea30ada95a37122e91fddc116/chrome/test/data/policy/policy_test_cases.json
[modify] https://crrev.com/f24d2e318f99b4eea30ada95a37122e91fddc116/components/policy/resources/policy_templates.json
[modify] https://crrev.com/f24d2e318f99b4eea30ada95a37122e91fddc116/testing/buildbot/filters/mojo.fyi.chromeos.network_browser_tests.filter
[modify] https://crrev.com/f24d2e318f99b4eea30ada95a37122e91fddc116/third_party/closure_compiler/externs/language_settings_private.js
[modify] https://crrev.com/f24d2e318f99b4eea30ada95a37122e91fddc116/tools/metrics/histograms/enums.xml

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/50ed7cec89d03e8077bb78bc76a53b04c3e5c2fd

commit 50ed7cec89d03e8077bb78bc76a53b04c3e5c2fd
Author: Ben Pastene <bpastene@chromium.org>
Date: Mon Oct 29 18:16:04 2018

Move telemetry_unittests out of experimental on cros VM CQ bot.

It's been at 100% experimental for a while now and it doesn't appear
to be blowing out our capacity.

Bug:  876494 
Change-Id: I90a4fda3350bd96a03244d9b53b0471ce66923a8
Reviewed-on: https://chromium-review.googlesource.com/c/1301971
Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#603567}
[modify] https://crrev.com/50ed7cec89d03e8077bb78bc76a53b04c3e5c2fd/testing/buildbot/chromium.chromiumos.json
[modify] https://crrev.com/50ed7cec89d03e8077bb78bc76a53b04c3e5c2fd/testing/buildbot/test_suites.pyl

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/05895502a6de3e2faf76f11c447b7bb79e2bbee4

commit 05895502a6de3e2faf76f11c447b7bb79e2bbee4
Author: Ben Pastene <bpastene@chromium.org>
Date: Tue Oct 30 17:55:08 2018

telemetry_unittests has been on the cros cq bot now for a couple days, and things seem pretty stable.

I'm going to close this out once and for all before something has a chance to break again ;)