Add ChromeOS VM tests to the CQ |
||||
Issue descriptionAny standard gtest suite should be compatible. (The bigger ones may not fully pass, but they'll at least run.) Let's get the smaller green ones up and running. All that's needed is: - carve off a small swarming pool for the vm tests (so that the vm image stays perpetually warm in the cache) - add the tests to the testing spec of the amd64-generic simplechrome bot telemetry and vm-sanity tests are still not quite ready, but I'm working on that in bug 839187 and bug 832374 . Will add those once they're ready.
,
May 9 2018
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/config/+/599beeb31d14edd2edeb2aaaf24aeb4441f8dd1a commit 599beeb31d14edd2edeb2aaaf24aeb4441f8dd1a Author: Ben Pastene <bpastene@chromium.org> Date: Wed May 09 20:15:59 2018
,
May 9 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/c55041d7fbb3cc88a500ec407b11ce24d21ed8c6 commit c55041d7fbb3cc88a500ec407b11ce24d21ed8c6 Author: Ben Pastene <bpastene@chromium.org> Date: Wed May 09 20:27:42 2018 Move cros vm tests to their own swarming pool. Should help with keeping the vm image warm in the bots' caches. Currently 20 bots in the pool. Will scale up if needed. TBR=dpranke@chromium.org Bug: 840967, 832374 Change-Id: Ia09fdde0a94d0d798cd1c5073f0651b1ec92ac66 Reviewed-on: https://chromium-review.googlesource.com/1052824 Reviewed-by: Ben Pastene <bpastene@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> Cr-Commit-Position: refs/heads/master@{#557303} [modify] https://crrev.com/c55041d7fbb3cc88a500ec407b11ce24d21ed8c6/testing/buildbot/chromium.fyi.json [modify] https://crrev.com/c55041d7fbb3cc88a500ec407b11ce24d21ed8c6/testing/buildbot/waterfalls.pyl
,
May 10 2018
CL's ready to go (https://chromium-review.googlesource.com/c/chromium/src/+/1053235) and PSA has been sent. Planning on landing it tomorrow morning barring any objections to the announcement.
,
May 10 2018
I always thought we'd go the route of FYI bots->optional try bots->CQ bots. Do we have confidence that this won't trip up a bunch of folks tomorrow? I'm excited about this! Great work!
,
May 10 2018
We've had the tests running on FYI bots for a while now.
,
May 10 2018
Ah, ok! Good to hear.
,
May 10 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/f3aad9060255f89503a55169e228a7ed8eb61930 commit f3aad9060255f89503a55169e228a7ed8eb61930 Author: Ben Pastene <bpastene@chromium.org> Date: Thu May 10 17:08:13 2018 Add 7 small gtests to the amd64-generic CrOS simplechrome CQ bot. Also add a couple not-so-small gtests (base_unittests and net_unittests) to the fyi test bot. Current test pool has 20 bots. Will bump more if needed, but given how rarely I think analyze would trigger these specific tests, maybe not needed? Bug: 840967 Change-Id: Id71c7f0899790da95a8c3ff60ea62d3f831511ee Reviewed-on: https://chromium-review.googlesource.com/1053235 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Achuith Bhandarkar <achuith@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> Cr-Commit-Position: refs/heads/master@{#557562} [modify] https://crrev.com/f3aad9060255f89503a55169e228a7ed8eb61930/testing/buildbot/chromium.chromiumos.json [modify] https://crrev.com/f3aad9060255f89503a55169e228a7ed8eb61930/testing/buildbot/chromium.fyi.json [modify] https://crrev.com/f3aad9060255f89503a55169e228a7ed8eb61930/testing/buildbot/test_suites.pyl [modify] https://crrev.com/f3aad9060255f89503a55169e228a7ed8eb61930/testing/buildbot/waterfalls.pyl
,
May 10 2018
We could go the route of adding the tests to mandatory CQ bots, but make the tests experimental (ie: if they fail, it doesn't block the CL). Though I feel confident that that shouldn't be necessary. These have been running on the fyi bot for a while and look very stable.
,
May 10 2018
And you have the ability to yank them if they suddenly start failing, right?
,
May 10 2018
Yeah; the CL that added them (https://chromium-review.googlesource.com/1053235) is one-click revertable
,
May 10 2018
Already had a few test-triggering CLs by now and things look fine: https://ci.chromium.org/buildbot/tryserver.chromium.chromiumos/chromeos-amd64-generic-rel/116688 https://ci.chromium.org/buildbot/tryserver.chromium.chromiumos/chromeos-amd64-generic-rel/116690 I'm also keeping an eye on test bot capacity: https://chromium-swarm.appspot.com/botlist?c=id&c=os&c=task&c=status&f=pool%3AChrome-CrOS-VM&l=100&q=pool%3A&s=id%3Aasc We've got 20 machines and they don't seem over-burdened as-of-yet, but I can bump that number up pretty easily if needed.
,
May 11 2018
Everything's looking good. And as an added bonus, all LKGM updates now trigger the full set of vm tests on the cq: https://ci.chromium.org/buildbot/tryserver.chromium.chromiumos/chromeos-amd64-generic-rel/117379 from https://chromium-review.googlesource.com/c/chromium/src/+/1054513 (Previously, the updates weren't even triggering a compile.) Since the lkgm is what controls what VM image we download, this prevents broken changes to the VM from getting rolled into src unawares. (Same should be true for chromite now that we got that on an auto-roller too.)
,
May 11 2018
Yeah, my only concern is that we're not doing VM verification through the simple chrome flow on the chromeos side, and breakages in amd64-generic sometimes go unnoticed. I'm not sure if it's now possible for a chromeos breakage can affect the chromium CQ, but the additional testing should be there. I'll file a bug for this.
,
May 11 2018
This is all fantastic though!
,
Jun 2 2018
As a quick update of things so far, there's 7 smaller gtest suites up and running in VMs on the CQ today (https://luci-milo.appspot.com/p/chromium/builders/luci.chromium.try/chromeos-amd64-generic-rel). I'm currently working on resolving infra-related issues on a few more suites, outlined below: - base_unittests: only 4 failing tests, all of which will get fixed with http://crrev.com/c/1079644 - net_unittests: only 4 failing tests; 3 will get fixed with http://crrev.com/c/1079644; still working on the last one - sanity_test: has a %5 of freezing/timing out, tracking the flake in bug 848402 - telemetry(_perf) unit tests: shares a similar flake rate as the sanity test, I'm 99% sure it's the same root cause, so should get fixed with bug 848402
,
Jun 13 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/20ea78e677f0adf7440297034fb68997054c7ab8 commit 20ea78e677f0adf7440297034fb68997054c7ab8 Author: Ben Pastene <bpastene@chromium.org> Date: Wed Jun 13 19:33:34 2018 Run chromeos VM tests as a non-root user. And enable base_unittests on the CQ. Bug: 840967 Change-Id: I485b9d785ef02fe9918e0f2c310e3b63ffab1b4a Reviewed-on: https://chromium-review.googlesource.com/1098314 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Reviewed-by: John Budorick <jbudorick@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> Cr-Commit-Position: refs/heads/master@{#566948} [modify] https://crrev.com/20ea78e677f0adf7440297034fb68997054c7ab8/build/chromeos/run_vm_test.py [modify] https://crrev.com/20ea78e677f0adf7440297034fb68997054c7ab8/testing/buildbot/chromium.chromiumos.json [modify] https://crrev.com/20ea78e677f0adf7440297034fb68997054c7ab8/testing/buildbot/test_suites.pyl
,
Jun 13 2018
,
Jun 14 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/c4bad6e827a64218b2c7fa7ec2a1e86ee28f0968 commit c4bad6e827a64218b2c7fa7ec2a1e86ee28f0968 Author: Ben Pastene <bpastene@chromium.org> Date: Thu Jun 14 05:30:34 2018 Add 7 more test suites to cros VM tester. This includes: cc_unittests display_unittests ipc_tests jingle_unittests latency_unittests pdf_unittests printing_unittests Bug: 840967 Change-Id: I0e0cc44ab2f1ce79aeddefd2cbc2d6ff22fa5d69 Reviewed-on: https://chromium-review.googlesource.com/1100375 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> Cr-Commit-Position: refs/heads/master@{#567153} [modify] https://crrev.com/c4bad6e827a64218b2c7fa7ec2a1e86ee28f0968/testing/buildbot/chromium.chromiumos.json [modify] https://crrev.com/c4bad6e827a64218b2c7fa7ec2a1e86ee28f0968/testing/buildbot/test_suites.pyl
,
Jun 14 2018
After #19, we've got 15 gtest suites running in VMs on the CQ. These are also running on the linux-chromeos release bot: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/linux-chromeos-rel Do we need the suites running on both bots? Is there anything we get out running the tests in the linux-chromeos build that a true chromeos environment doesn't get us? If not, I propose we remove them from the desktop bot. Less load = more free resources = always a good thing
,
Jun 14 2018
Are the unit tests in chromeos-amd64-generic-rel running on the builder or in the VM? It seems like we should continue running unit tests in a linux environment on linux-chromeos-rel, since that is what developers primarily do.
,
Jun 14 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/2421ea07f25d0dc718df07d6d0b6bb0badfc939c commit 2421ea07f25d0dc718df07d6d0b6bb0badfc939c Author: Ben Pastene <bpastene@chromium.org> Date: Thu Jun 14 20:47:44 2018 Fix duration check in base_unittest's TimeTicks.ThreadNow. This test is flaky in CrOS VMs: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel/14479 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel/14493 Flakiness dashboard is quite black: https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=base_unittests&tests=ThreadTicks.ThreadNow Bug: 352633, 840967 Change-Id: Ia6efb4070bcef525992015f766599846a13ee1ec Reviewed-on: https://chromium-review.googlesource.com/1101412 Reviewed-by: Gabriel Charette <gab@chromium.org> Commit-Queue: Gabriel Charette <gab@chromium.org> Cr-Commit-Position: refs/heads/master@{#567392} [modify] https://crrev.com/2421ea07f25d0dc718df07d6d0b6bb0badfc939c/base/time/time_unittest.cc
,
Jun 14 2018
> It seems like we should continue running unit tests in a linux environment > on linux-chromeos-rel, since that is what developers primarily do. It seems like it'd mostly be a question of what would break only in one environment or another. If they always break in both, there's no point in running both.
,
Jun 14 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/f42d036f2f2aac22764b9ead9069fd752c2273ce commit f42d036f2f2aac22764b9ead9069fd752c2273ce Author: Ben Pastene <bpastene@chromium.org> Date: Thu Jun 14 21:37:21 2018 Catch unknown gtest args for cros VM tests and pass them down. The retry-without-patch recipe flow is currently broken (whoops!!) https://luci-milo.appspot.com/p/chromium/builders/luci.chromium.try/chromeos-amd64-generic-rel/25559 Bug: 840967 Change-Id: I0beeb5657f79475089fd6bb355263d7b8543796c Reviewed-on: https://chromium-review.googlesource.com/1101316 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Reviewed-by: John Budorick <jbudorick@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> Cr-Commit-Position: refs/heads/master@{#567431} [modify] https://crrev.com/f42d036f2f2aac22764b9ead9069fd752c2273ce/build/chromeos/run_vm_test.py
,
Jun 19 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/da92230f1006a44b97f8c5b9a85d9785d9ea7694 commit da92230f1006a44b97f8c5b9a85d9785d9ea7694 Author: Ben Pastene <bpastene@chromium.org> Date: Tue Jun 19 01:47:21 2018 Promote the cros_vm_sanity_test from experimental on the cros VM CQ bot. It appears that the timeout+retries I added in CL:1102121 are successfully making the test more stable. See the following tests that froze, were caught+retried, and passed: https://chromium-swarm.appspot.com/task?id=3e2d8a94a80e7a10 https://chromium-swarm.appspot.com/task?id=3e2d2fdf269c5910 Bug: 840967 Change-Id: Ie4f02862201965e27af437c88e9a5e4ec0feccbf Reviewed-on: https://chromium-review.googlesource.com/1105314 Reviewed-by: John Budorick <jbudorick@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> Cr-Commit-Position: refs/heads/master@{#568289} [modify] https://crrev.com/da92230f1006a44b97f8c5b9a85d9785d9ea7694/testing/buildbot/chromium.chromiumos.json [modify] https://crrev.com/da92230f1006a44b97f8c5b9a85d9785d9ea7694/testing/buildbot/test_suites.pyl
,
Jun 29 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/33d45ca1a8ce607ac54f330361e530230cadfedb commit 33d45ca1a8ce607ac54f330361e530230cadfedb Author: Ben Pastene <bpastene@chromium.org> Date: Fri Jun 29 17:40:24 2018 Add mojo & ozone unittests to cros VM CQ bot. Bug: 840967 Change-Id: Id556f0b2740b0ee9fcd799234c5936c71daf81fc Reviewed-on: https://chromium-review.googlesource.com/1118916 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> Cr-Commit-Position: refs/heads/master@{#571518} [modify] https://crrev.com/33d45ca1a8ce607ac54f330361e530230cadfedb/testing/buildbot/chromium.chromiumos.json [modify] https://crrev.com/33d45ca1a8ce607ac54f330361e530230cadfedb/testing/buildbot/test_suites.pyl
,
Jun 30 2018
,
Jul 17
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/7d5467abddf0d2bcb7079874ab6bd80cb293d7c2 commit 7d5467abddf0d2bcb7079874ab6bd80cb293d7c2 Author: Ben Pastene <bpastene@chromium.org> Date: Tue Jul 17 23:51:01 2018 Add media_unittests to cros VM tester. Some tests are failing, so this adds them to a filter file and adds it in a la fuchsia. Bug: 860066, 840967 Change-Id: I6ebe67a0d4a5ab5e58c543524d36ada6ae55219f Reviewed-on: https://chromium-review.googlesource.com/1136884 Reviewed-by: John Budorick <jbudorick@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> Cr-Commit-Position: refs/heads/master@{#575846} [modify] https://crrev.com/7d5467abddf0d2bcb7079874ab6bd80cb293d7c2/build/config/chromeos/rules.gni [modify] https://crrev.com/7d5467abddf0d2bcb7079874ab6bd80cb293d7c2/testing/buildbot/chromium.chromiumos.json [modify] https://crrev.com/7d5467abddf0d2bcb7079874ab6bd80cb293d7c2/testing/buildbot/filters/BUILD.gn [add] https://crrev.com/7d5467abddf0d2bcb7079874ab6bd80cb293d7c2/testing/buildbot/filters/chromeos.media_unittests.filter [modify] https://crrev.com/7d5467abddf0d2bcb7079874ab6bd80cb293d7c2/testing/buildbot/test_suites.pyl
,
Jul 19
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/ba2572a6bdfb93aff61c9133464eec544a571d71 commit ba2572a6bdfb93aff61c9133464eec544a571d71 Author: Ben Pastene <bpastene@chromium.org> Date: Thu Jul 19 17:21:31 2018 Add sandbox_linux_unittests to cros VM bot. Bug: 860066, 840967 Change-Id: I5dbc65b55d423f7a79de63dcdf6c6f5dd36c463b Reviewed-on: https://chromium-review.googlesource.com/1137439 Reviewed-by: John Budorick <jbudorick@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> Cr-Commit-Position: refs/heads/master@{#576535} [modify] https://crrev.com/ba2572a6bdfb93aff61c9133464eec544a571d71/testing/buildbot/chromium.chromiumos.json [modify] https://crrev.com/ba2572a6bdfb93aff61c9133464eec544a571d71/testing/buildbot/test_suites.pyl
,
Jul 20
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/e3560e33fe397c5f7d6b93317773a908030b6376 commit e3560e33fe397c5f7d6b93317773a908030b6376 Author: Ben Pastene <bpastene@chromium.org> Date: Fri Jul 20 18:19:55 2018 Run chromeos_unittests on cros VM fyi bot. Bug: 840967 Change-Id: I3f9e8396cc2cc394bcde660cc8bfd8ff183fa663 Reviewed-on: https://chromium-review.googlesource.com/1144053 Commit-Queue: Ben Pastene <bpastene@chromium.org> Reviewed-by: John Budorick <jbudorick@chromium.org> Cr-Commit-Position: refs/heads/master@{#576928} [modify] https://crrev.com/e3560e33fe397c5f7d6b93317773a908030b6376/testing/buildbot/chromium.fyi.json [modify] https://crrev.com/e3560e33fe397c5f7d6b93317773a908030b6376/testing/buildbot/filters/BUILD.gn [add] https://crrev.com/e3560e33fe397c5f7d6b93317773a908030b6376/testing/buildbot/filters/chromeos.chromeos_unittests.filter [modify] https://crrev.com/e3560e33fe397c5f7d6b93317773a908030b6376/testing/buildbot/test_suites.pyl
,
Jul 20
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/63f1f746d473a01f328087e15a2993e2163153e3 commit 63f1f746d473a01f328087e15a2993e2163153e3 Author: Ben Pastene <bpastene@chromium.org> Date: Fri Jul 20 21:20:35 2018 Run aura_unittests on cros VM fyi bot. TBR=jbudorick Bug: 840967 Change-Id: Ib42d22e7d7c461ceef4684e40973c27b6fac68d0 Reviewed-on: https://chromium-review.googlesource.com/1145559 Reviewed-by: Ben Pastene <bpastene@chromium.org> Reviewed-by: John Budorick <jbudorick@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> Cr-Commit-Position: refs/heads/master@{#576991} [modify] https://crrev.com/63f1f746d473a01f328087e15a2993e2163153e3/testing/buildbot/chromium.fyi.json [modify] https://crrev.com/63f1f746d473a01f328087e15a2993e2163153e3/testing/buildbot/test_suites.pyl
,
Jul 27
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/8046431c221ea0b3ed026179d0dbc54ae1e7e602 commit 8046431c221ea0b3ed026179d0dbc54ae1e7e602 Author: Ben Pastene <bpastene@chromium.org> Date: Fri Jul 27 21:02:02 2018 Enable services_unittests on amd64-generic cros VM fyi bot. TBR=jbudorick@chromium.org Bug: 840967 Change-Id: I69e89b0fc3ec3f658cd90e930d0cef7410cd3ae8 Reviewed-on: https://chromium-review.googlesource.com/1152040 Reviewed-by: Ben Pastene <bpastene@chromium.org> Reviewed-by: John Budorick <jbudorick@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> Cr-Commit-Position: refs/heads/master@{#578781} [modify] https://crrev.com/8046431c221ea0b3ed026179d0dbc54ae1e7e602/testing/buildbot/chromium.fyi.json [modify] https://crrev.com/8046431c221ea0b3ed026179d0dbc54ae1e7e602/testing/buildbot/filters/BUILD.gn [add] https://crrev.com/8046431c221ea0b3ed026179d0dbc54ae1e7e602/testing/buildbot/filters/chromeos.services_unittests.filter [modify] https://crrev.com/8046431c221ea0b3ed026179d0dbc54ae1e7e602/testing/buildbot/test_suites.pyl
,
Aug 6
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/e6977fa40a62dec93ebc85426cb786b169e4badd commit e6977fa40a62dec93ebc85426cb786b169e4badd Author: Ben Pastene <bpastene@chromium.org> Date: Mon Aug 06 23:09:57 2018 Enable headless ozone on main amd64-generic simplechrome bot. And add a few test suites that need that arg to run. Bug: 840967 Change-Id: I9954a08cc22f7237b0e0d2093214b544b6f3d897 Reviewed-on: https://chromium-review.googlesource.com/1162863 Reviewed-by: John Budorick <jbudorick@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> Cr-Commit-Position: refs/heads/master@{#581031} [modify] https://crrev.com/e6977fa40a62dec93ebc85426cb786b169e4badd/testing/buildbot/chromium.chromiumos.json [modify] https://crrev.com/e6977fa40a62dec93ebc85426cb786b169e4badd/testing/buildbot/chromium.fyi.json [modify] https://crrev.com/e6977fa40a62dec93ebc85426cb786b169e4badd/testing/buildbot/test_suites.pyl [modify] https://crrev.com/e6977fa40a62dec93ebc85426cb786b169e4badd/tools/mb/mb_config.pyl
,
Aug 28
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/ae61c89ca72d9231f5af79261fe1a544f3d219f8 commit ae61c89ca72d9231f5af79261fe1a544f3d219f8 Author: Ben Pastene <bpastene@chromium.org> Date: Tue Aug 28 20:52:09 2018 Run net_unittests on the simplechrome amd64-generic CQ bot. And remove it from the FYI bot. The last remaining flaky test was disabled in crrev.com/c/1189033 This suite should be ready for the CQ. Bug: 840967 Change-Id: I39dde22b15dfd5d6efedd1a047923171922b020a Reviewed-on: https://chromium-review.googlesource.com/1192413 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> Cr-Commit-Position: refs/heads/master@{#586845} [modify] https://crrev.com/ae61c89ca72d9231f5af79261fe1a544f3d219f8/testing/buildbot/chromium.chromiumos.json [modify] https://crrev.com/ae61c89ca72d9231f5af79261fe1a544f3d219f8/testing/buildbot/chromium.fyi.json [modify] https://crrev.com/ae61c89ca72d9231f5af79261fe1a544f3d219f8/testing/buildbot/test_suites.pyl |
||||
►
Sign in to add a comment |
||||
Comment 1 by bugdroid1@chromium.org
, May 9 2018