Spin up a non-blocking bot that runs tests on cros vms. |
|||||||||||
Issue descriptionGoing to look identical to https://ci.chromium.org/buildbot/chromium.chromiumos/chromeos-amd64-generic-rel/ + some tests. Blocked on https://chromium-review.googlesource.com/c/chromium/src/+/994062 landing to at least get the tests running in VMs, but will set up the bot anyway. Filing this to track the multitude of config changes needed to create a new bot. ⛆ |
|
|
,
Apr 13 2018
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/config/+/6c9359e5eab580f2f3c2cdbeb461c0e2c83ffb1a commit 6c9359e5eab580f2f3c2cdbeb461c0e2c83ffb1a Author: Benjamin Pastene <bpastene@chromium.org> Date: Fri Apr 13 00:51:38 2018
,
Apr 13 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/f06103dc4019133ade92a65bbaac674b1c6da669 commit f06103dc4019133ade92a65bbaac674b1c6da669 Author: Benjamin Pastene <bpastene@chromium.org> Date: Fri Apr 13 01:32:11 2018 Add recipe config for new chromeos-amd64-generic-rel-vm-tests bot. Copied from the dict at https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium_tests/chromium_chromiumos.py?l=79 TBR=jbudorick@chromium.org Bug: 832374 Change-Id: I2a9d73c18b01732c5a8210f7fed223fa4413d446 Reviewed-on: https://chromium-review.googlesource.com/1011330 Commit-Queue: Ben Pastene <bpastene@chromium.org> Reviewed-by: Ben Pastene <bpastene@chromium.org> [modify] https://crrev.com/f06103dc4019133ade92a65bbaac674b1c6da669/scripts/slave/recipe_modules/chromium_tests/chromium_fyi.py
,
Apr 13 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/ed2aa1ee3ef5706aab8f58b401d4816364b42e1e commit ed2aa1ee3ef5706aab8f58b401d4816364b42e1e Author: Benjamin Pastene <bpastene@chromium.org> Date: Fri Apr 13 17:00:13 2018 Add LUCI configs for new chromeos-amd64-generic-rel-vm-tests bot. TBR=jbudorick@chromium.org Bug: 832374 Change-Id: I7500d930c5e8de66661e10867e35333d13d04355 Reviewed-on: https://chromium-review.googlesource.com/1011622 Commit-Queue: Ben Pastene <bpastene@chromium.org> Reviewed-by: Ben Pastene <bpastene@chromium.org> Cr-Commit-Position: refs/heads/master@{#550656} [modify] https://crrev.com/ed2aa1ee3ef5706aab8f58b401d4816364b42e1e/infra/config/global/cr-buildbucket.cfg [modify] https://crrev.com/ed2aa1ee3ef5706aab8f58b401d4816364b42e1e/infra/config/global/luci-scheduler.cfg
,
Apr 17 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/c87dfaf09b63ddfe3438f8da391f7ba94030634e commit c87dfaf09b63ddfe3438f8da391f7ba94030634e Author: Benjamin Pastene <bpastene@chromium.org> Date: Tue Apr 17 03:03:42 2018 Add mb config and test spec for CrOS vm tester bot. https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel-vm-tests https://chromium-review.googlesource.com/c/chromium/tools/build/+/1013289 needs to land before the tests actually do anything. Hopefully I'll be able to land that sometime soon. This at least gets the configs set up. Bug: 832374 Change-Id: I93665b1307c82e84e5a5d83d2092a792786573f6 Reviewed-on: https://chromium-review.googlesource.com/1014484 Commit-Queue: Ben Pastene <bpastene@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Cr-Commit-Position: refs/heads/master@{#551216} [modify] https://crrev.com/c87dfaf09b63ddfe3438f8da391f7ba94030634e/testing/buildbot/chromium.fyi.json [modify] https://crrev.com/c87dfaf09b63ddfe3438f8da391f7ba94030634e/testing/buildbot/test_suites.pyl [modify] https://crrev.com/c87dfaf09b63ddfe3438f8da391f7ba94030634e/testing/buildbot/waterfalls.pyl [modify] https://crrev.com/c87dfaf09b63ddfe3438f8da391f7ba94030634e/tools/mb/mb_config.pyl
,
Apr 17 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/ed2aa1ee3ef5706aab8f58b401d4816364b42e1e commit ed2aa1ee3ef5706aab8f58b401d4816364b42e1e Author: Benjamin Pastene <bpastene@chromium.org> Date: Fri Apr 13 17:00:13 2018 Add LUCI configs for new chromeos-amd64-generic-rel-vm-tests bot. TBR=jbudorick@chromium.org Bug: 832374 Change-Id: I7500d930c5e8de66661e10867e35333d13d04355 Reviewed-on: https://chromium-review.googlesource.com/1011622 Commit-Queue: Ben Pastene <bpastene@chromium.org> Reviewed-by: Ben Pastene <bpastene@chromium.org> Cr-Commit-Position: refs/heads/master@{#550656} [modify] https://crrev.com/ed2aa1ee3ef5706aab8f58b401d4816364b42e1e/infra/config/global/cr-buildbucket.cfg [modify] https://crrev.com/ed2aa1ee3ef5706aab8f58b401d4816364b42e1e/infra/config/global/luci-scheduler.cfg
,
Apr 18 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/b47ca36ae32a08b718f7885f0fbda26a4ab08a65 commit b47ca36ae32a08b718f7885f0fbda26a4ab08a65 Author: Benjamin Pastene <bpastene@chromium.org> Date: Wed Apr 18 19:16:57 2018 Fix name of cros vm test bot in mb_config. Whoops TBR=dpranke@chromium.org Bug: 832374 Change-Id: Ib2caf7bcb5d6cc8e15312122d8cb24f6eb0ca23c Reviewed-on: https://chromium-review.googlesource.com/1015408 Reviewed-by: Ben Pastene <bpastene@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> Cr-Commit-Position: refs/heads/master@{#551775} [modify] https://crrev.com/b47ca36ae32a08b718f7885f0fbda26a4ab08a65/tools/mb/mb_config.pyl
,
Apr 18 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/fab64fff87d182b38cecccce779a57b682c7fa3e commit fab64fff87d182b38cecccce779a57b682c7fa3e Author: Benjamin Pastene <bpastene@chromium.org> Date: Wed Apr 18 22:21:16 2018 Clobber the cros-chrome-sdk cache everytime when fetching artifacts. Adds 2-3 minutes to every cros-flavored "gclient runhooks", which is unfortunate. Should be unnecessary when/if crbug.com/834134 is fixed. Bug: 832374 Change-Id: I2696fc736350ecbe2b641112d2140db66118a00e Reviewed-on: https://chromium-review.googlesource.com/1016137 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> Cr-Commit-Position: refs/heads/master@{#551850} [modify] https://crrev.com/fab64fff87d182b38cecccce779a57b682c7fa3e/DEPS
,
Apr 19 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/b254d023b48d450f39ef865746317f164c2ac549 commit b254d023b48d450f39ef865746317f164c2ac549 Author: Benjamin Pastene <bpastene@chromium.org> Date: Thu Apr 19 17:11:47 2018 Don't wrap runhooks in the cros chrome-sdk for simplechrome bots. And point all other invocations of it to the new cache location. (Used to be "src/../.cros_cache" but now it's "src/build/cros_cache") And also set the gclient var cros_board so the gclient hook downloads the right board for each bot. Recipe-Nontrivial-Roll: build_limited_scripts_slave Bug: 832374 , 732531 Change-Id: I9aea093d7a6e519922eb375cf82bc0da1ebd0336 Reviewed-on: https://chromium-review.googlesource.com/1013289 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> [modify] https://crrev.com/b254d023b48d450f39ef865746317f164c2ac549/scripts/slave/recipe_modules/chromium/tests/run_mb.expected/cros_board.json [modify] https://crrev.com/b254d023b48d450f39ef865746317f164c2ac549/scripts/slave/recipe_modules/chromium/examples/full.py [modify] https://crrev.com/b254d023b48d450f39ef865746317f164c2ac549/scripts/slave/recipe_modules/chromium/api.py [modify] https://crrev.com/b254d023b48d450f39ef865746317f164c2ac549/scripts/slave/recipe_modules/chromium_tests/chromium_fyi.py [modify] https://crrev.com/b254d023b48d450f39ef865746317f164c2ac549/scripts/slave/README.recipes.md [modify] https://crrev.com/b254d023b48d450f39ef865746317f164c2ac549/scripts/slave/recipe_modules/chromium_tests/chromium_chromiumos.py [modify] https://crrev.com/b254d023b48d450f39ef865746317f164c2ac549/scripts/slave/recipe_modules/filter/tests/analyze.expected/basic.json [modify] https://crrev.com/b254d023b48d450f39ef865746317f164c2ac549/scripts/slave/recipe_modules/chromium/gclient_config.py [modify] https://crrev.com/b254d023b48d450f39ef865746317f164c2ac549/scripts/slave/recipe_modules/chromium/tests/compile.expected/chromeos.json [modify] https://crrev.com/b254d023b48d450f39ef865746317f164c2ac549/scripts/slave/recipe_modules/chromium/tests/runhooks.expected/chromeos.json [add] https://crrev.com/b254d023b48d450f39ef865746317f164c2ac549/scripts/slave/recipe_modules/chromium/examples/full.expected/chromeos_simplechrome.json [modify] https://crrev.com/b254d023b48d450f39ef865746317f164c2ac549/scripts/slave/recipes/swarming/staging.expected/cros.json
,
Apr 19 2018
Hurrah it works! https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel-vm-tests/524 And the tests are actually passing 100%: https://chromium-swarm.appspot.com/task?id=3cf7def6e69e6410 https://chromium-swarm.appspot.com/task?id=3cf7defaa9054810 https://chromium-swarm.appspot.com/task?id=3cf7deff74c78c10 Though the steps are purple since the result-collecting logic is a little confused. We're not pulling the test-results json file from the vm after the test finishes, so the recipe doesn't actually know the detailed results. I'll make the test wrapper pull it from the vm before finishing. Also, sharding isn't supported atm, but that's just a matter of passing down the "--shard-index" and "--shard-count" gtest args down to the test. I'll get to that too.
,
Apr 20 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/be4dea1713bc7a1c0461744289cfee9c3b88acd4 commit be4dea1713bc7a1c0461744289cfee9c3b88acd4 Author: Benjamin Pastene <bpastene@chromium.org> Date: Fri Apr 20 16:50:54 2018 Add standard gtest args to the cros vm_test wrapper. This adds support for test sharding and json results. See https://ci.chromium.org/buildbot/tryserver.chromium.chromiumos/chromeos-amd64-generic-rel/103454 for this patch in action. Bug: 832374 Change-Id: I1a1bea2412e25af9d5ba7a8daf0b6cb34e7b6ae4 Reviewed-on: https://chromium-review.googlesource.com/1020535 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> Cr-Commit-Position: refs/heads/master@{#552356} [modify] https://crrev.com/be4dea1713bc7a1c0461744289cfee9c3b88acd4/build/chromeos/run_vm_test.py
,
Apr 20 2018
The issues in #10 are solved, and now we got a fully green build: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel-vm-tests/655 \o/ So now all gtests should be ready to start running in the VMs. That covers ~90% of our tests. To get the rest I still need to cover: - telemetry/catapult unittests: Unlike gtest binaries, this expects to run on the host and be given hostname and port of cros device. I think I'll just need to call cros_vm --start/--stop before/afterwards and pass it localhost and VM port. (see also https://chromium.googlesource.com/chromiumos/docs/+/master/cros_vm.md#run-telemetry-unit-tests). - vm sanity test: This is the default behavior of cros_run_vm_test when no --cmd is specified. IIUC, it just opens chrome, loads some basic pages/runs some JS and closes. We'd probably want to run this on the bots as well. Just need to make sure we build chrome first (I believe the target chromiumos_preflight takes care of that) and deploy it to the device. cros_run_vm_test can take care of that, so this should be straightforward.
,
Apr 20 2018
Great!
,
Apr 21 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/6e06160799aafc332b225a52ee9bf130211d2b0f commit 6e06160799aafc332b225a52ee9bf130211d2b0f Author: Benjamin Pastene <bpastene@chromium.org> Date: Sat Apr 21 00:09:37 2018 Add a few more suites to cros vm test bot. And double shard crypto_unittests to test sharding & merging logic. Bug: 832374 Change-Id: I165160b5482efb505861f7f4bd0e0f1497923efc Reviewed-on: https://chromium-review.googlesource.com/1022255 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> Cr-Commit-Position: refs/heads/master@{#552525} [modify] https://crrev.com/6e06160799aafc332b225a52ee9bf130211d2b0f/testing/buildbot/chromium.fyi.json [modify] https://crrev.com/6e06160799aafc332b225a52ee9bf130211d2b0f/testing/buildbot/test_suites.pyl
,
Apr 23 2018
Jotting down my current plan for the last two test types. telemetry: - break the data of generate_vm_runner_script target into a group: (that's what handles the logic of adding the vm image to the isolate) https://codesearch.chromium.org/chromium/src/build/config/chromeos/rules.gni?rcl=dcd041159fc60f661b18aa13b9349bff634b532c&l=23 - add that group as a dep to the telemetry test target for chromeos: (so that all telemetry isolates have the vm image) https://codesearch.chromium.org/chromium/src/chrome/test/BUILD.gn?rcl=30c65c50c700772bfe56e7a6a01724fc74367733&l=5529 - wrap the test call in build/chromeos/run_vm_test.py which will handle launching/tearing down the VM https://codesearch.chromium.org/chromium/src/testing/buildbot/gn_isolate_map.pyl?l=1062 or maybe https://codesearch.chromium.org/chromium/src/tools/mb/mb.py?l=1068 or maybe create a whole new target? (not sure about this part yet) - then I think it should be good to go Another option is to let telemetry handle the isolating of the VM deps as well as the startup/teardown of the VM, but I'd rather we keep these all in one place (ie: //build/config/chromeos/rules.gni and //build/chromeos/run_vm_test.py") to avoid duplication. vm sanity test: - create a dumb test target (maybe "chromeos_vm_sanity_test") https://codesearch.chromium.org/chromium/src/chromeos/BUILD.gn - it won't do much but depend on //chrome - since it's a test, a vm wrapper script will get generated: https://codesearch.chromium.org/chromium/src/testing/test.gni?rcl=0f2bbbfbc10159502a98aebd3e6cf2f44a145c6d&l=360 - then I hardcode special logic in the script that runs the vm sanity test when "--test-exe" is chromeos_vm_sanity_test (needed because there won't be an actual chromeos_vm_sanity_test test binary) https://codesearch.chromium.org/chromium/src/build/chromeos/run_vm_test.py?l=60
,
Apr 26 2018
we definitely want to run the VM sanity test - it captures a good 80% of the chromium CLs that break CrOS. All of these tests will run on the same bot chromeos-amd64-generic-rel-vm-tests, correct? https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel-vm-tests
,
Apr 26 2018
Noted; I'll be sure to get the sanity test up and running. And the chromeos-amd64-generic-rel-vm-tests bot is just my playground for now. Once we're ready to run these tests on the CQ, we'll add the tests to the real amd64-generic simplechrome bot (this one's actually sheriffed): https://ci.chromium.org/buildbot/chromium.chromiumos/chromeos-amd64-generic-rel/ And its CQ bot: https://ci.chromium.org/buildbot/tryserver.chromium.chromiumos/chromeos-amd64-generic-rel/
,
May 2 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/e5e7dc3762a1b18646810303e106da173ac603b5 commit e5e7dc3762a1b18646810303e106da173ac603b5 Author: Ben Pastene <bpastene@chromium.org> Date: Wed May 02 01:42:14 2018 Add a host-side mode of execution for cros vm test runner. This will add a positional arg to run_vm_test.py that can take one of two values: - "host-cmd": Runs a command on the host. Just launches the vm then immediately calls the passed in cmd on the host. It's assumed the cmd will handle data deps, ssh commands, etc. - "vm-test": Runs a test in the vm. Takes care of spawning the vm, pushing the test and its deps to the vm, executing the test, and tears down the vm. (This is the current default behavior.) Bug: 832374 Change-Id: Ib8bd1b2e7dd393f8a4578979fa164381a8342ef2 Reviewed-on: https://chromium-review.googlesource.com/1038747 Commit-Queue: Ben Pastene <bpastene@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Cr-Commit-Position: refs/heads/master@{#555279} [modify] https://crrev.com/e5e7dc3762a1b18646810303e106da173ac603b5/build/chromeos/create_vm_test_script.py [modify] https://crrev.com/e5e7dc3762a1b18646810303e106da173ac603b5/build/chromeos/run_vm_test.py [modify] https://crrev.com/e5e7dc3762a1b18646810303e106da173ac603b5/build/config/chromeos/rules.gni
,
May 3 2018
,
May 4 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/c60bf502e3e92c32c7c297a429563c83db02277e commit c60bf502e3e92c32c7c297a429563c83db02277e Author: Ben Pastene <bpastene@chromium.org> Date: Fri May 04 01:10:33 2018 Create gn target that builds the cros-vm-launcher. This "cros_vm_launcher" target generates a script at build time (located at $out_dir/bin/launch_cros_vm) that you prepend to any other command and it will ensure a cros VM is running for the duration of the command. Also add it as a dependency to telemetry unittests. Invoking it in a bot's tests will come in a later CL. Bug: 828607, 832374 Change-Id: I56ee169c81127bf8cac1114b719f1cdeeb2c3d31 Reviewed-on: https://chromium-review.googlesource.com/1040117 Reviewed-by: Ned Nguyen <nednguyen@google.com> Reviewed-by: Kenneth Russell <kbr@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> Cr-Commit-Position: refs/heads/master@{#555945} [modify] https://crrev.com/c60bf502e3e92c32c7c297a429563c83db02277e/chromeos/BUILD.gn [modify] https://crrev.com/c60bf502e3e92c32c7c297a429563c83db02277e/tools/perf/chrome_telemetry_build/BUILD.gn
,
May 4 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/8ab6954de550aebdf36d72e899910731fe9b6186 commit 8ab6954de550aebdf36d72e899910731fe9b6186 Author: Ben Pastene <bpastene@chromium.org> Date: Fri May 04 04:08:24 2018 Add telemetry_perf_unittests to the cros vm test bot. Will all likely fail until I fix crbug.com/839187 This'll at least get them running. Bug: 836447 , 832374 Change-Id: I1d206ab931230652b4ba3dc8ba519f386a80da96 Reviewed-on: https://chromium-review.googlesource.com/1043258 Commit-Queue: Ben Pastene <bpastene@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Ned Nguyen <nednguyen@google.com> Cr-Commit-Position: refs/heads/master@{#555974} [modify] https://crrev.com/8ab6954de550aebdf36d72e899910731fe9b6186/testing/buildbot/chromium.fyi.json [modify] https://crrev.com/8ab6954de550aebdf36d72e899910731fe9b6186/testing/buildbot/test_suites.pyl [modify] https://crrev.com/8ab6954de550aebdf36d72e899910731fe9b6186/testing/buildbot/waterfalls.pyl [modify] https://crrev.com/8ab6954de550aebdf36d72e899910731fe9b6186/tools/mb/mb.py
,
May 4 2018
telemetry's looking good: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel-vm-tests/1872 However it doesn't seem to run a lot of benchmarks: * Passed: 88 (88 expected, 0 unexpected) * Skipped: 169 (169 expected, 0 unexpected) It must skip a lot on CrOS by default, so probably WAI? Moving onto the sanity test...
,
May 8 2018
,
May 9 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/c55041d7fbb3cc88a500ec407b11ce24d21ed8c6 commit c55041d7fbb3cc88a500ec407b11ce24d21ed8c6 Author: Ben Pastene <bpastene@chromium.org> Date: Wed May 09 20:27:42 2018 Move cros vm tests to their own swarming pool. Should help with keeping the vm image warm in the bots' caches. Currently 20 bots in the pool. Will scale up if needed. TBR=dpranke@chromium.org Bug: 840967, 832374 Change-Id: Ia09fdde0a94d0d798cd1c5073f0651b1ec92ac66 Reviewed-on: https://chromium-review.googlesource.com/1052824 Reviewed-by: Ben Pastene <bpastene@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> Cr-Commit-Position: refs/heads/master@{#557303} [modify] https://crrev.com/c55041d7fbb3cc88a500ec407b11ce24d21ed8c6/testing/buildbot/chromium.fyi.json [modify] https://crrev.com/c55041d7fbb3cc88a500ec407b11ce24d21ed8c6/testing/buildbot/waterfalls.pyl
,
May 11 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/5b0d0dd175c71ae0cc403897ec0a1adbb2844939 commit 5b0d0dd175c71ae0cc403897ec0a1adbb2844939 Author: Achuith Bhandarkar <achuith@chromium.org> Date: Fri May 11 19:10:49 2018 cros_run_vm_test: Support for --host-cmd BUG= chromium:832374 TEST=cros_run_vm_test --host-cmd -- test_that localhost:9222 login_Cryptohome Fail cases: cros_run_vm_test test_that localhost:9222 login_Cryptohome cros_run_vm_test --host-cmd test_that localhost:9222 login_Cryptohome cros_run_vm_test --host-cmd Change-Id: I32875d59bd61f7dfdd34795bdc0322984c050d0f Reviewed-on: https://chromium-review.googlesource.com/1053099 Commit-Ready: Achuith Bhandarkar <achuith@chromium.org> Tested-by: Achuith Bhandarkar <achuith@chromium.org> Reviewed-by: Achuith Bhandarkar <achuith@chromium.org> Reviewed-by: Ben Pastene <bpastene@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> [modify] https://crrev.com/5b0d0dd175c71ae0cc403897ec0a1adbb2844939/scripts/cros_run_vm_test.py
,
May 11 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/f812c075051dd302e9bae4572715bce9a78f8bd9 commit f812c075051dd302e9bae4572715bce9a78f8bd9 Author: chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Date: Fri May 11 21:16:45 2018 Roll src/third_party/chromite/ 746c45d5d..7657336de (3 commits) https://chromium.googlesource.com/chromiumos/chromite.git/+log/746c45d5d8b1..7657336de212 $ git log 746c45d5d..7657336de --date=short --no-merges --format='%ad %ae %s' 2018-05-08 mikenichols [chromite] Capture job_name and git timing for ts_mon in cbuildbot runs. 2018-05-10 khmel arc: Retry copying resource in case race condition for android uprev 2018-05-09 achuith cros_run_vm_test: Support for --host-cmd Created with: roll-dep src/third_party/chromite BUG= chromium:829289 , chromium:841937 , chromium:832374 The AutoRoll server is located here: https://chromite-chromium-roll.skia.org Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, please contact the current sheriff, who should be CC'd on the roll, and stop the roller if necessary. TBR=chrome-os-gardeners@chromium.org Change-Id: I7712f93eb43ad2661af0fe11a71a9b228ce916ad Reviewed-on: https://chromium-review.googlesource.com/1055816 Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Cr-Commit-Position: refs/heads/master@{#558020} [modify] https://crrev.com/f812c075051dd302e9bae4572715bce9a78f8bd9/DEPS
,
May 14 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/83b500c126930271f4da9c246c768d61ada04a84 commit 83b500c126930271f4da9c246c768d61ada04a84 Author: Ben Pastene <bpastene@chromium.org> Date: Mon May 14 18:00:28 2018 Add a sanity-test mode to the cros vm test runner. When a test command is specified, the run_cros_vm_test will run the vm sanity test. This test is baked into the VM and smoke checks the system browser by loading empty pages and running some simple javascript. Since it depends on the system browser, we need to first deploy a locally built Chrome to the VM (via the --deploy arg). The deploy script needs to know the gn args that built the browser to know which libs to copy over. Since we don't have the full set of args at test-time, give it a dummy set of args to trigger the default deploying behavior. Bug: 832374 Change-Id: I84f7fdd94672fcb220eaa4d9625395c1d0ab7636 Reviewed-on: https://chromium-review.googlesource.com/1050981 Commit-Queue: Ben Pastene <bpastene@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Cr-Commit-Position: refs/heads/master@{#558372} [modify] https://crrev.com/83b500c126930271f4da9c246c768d61ada04a84/build/chromeos/run_vm_test.py
,
May 14 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/9ec70f930a6c40ee7f5a3ac56418de2718593392 commit 9ec70f930a6c40ee7f5a3ac56418de2718593392 Author: Ben Pastene <bpastene@chromium.org> Date: Mon May 14 21:09:22 2018 Add GN target for building the cros VM sanity test. The actual test binary is actually baked into the VM image. The gn target here just builds chrome and a wrapper script around the test runner. Bug: 832374 Change-Id: I062c80966abb6a234511ba125cdfb5c86367d9ad Reviewed-on: https://chromium-review.googlesource.com/1057961 Commit-Queue: Ben Pastene <bpastene@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Cr-Commit-Position: refs/heads/master@{#558459} [modify] https://crrev.com/9ec70f930a6c40ee7f5a3ac56418de2718593392/build/config/chromeos/rules.gni [modify] https://crrev.com/9ec70f930a6c40ee7f5a3ac56418de2718593392/chromeos/BUILD.gn
,
May 15 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/9cba3376c389e164353cf417e70b926b86eea095 commit 9cba3376c389e164353cf417e70b926b86eea095 Author: Ben Pastene <bpastene@chromium.org> Date: Tue May 15 03:45:05 2018 Run the cros vm sanity test on the fyi cros vm bot. Bug: 832374 Change-Id: I13d52c97809ec4da1e2f2987d0b76c9d9be91810 Reviewed-on: https://chromium-review.googlesource.com/1058507 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> Cr-Commit-Position: refs/heads/master@{#558602} [modify] https://crrev.com/9cba3376c389e164353cf417e70b926b86eea095/testing/buildbot/chromium.fyi.json [modify] https://crrev.com/9cba3376c389e164353cf417e70b926b86eea095/testing/buildbot/gn_isolate_map.pyl [modify] https://crrev.com/9cba3376c389e164353cf417e70b926b86eea095/testing/buildbot/test_suites.pyl
,
May 16 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/39f23e84c22a9815c1537e05aeb38490ef1d6ad9 commit 39f23e84c22a9815c1537e05aeb38490ef1d6ad9 Author: Ben Pastene <bpastene@chromium.org> Date: Wed May 16 02:13:18 2018 Create a simple json results file when running the cros VM sanity test. This uses build/android's gtest result lib. The results will contain only a single test, and will either be PASS or FAIL depending on the return code. Bug: 832374 Change-Id: I7294206340b17fda3f9f93a5085cd95b9b5faceb Reviewed-on: https://chromium-review.googlesource.com/1060023 Reviewed-by: John Budorick <jbudorick@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> Cr-Commit-Position: refs/heads/master@{#558929} [modify] https://crrev.com/39f23e84c22a9815c1537e05aeb38490ef1d6ad9/build/chromeos/run_vm_test.py [modify] https://crrev.com/39f23e84c22a9815c1537e05aeb38490ef1d6ad9/build/config/chromeos/rules.gni
,
May 16 2018
Sanity test is up and running on the fyi bot: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel-vm-tests/2320 The flakiness dashboard is even tracking it: https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=cros_vm_sanity_test Although, passing runs are white w/ a question mark... That's weird. I woulda expected them to be colored green. I'll need to look into that. Maybe I got something wrong. And we've had one sanity test failure since turning it on: https://chromium-swarm.appspot.com/user/task/3d8297ad222d8d10 Maybe some type of flake? I see "ERROR:root:Failed with DevToolsClientConnectionError while starting the browser backend." in the logs. We'll probably need to sort that out before turning the test on in the CQ.
,
May 17 2018
Did a little playing around with base_unittests. We're seeing the following failures on that suite: FileUtilTest.ChangeFilePermissionsAndWrite ChromeOSMemoryPressureMonitorTest.CheckMemoryPressure FileUtilTest.ChangeDirectoryPermissionsAndEnumerate FileUtilTest.ChangeFilePermissionsAndRead TaskSchedulerWorkerTest.BumpPriorityOfAliveThreadDuringShutdown Turns out that all the file permission tests fail if we're running as root in the VM, but pass if we're running as chronos. Which is a little strange. I honestly don't know if the proper fix is to get the tests compatible with running as root, or not running them as root... ChromeOSMemoryPressureMonitorTest.CheckMemoryPressure still fails however, which is also a little strange, since that test looks to be explicitly targeted for chromeos.
,
May 17 2018
,
May 17 2018
Should be straightforward to run as chronos using sudo -u chronos, right? That seems like a good approach? I'm guessing the test memory pressure monitor is somehow interacting with the chromeos system memory pressure monitor.
,
May 17 2018
Yeah, I think we'd want to add the option to run as non-root. We'd have to make sure that all the files pushed to the VM are readable/writable/executable as the new user. Might be best to add that to cros_run_vm_test. Maybe add an --as-user option? And the memory pressure test is beyond me, so I filed bug 844102 for it.
,
May 17 2018
Sure, sg. I'll take a look.
,
May 18 2018
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/config/+/61f8c2175d506421d95472da33de8764c2299dda commit 61f8c2175d506421d95472da33de8764c2299dda Author: Ben Pastene <bpastene@chromium.org> Date: Fri May 18 00:44:29 2018
,
May 18 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/2b1ce59023459ba15057ad15320094f73088c20e commit 2b1ce59023459ba15057ad15320094f73088c20e Author: Ben Pastene <bpastene@chromium.org> Date: Fri May 18 17:42:29 2018 Enable the CrOS sanity test on the amd64-generic CQ bot as an experiment Since it's 100% experiment, it won't block CLs if it fails. (In theory, at least.) https://chrome-internal-review.googlesource.com/c/infradata/config/+/628212 increases test pool size in preparation for this. Bug: 832374 Change-Id: Ie520e3cb75aec5c0df9ad497daa1cc72321c52f0 Reviewed-on: https://chromium-review.googlesource.com/1065198 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Achuith Bhandarkar <achuith@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> Cr-Commit-Position: refs/heads/master@{#559943} [modify] https://crrev.com/2b1ce59023459ba15057ad15320094f73088c20e/testing/buildbot/chromium.chromiumos.json [modify] https://crrev.com/2b1ce59023459ba15057ad15320094f73088c20e/testing/buildbot/test_suites.pyl
,
May 18 2018
We've had 100+ sanity test runs so far on the CQ, all passing: https://chromium-swarm.appspot.com/tasklist?c=name&c=state&c=created_ts&c=duration&c=pending_time&c=pool&c=bot&et=1526673300000&f=name-tag%3Acros_vm_sanity_test&f=buildername-tag%3Achromeos-amd64-generic-rel&f=state%3ACOMPLETED_FAILURE&l=50&n=true&q=state&s=created_ts%3Adesc&st=1526586900000 I'll plan on taking it out of experimental next week if it still looks good.
,
May 18 2018
Although... the sanity test is the first test on the bot to have a dependency on chrome itself, so now compile/test times have shot-up. The slave pool is a bit small for the new load: http://shortn/_3OXNwpAJXr I'll add some more bots.
,
May 18 2018
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/config/+/a9cc4d5256b4559ebef5cdf3749ef9d2314bea5f commit a9cc4d5256b4559ebef5cdf3749ef9d2314bea5f Author: Ben Pastene <bpastene@chromium.org> Date: Fri May 18 20:15:30 2018
,
May 19 2018
Looks like the sanity test (like telemetry) can hang indefinitely every now and then without any output and eventually time out: https://chromium-swarm.appspot.com/task?id=3d8d95ebeb998410 https://chromium-swarm.appspot.com/task?id=3d8da9134094f510 https://chromium-swarm.appspot.com/task?id=3d8dd43c1f4c3910 Managed to hop on a bot while it was doing this. SSH'ed into the vm and the sanity test was alive. strace'ing it seemed to indicate that it was doing *something*, but I'm not the best at deciphering syscalls. The chrome logs had some interesting errors tho: localhost ~ # cat /var/log/chrome/chrome | grep ERROR [2635:2635:0518/164212.961158:ERROR:vaapi_wrapper.cc(313)] vaInitialize failed: unknown libva error [2589:2589:0518/164213.027230:ERROR:input_method_manager_impl.cc(1080)] IMEEngine for "fgoepimhcoialccpbmpnnblemnepkkao" is not registered [2635:2635:0518/164213.055287:ERROR:sandbox_linux.cc(378)] InitializeSandbox() called with multiple threads in process gpu-process. This error can be safely ignored in VMTests. [2589:2589:0518/164213.475956:ERROR:object_proxy.cc(616)] Failed to call method: org.chromium.PowerManager.GetScreenBrightnessPercent: object_path= /org/chromium/PowerManager: org.freedesktop.DBus.Error.NotSupported: Backlight uninitialized [2635:2651:0518/164214.303108:ERROR:drm_device.cc(751)] CTM is empty. Expected a 3x3 matrix. [2635:2651:0518/164214.303918:ERROR:drm_display.cc(181)] Failed to set color correction for display: crtc_id = 27 Looks like most of those are display/screen related things, so maybe not relevant? The task timed out before I could dig in further, so I'll bump the timeout and give it another shot the next time I find one.
,
May 21 2018
This style of test (though not this exact one) runs in the CQ/PFQ/waterfall hundreds of times a day, so it ought to be stable. The chrome errors are all bogus unfortunately :/ We should know more about what's going on with streaming output once I've enabled that for vm_sanity. Unfortunately that CL is stuck behind the refactor CL; hopefully we can get both squared away soon.
,
May 21 2018
Noted. It's a very small subset of our test runs that seem to hang (< 1%). And it's not just confined to the sanity test, telemetry_perf_unittests can sometimes hang in the exact same way: https://chromium-swarm.appspot.com/task?id=3d9c1da69e0f4110 https://chromium-swarm.appspot.com/task?id=3d9bb87871cf8d10 Still keeping an eye out for an reoccurrence.
,
May 22 2018
Another thing that we can do is add a retry (or more than 1 retry) to the test which will almost certainly mask the problem. It does improve the possibility of flaky failures going through, and not detecting an increase in the flakiness of this particular phenomenon, etc.
,
May 23 2018
Found a hung telemetry test. From scanning the process tree of the test on host, it looked like it was trying to restart the ui on the VM. The ui logs on the VM has an interesting looking stack trace. I also found a hung sanity test, and its ui logs had the same stack trace. Seems to indicate that this might be relevant. Any ideas? localhost ~ # cat /var/log/ui/ui.LATEST libva error: va_getDriverName() failed with unknown libva error,driver_name=(null) [2518:2518:0523/102800.481656:ERROR:vaapi_wrapper.cc(313)] vaInitialize failed: unknown libva error libEGL warning: MESA-LOADER: failed to retrieve device information libEGL warning: DRI2: failed to open vgem (search paths /usr/lib64/dri) libEGL warning: MESA-LOADER: failed to retrieve device information libEGL warning: DRI2: failed to open virtio_gpu (search paths /usr/lib64/dri) libEGL warning: No hardware driver found, falling back to software rendering [2518:2518:0523/102800.592065:ERROR:sandbox_linux.cc(378)] InitializeSandbox() called with multiple threads in process gpu-process. This error can be safely ignored in VMTests. [2477:2477:0523/102800.650649:ERROR:input_method_manager_impl.cc(1080)] IMEEngine for "fgoepimhcoialccpbmpnnblemnepkkao" is not registered DevTools listening on ws://127.0.0.1:37832/devtools/browser/c9ba565b-b9d5-4adf-9b2c-771827a848b5 device-enumerator: scan all dirs device-enumerator: scanning /sys/bus device-enumerator: scanning /sys/class device-enumerator: scan all dirs device-enumerator: scanning /sys/bus device-enumerator: scanning /sys/class [2477:2477:0523/102801.065581:ERROR:object_proxy.cc(616)] Failed to call method: org.chromium.PowerManager.GetScreenBrightnessPercent: object_path= /org/chromium/PowerManager: org.freedesktop.DBus.Error.NotSupported: Backlight uninitialized device-enumerator: scan all dirs device-enumerator: scanning /sys/bus device-enumerator: scanning /sys/class [2518:2532:0523/102802.205807:ERROR:drm_device.cc(751)] CTM is empty. Expected a 3x3 matrix. [2518:2532:0523/102802.205924:ERROR:drm_display.cc(181)] Failed to set color correction for display: crtc_id = 27 [2477:2477:0523/102811.093821:ERROR:object_proxy.cc(616)] Failed to call method: org.chromium.CryptohomeInterface.RemoveFirmwareManagementParameters: object_path= /org/chromium/Cryptohome: org.freedesktop.DBus.Error.ServiceUnknown: The name org.chromium.Cryptohome was not provided by any .service files [2477:2477:0523/102811.094524:FATAL:optional.h(583)] Check failed: storage_.is_populated_. #0 0x5939bb43c1ec <unknown> #1 0x5939bb3c8f80 <unknown> #2 0x5939ba532cb8 <unknown> #3 0x5939ba51da35 <unknown> #4 0x5939bbdb51a7 <unknown> #5 0x5939b996f24e <unknown> #6 0x5939bbdd2d88 <unknown> #7 0x5939bbdd5853 <unknown> #8 0x5939bbdd3156 <unknown> #9 0x5939bbdd60b1 <unknown> #10 0x5939bb412839 <unknown> #11 0x5939bb412a72 <unknown> #12 0x5939bb3b6fcc <unknown> #13 0x5939bb3ce997 <unknown> #14 0x5939bb3cee77 <unknown> #15 0x5939bb44cf09 <unknown> #16 0x5939bb3eec65 <unknown> #17 0x5939bb0c7858 <unknown> #18 0x5939b9a7b574 <unknown> #19 0x5939b9a7dde2 <unknown> #20 0x5939b9a77bc1 <unknown> #21 0x5939bb0b5654 <unknown> #22 0x5939bb0be2ca <unknown> #23 0x5939bb0b3511 <unknown> #24 0x5939b8f87b1f <unknown> #25 0x782cb0134736 __libc_start_main #26 0x5939b8f87949 <unknown> Received signal 6 #0 0x5939bb43c1ec <unknown> #1 0x5939bb43bd51 <unknown> #2 0x782cb0d622e0 <unknown> #3 0x782cb0147dd2 gsignal #4 0x782cb0149bf6 abort #5 0x5939bb43aae5 <unknown> #6 0x5939bb3c93b6 <unknown> #7 0x5939ba532cb8 <unknown> #8 0x5939ba51da35 <unknown> #9 0x5939bbdb51a7 <unknown> #10 0x5939b996f24e <unknown> #11 0x5939bbdd2d88 <unknown> #12 0x5939bbdd5853 <unknown> #13 0x5939bbdd3156 <unknown> #14 0x5939bbdd60b1 <unknown> #15 0x5939bb412839 <unknown> #16 0x5939bb412a72 <unknown> #17 0x5939bb3b6fcc <unknown> #18 0x5939bb3ce997 <unknown> #19 0x5939bb3cee77 <unknown> #20 0x5939bb44cf09 <unknown> #21 0x5939bb3eec65 <unknown> #22 0x5939bb0c7858 <unknown> #23 0x5939b9a7b574 <unknown> #24 0x5939b9a7dde2 <unknown> #25 0x5939b9a77bc1 <unknown> #26 0x5939bb0b5654 <unknown> #27 0x5939bb0be2ca <unknown> #28 0x5939bb0b3511 <unknown> #29 0x5939b8f87b1f <unknown> #30 0x782cb0134736 __libc_start_main #31 0x5939b8f87949 <unknown> r8: 0000000000000000 r9: 0000782cafc0c740 r10: 0000000000000008 r11: 0000000000000202 r12: 00007ffe1f3ab459 r13: 00007ffe1f3aba00 r14: 00007ffe1f3ab490 r15: 00007ffe1f3ab458 di: 00000000000009ad si: 00000000000009ad bp: 00007ffe1f3ab420 bx: 00007ffe1f3ab490 dx: 0000000000000006 ax: 0000000000000000 cx: 0000782cb0147dd2 sp: 00007ffe1f3ab2f8 ip: 0000782cb0147dd2 efl: 0000000000000202 cgf: 0000000000000033 erf: 0000000000000000 trp: 0000000000000000 msk: 0000000000000000 cr2: 0000000000000000 [end of stack trace] Calling _exit(1). Core file will not be generated. libva error: va_getDriverName() failed with unknown libva error,driver_name=(null) [2762:2762:0523/102811.781412:ERROR:vaapi_wrapper.cc(313)] vaInitialize failed: unknown libva error libEGL warning: MESA-LOADER: failed to retrieve device information libEGL warning: DRI2: failed to open vgem (search paths /usr/lib64/dri) libEGL warning: MESA-LOADER: failed to retrieve device information libEGL warning: DRI2: failed to open virtio_gpu (search paths /usr/lib64/dri) libEGL warning: No hardware driver found, falling back to software rendering [2762:2762:0523/102811.899753:ERROR:sandbox_linux.cc(378)] InitializeSandbox() called with multiple threads in process gpu-process. This error can be safely ignored in VMTests. [2709:2709:0523/102811.914590:ERROR:input_method_manager_impl.cc(1080)] IMEEngine for "fgoepimhcoialccpbmpnnblemnepkkao" is not registered DevTools listening on ws://127.0.0.1:33163/devtools/browser/1425a412-1baf-47b6-afa4-e2d31afa0dc9 device-enumerator: scan all dirs device-enumerator: scanning /sys/bus device-enumerator: scanning /sys/class device-enumerator: scan all dirs device-enumerator: scanning /sys/bus device-enumerator: scanning /sys/class [2709:2709:0523/102812.633663:ERROR:object_proxy.cc(616)] Failed to call method: org.chromium.PowerManager.GetScreenBrightnessPercent: object_path= /org/chromium/PowerManager: org.freedesktop.DBus.Error.NotSupported: Backlight uninitialized [2762:2776:0523/102813.836394:ERROR:drm_device.cc(751)] CTM is empty. Expected a 3x3 matrix. [2762:2776:0523/102813.836512:ERROR:drm_display.cc(181)] Failed to set color correction for display: crtc_id = 27 device-enumerator: scan all dirs device-enumerator: scanning /sys/bus device-enumerator: scanning /sys/class
,
May 23 2018
Also, I wonder if not stripping the symbols when deploying chrome to the VM would help with the unsymbolized stack trace. 'Cause all those "unknowns" aren't very helpful.
,
May 23 2018
The most useful thing would be to know where exactly in the sequence we're failing, and we won't know that without the stdout of vm_sanity :/ You can use --no-strip, but I believe the binary becomes enormous. We don't have space on the root partition for this binary, but I think deploy knows how to copy the file to a different partition and set up the symlinks. You can try it?
,
May 25 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/3bd6ea8c07ab4ae998354ed0d07615efbe21129c commit 3bd6ea8c07ab4ae998354ed0d07615efbe21129c Author: Ben Pastene <bpastene@chromium.org> Date: Fri May 25 19:06:47 2018 Bump execution and IO timeout to 60min on cros vm FYI tester. This'll give me more time to debug hung VM tasks. As it is, 20min isn't really enough time for my ssh'ing to yield much results. TBR=dpranke@chromium.org Bug: 832374 Change-Id: I58f9b065190c1029c495756d914304abcca7b5a0 Reviewed-on: https://chromium-review.googlesource.com/1070450 Reviewed-by: Ben Pastene <bpastene@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> Cr-Commit-Position: refs/heads/master@{#561963} [modify] https://crrev.com/3bd6ea8c07ab4ae998354ed0d07615efbe21129c/testing/buildbot/chromium.fyi.json [modify] https://crrev.com/3bd6ea8c07ab4ae998354ed0d07615efbe21129c/testing/buildbot/waterfalls.pyl
,
May 29 2018
It looks like the flake rate of the sanity test jumped significantly sometime over the weekend: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel?limit=200 Previously it was failing around 1% of tests, but now it's around 15-20%. It seems to be generally caused by browser/devtools startup timeouts. I wonder what changed...
,
May 29 2018
Yup, and I've seen more than one failure stack.
,
May 30 2018
Using the nostrip + mount option like you suggested, I managed to get a more symbolized version of the trace in #46: [2695:2695:0529/180758.401359:ERROR:object_proxy.cc(616)] Failed to call method: org.chromium.CryptohomeInterface.RemoveFirmwareManagementParameters: object_path= /org/chromium/Cryptohome: org.freedesktop.DBus.Error.ServiceUnknown: The name org.chromium.Cryptohome was not provided by any .service files [2695:2695:0529/180758.401911:FATAL:optional.h(583)] Check failed: storage_.is_populated_. #0 0x557cbc98070c base::debug::StackTrace::StackTrace() #1 0x557cbc90e930 logging::LogMessage::~LogMessage() #2 0x557cbba60bd8 chromeos::AutoEnrollmentController::OnFirmwareManagementParametersRemoved() #3 0x557cbba4b805 _ZN4base8internal7InvokerINS0_9BindStateIMN8chromeos26EasyUnlockGetKeysOperationEFvNS_8OptionalIN10cryptohome9BaseReplyEEEEJNS_7WeakPtrIS4_EEEEEFvS8_EE7RunOnceEPNS0_13BindStateBaseEOS8_ #4 0x557cbd2dcf57 chromeos::(anonymous namespace)::CryptohomeClientImpl::OnBaseReplyMethod() #5 0x557cbae9473e _ZN4base8internal7InvokerINS0_9BindStateIMN8download19InProgressCacheImplEFvNS_12OnceCallbackIFvvEEERKNSt3__16vectorIcNS8_9allocatorIcEEEEEJNS_7WeakPtrIS4_EES7_EEEFvSE_EE7RunOnceEPNS0_13BindStateBaseESE_ #6 0x557cbd2fab68 dbus::ObjectProxy::OnCallMethod() #7 0x557cbd2fd633 _ZN4base8internal7InvokerINS0_9BindStateIMN4dbus11ObjectProxyEFvRKNSt3__112basic_stringIcNS5_11char_traitsIcEENS5_9allocatorIcEEEESD_NS_12OnceCallbackIFvPNS3_8ResponseEEEESG_PNS3_13ErrorResponseEEJ13scoped_refptrIS4_ESB_SB_SI_EEEFvSG_SK_EE7RunOnceEPNS0_13BindStateBaseESG_SK_ #8 0x557cbd2faf36 dbus::ObjectProxy::RunResponseOrErrorCallback() #9 0x557cbd2fde91 _ZN4base8internal7InvokerINS0_9BindStateIMN4dbus11ObjectProxyEFvNS4_19ReplyCallbackHolderENS_9TimeTicksEPNS3_8ResponseEPNS3_13ErrorResponseEEJ13scoped_refptrIS4_ES5_S6_S8_SA_EEEFvvEE7RunOnceEPNS0_13BindStateBaseE #10 0x557cbc958419 base::(anonymous namespace)::PostTaskAndReplyRelay::RunTaskAndPostReply() #11 0x557cbc958652 _ZN4base8internal7InvokerINS0_9BindStateIPFvNS_12_GLOBAL__N_121PostTaskAndReplyRelayEEJS4_EEEFvvEE7RunOnceEPNS0_13BindStateBaseE #12 0x557cbc8fc97c base::debug::TaskAnnotator::RunTask() #13 0x557cbc914357 base::MessageLoop::RunTask() #14 0x557cbc914837 base::MessageLoop::DoWork() #15 0x557cbc991429 base::MessagePumpLibevent::Run() #16 0x557cbc9346c5 base::RunLoop::Run() #17 0x557cbc607ac8 ChromeBrowserMainParts::MainMessageLoopRun() #18 0x557cbafa1174 content::BrowserMainLoop::RunMainMessageLoopParts() #19 0x557cbafa3a72 content::BrowserMainRunnerImpl::Run() #20 0x557cbaf9d761 content::BrowserMain() #21 0x557cbc5f58a4 content::ContentMainRunnerImpl::Run() #22 0x557cbc5fe51a service_manager::Main() #23 0x557cbc5f3761 content::ContentMain() #24 0x557cba4a449f ChromeMain #25 0x7c86dcf44736 __libc_start_main #26 0x557cba4a42c9 _start Looks like OnFirmwareManagementParametersRemoved() is dereferencing a null/empty pointer. Poking around, I found https://chromium-review.googlesource.com/c/chromium/src/+/1072718 which landed earlier today and removed a pointer dereference in that method. Maybe that'll help things? Let's wait and see...
,
Jun 2 2018
Prob fair to say that this bug has outgrown its original intent. The fyi bot (where I'm putting tests that I'm still working on) is up and running at: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel-vm-tests So I'll close this out and file other bugs to track all the issues I've mentioned previously above.
,
Jun 11 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/cdd784010f13bcf6bda05887ef458eb3d937631c commit cdd784010f13bcf6bda05887ef458eb3d937631c Author: Ben Pastene <bpastene@chromium.org> Date: Mon Jun 11 20:44:34 2018 cros_run_vm_test: Add option that runs the test as user chronos. Some of chromium's tests fail if the test is run as root, but pass if they run as a less-privileged user. This adds an option to run the test as chronos. This temporarily changes the username field in the RemoteDevice: https://codesearch.chromium.org/chromium/src/third_party/chromite/lib/remote_access.py?rcl=a05ffccfb494de4c72d52b13b3bc53bdcf81a562&l=271 And changes it back to root after the test. BUG= chromium:832374 TEST=./bin/cros_run_vm_test --start --as-chronos --cmd -- "whoami ; whoami && whoami" Change-Id: I46ee3fe0f7c2c800483cc2392cc8cc795621e243 Reviewed-on: https://chromium-review.googlesource.com/1079644 Commit-Ready: Ben Pastene <bpastene@chromium.org> Tested-by: Ben Pastene <bpastene@chromium.org> Reviewed-by: Achuith Bhandarkar <achuith@chromium.org> [modify] https://crrev.com/cdd784010f13bcf6bda05887ef458eb3d937631c/lib/remote_access.py [modify] https://crrev.com/cdd784010f13bcf6bda05887ef458eb3d937631c/scripts/cros_run_vm_test.py |
||||||||
►
Sign in to add a comment |
|||||||||||
Comment 1 by bpastene@chromium.org
, Apr 13 2018Labels: -Pri-3 Pri-1