CrOS: Add Vulkan CTS to daily testing |
||||||||
Issue description
In preparation for enabling Vulkan on CrOS, let's begin validating the system's readiness: the readiness of the driver itself as well as system integration. A great first step is to add the Vulkan CTS to daily testing for select boards.
I propose we first focus on Intel boards with Skylake and Apollo Lake GPUs, as the kernels on those boards are recent enough to support Vulkan.
Pass Criteria:
- Autotest contains a control file (graphics_dEQP/control.vk-master)
for running the Vulkan tests in the Android CTS's mustpass list
(see $DEQP_TOP/android/cts/master/vk-master.txt).
- At least one board includes graphics_dEQP/control.vk-master in
its daily test run.
,
Feb 6 2017
,
Feb 7 2017
This sounds very promising!
,
Feb 8 2017
I submitted all the patches on Monday. https://chromium-review.googlesource.com/c/438746/ https://chromium-review.googlesource.com/c/438747/ https://chromium-review.googlesource.com/c/438748/ https://chromium-review.googlesource.com/c/438766/ https://chromium-review.googlesource.com/c/438706/ https://chromium-review.googlesource.com/c/438705/
,
Feb 11 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/cb9045b9f04b786d843359796f9ee6de2cb8ca80 commit cb9045b9f04b786d843359796f9ee6de2cb8ca80 Author: Chad Versace <chadversary@chromium.org> Date: Sat Feb 11 03:02:56 2017 media-libs/vulkan-loader: Update to 1.0.39.0 Upstream Gentoo finally has an ebuild for the vulkan-loader, media-libs/vulkan-1.0.39.0.ebuild. This patch imports the upstream ebuild and applies some fixes for Chromium OS. Changes from upstream ebuild: - Downgrade EAPI from 6 to 5. (CrOS doesn't yet support 6). - Replace eclass cmake-multilib with cmake-utils. (CrOS doesn't yet have cmake-multilib). - Disable WSI support because CrOS has no window system. - Remove python3 from DEPENDS because it breaks cross-compilation. The vulkan-loader requires python>=3.3 as a *host* build dependency, not as a *target* build dependency. That dependency is satisfied by host package virtual/target-chromium-os-sdk instead of DEPENDS. - Unmask all architectures. I tested emerging this package to an ARM board (elm) and an Intel board (sentry). BUG= chromium:689177 BUG= chromium:642954 TEST=$ emerge-sentry =media-libs/vulkan-loader-1.0.39.0 TEST=$ emerge-elm =media-libs/vulkan-loader-1.0.39.0 Change-Id: I02b8930d784ffa4dd1215cb3c2e0507a7465ad32 Reviewed-on: https://chromium-review.googlesource.com/438746 Commit-Ready: Chad Versace <chadversary@chromium.org> Tested-by: Chad Versace <chadversary@chromium.org> Reviewed-by: Chad Versace <chadversary@chromium.org> [delete] https://crrev.com/dd276f7d0a1c8bacd98d685858c1b4f6fea76549/media-libs/vulkan-loader/vulkan-loader-1.0.24.0.ebuild [add] https://crrev.com/cb9045b9f04b786d843359796f9ee6de2cb8ca80/media-libs/vulkan-loader/metadata.xml [modify] https://crrev.com/cb9045b9f04b786d843359796f9ee6de2cb8ca80/media-libs/vulkan-loader/Manifest [add] https://crrev.com/cb9045b9f04b786d843359796f9ee6de2cb8ca80/media-libs/vulkan-loader/vulkan-loader-1.0.39.0.ebuild
,
Feb 11 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/513081ba6ae000bd838a4bbfbcb51007a2bb311b commit 513081ba6ae000bd838a4bbfbcb51007a2bb311b Author: Chad Versace <chadversary@chromium.org> Date: Sat Feb 11 03:02:56 2017 virtual/vulkan-icd: New ebuild This package indicates the presence of a Vulkan ICD. We need this ebuild because libvulkan.so (provided by media-libs/vulkan-loader) does not provide a Vulkan implementation. It is merely a loader library for the ICDs, which provide the actual implementation. We should probably upstream this ebuild to Gentoo. BUG= chromium:689177 TEST=Confirm that `USE=vulkan emerge-sentry virtual/vulkan-icd` installs deps media-libs/vulkan-loader and media-libs/mesa[vulkan]. Change-Id: I29c0c78ee5e016ec5b2c1a08d758382a35ebc567 Reviewed-on: https://chromium-review.googlesource.com/438747 Commit-Ready: Chad Versace <chadversary@chromium.org> Tested-by: Chad Versace <chadversary@chromium.org> Reviewed-by: Nicolas Boichat <drinkcat@chromium.org> [add] https://crrev.com/513081ba6ae000bd838a4bbfbcb51007a2bb311b/virtual/vulkan-icd/vulkan-icd-1.ebuild
,
Feb 13 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/d1d2e933735659c544345b7ed4723351056e4018 commit d1d2e933735659c544345b7ed4723351056e4018 Author: Chad Versace <chadversary@chromium.org> Date: Mon Feb 13 23:36:30 2017 chipset-apl: Enable Vulkan in Mesa and dEQP Needed to run the Vulkan CTS. This enables us to begin testing Vulkan before we begin using it in production. BUG= chromium:689177 TEST=Check that USE flags contain 'vulkan' in `equery-reef uses mesa deqp`. Then check build with `emerge-reef mesa deqp`. Depends-On: I10233e70cdf2303ace300bef39896e98f2d7636b Change-Id: Iacddedea3c717f9a6ed802fe9032dc89781bd639 Reviewed-on: https://chromium-review.googlesource.com/438705 Commit-Ready: Chad Versace <chadversary@chromium.org> Tested-by: Chad Versace <chadversary@chromium.org> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> [add] https://crrev.com/d1d2e933735659c544345b7ed4723351056e4018/chipset-apl/profiles/base/package.use
,
Feb 13 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/056da6c9930c7c243fce39edf3c63813a5072e34 commit 056da6c9930c7c243fce39edf3c63813a5072e34 Author: Chad Versace <chadversary@chromium.org> Date: Mon Feb 13 23:36:30 2017 chipset-skl: Enable Vulkan in Mesa and dEQP Needed to run the Vulkan CTS. This enables us to begin testing Vulkan before we begin using it in production. BUG= chromium:689177 TEST=Check that USE flags contain 'vulkan' in `equery-sentry uses mesa deqp`. Then check build with `emerge-sentry mesa deqp`. Depends-On: I10233e70cdf2303ace300bef39896e98f2d7636b Change-Id: I03f861aca66486a46717b368b4f815df53e2806d Reviewed-on: https://chromium-review.googlesource.com/438706 Commit-Ready: Chad Versace <chadversary@chromium.org> Tested-by: Chad Versace <chadversary@chromium.org> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> [add] https://crrev.com/056da6c9930c7c243fce39edf3c63813a5072e34/chipset-skl/profiles/base/package.use
,
Feb 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/1a0b9d49491da8e5399563183e014b1f51f8edf9 commit 1a0b9d49491da8e5399563183e014b1f51f8edf9 Author: Chad Versace <chadversary@chromium.org> Date: Tue Feb 14 22:50:39 2017 media-gfx/deqp: Build Vulkan CTS Recent versions of dEQP include the official Khronos Vulkan CTS. Build it if USE=vulkan. Chromium OS uses a custom dEQP platform, the "surfaceless" platform. This patch implements Vulkan support for that platform by copying upstream's Vulkan implementation for X11, then stripping out the non-essential bits. BUG= chromium:689177 TEST=$ USE=vulkan emerge-senty deqp $ cros deploy $SENTRY_IP deqp mesa vulkan-loader $ ssh $SENTRY_IP /usr/local/deqp/external/vulkancts/modules/vulkan/deqp-vk --deqp-case='dEQP-VK.api.smoke.*' # Confirm 6/6 tests pass. Depends-On: I29c0c78ee5e016ec5b2c1a08d758382a35ebc567 Depends-On: I02b8930d784ffa4dd1215cb3c2e0507a7465ad32 Change-Id: I10233e70cdf2303ace300bef39896e98f2d7636b Reviewed-on: https://chromium-review.googlesource.com/438748 Commit-Ready: Ilja H. Friedel <ihf@chromium.org> Tested-by: Chad Versace <chadversary@chromium.org> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> Reviewed-by: Stéphane Marchesin <marcheu@chromium.org> [modify] https://crrev.com/1a0b9d49491da8e5399563183e014b1f51f8edf9/media-gfx/deqp/Manifest [add] https://crrev.com/1a0b9d49491da8e5399563183e014b1f51f8edf9/media-gfx/deqp/files/0005-next-Add-Vulkan-support-to-tcu-surfaceless-Platform.patch [rename] https://crrev.com/1a0b9d49491da8e5399563183e014b1f51f8edf9/media-gfx/deqp/deqp-2017.02.08-r2.ebuild
,
Feb 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/360a64ab4f40744e63f467409d9252f39516d0d0 commit 360a64ab4f40744e63f467409d9252f39516d0d0 Author: Chad Versace <chadversary@chromium.org> Date: Tue Feb 14 22:50:40 2017 graphics_dEQP: Add Vulkan CTS Recent versions of dEQP include the Vulkan CTS. Add the following control files for it: control.vk-master # The mustpass list from the Android CTS control.vk-master.hasty control.vk.api.hasty control.vk.api.smoke control.vk.binding_model.hasty control.vk.glsl.hasty control.vk.info control.vk.pipeline.hasty control.vk.spirv_assembly.hasty To run these tests, media-gfx/deqp must be built with USE=vulkan. I tested this on reef (which has an Apollo Lake GPU) with mesa-17.1.0-r3, and all tests passed in suite 'graphics_dEQP.vk-master.hasty'. I also did a smoke test on sentry (Skylake GPU) with 'graphics_dEQP.vk.api.smoke'. BUG= chromium:689177 TEST=$ USE=vulkan emerge-sentry deqp $ cros deploy $SENTRY_IP deqp mesa vulkan-loader $ test_that --board=sentry $SENTRY_IP graphics_dEQP.vk.api.smoke # Confirm 6/6 tests pass. v4: Fix crash when deqp-vk is not installed. For each dEQP executable (deqp-egl, deqp-vk, deqp-gles3, etc), the script chdir'd into the executable's directory before testing if the executable existed. This made Python unhappy by raising an OSError. Depends-On: I10233e70cdf2303ace300bef39896e98f2d7636b Change-Id: I854876d100ad6ece737459661ca2a6b2a05e395c Reviewed-on: https://chromium-review.googlesource.com/438766 Commit-Ready: Ilja H. Friedel <ihf@chromium.org> Tested-by: Chad Versace <chadversary@chromium.org> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> [add] https://crrev.com/360a64ab4f40744e63f467409d9252f39516d0d0/client/site_tests/graphics_dEQP/control.vk-master.hasty [add] https://crrev.com/360a64ab4f40744e63f467409d9252f39516d0d0/client/site_tests/graphics_dEQP/control.vk-master [add] https://crrev.com/360a64ab4f40744e63f467409d9252f39516d0d0/client/site_tests/graphics_dEQP/control.vk.info [add] https://crrev.com/360a64ab4f40744e63f467409d9252f39516d0d0/client/site_tests/graphics_dEQP/control.vk.binding_model.hasty [add] https://crrev.com/360a64ab4f40744e63f467409d9252f39516d0d0/client/site_tests/graphics_dEQP/control.vk.spirv_assembly.hasty [add] https://crrev.com/360a64ab4f40744e63f467409d9252f39516d0d0/client/site_tests/graphics_dEQP/control.vk.glsl.hasty [add] https://crrev.com/360a64ab4f40744e63f467409d9252f39516d0d0/client/site_tests/graphics_dEQP/control.vk.api.hasty [modify] https://crrev.com/360a64ab4f40744e63f467409d9252f39516d0d0/client/site_tests/graphics_dEQP/generate_controlfiles.py [modify] https://crrev.com/360a64ab4f40744e63f467409d9252f39516d0d0/client/site_tests/graphics_dEQP/graphics_dEQP.py [add] https://crrev.com/360a64ab4f40744e63f467409d9252f39516d0d0/client/site_tests/graphics_dEQP/control.vk.api.smoke [add] https://crrev.com/360a64ab4f40744e63f467409d9252f39516d0d0/client/site_tests/graphics_dEQP/control.vk.pipeline.hasty
,
Feb 15 2017
Done. The Vulkan CTS has been added to the daily test runs for Skylake and Apollo Lake. And quick smoke tests have been added to the CQ tests.
,
Feb 16 2017
Re-opening this issue, because it doesn't work for a cleanly installed chromiumos_test_image.bin. On a clean image install, all the Vulkan CTS tests trivially pass.
The problem is the installation location of libvulkan.so in chromiumos_test_image.bin. Because nothing in the base image depends on libvulkan.so, the image builder scripts install it at /usr/local/lib64/libvulkan.so (even though emerge-$BOARD correctly installs it to chroot/tmp/build/$BOARD/usr/lib64/libvulkan.so). graphics_dEQP.py expects to find it in /usr/lib64.
Possible solutions:
- In graphics_dEQP.py, add /usr/local/lib64 to LD_LIBRARY_PATH. This has precedent in other autotest scripts.
- In virtual/target-chromium-os, add the dependency "vulkan? (virtual/vulkan-icd)", which will add a transitive dependency on media-libs/vulkan-loader, pulling it into the base image.
Ilja, any advice?
,
Feb 16 2017
As we ultimately want it in the base image, so the second approach is preferred.
,
Feb 16 2017
Ok. I'll submit a patch against tafget-chromium-os.ebuild.
,
Feb 18 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/0a37d8effca38bc9c7b36d5fa218f2561e3b24cc commit 0a37d8effca38bc9c7b36d5fa218f2561e3b24cc Author: Chad Versace <chadversary@chromium.org> Date: Sat Feb 18 08:39:28 2017 graphics_dEQP: Prepend /usr/local/$libdir to LD_LIBRARY_PATH Prepend "/usr/local/lib:/usr/local/lib64" to LD_LIBRARY_PATH for all dEQP test processes. Fixes the Vulkan CTS. Currently, no package in the base image requires media-libs/vulkan-loader. Only media-gfx/deqp does, which is in the test image. Therefore the build_image scripts install libvulkan.so into /usr/local. When in a login shell on Chrome OS, LD_LIBRARY_PATH does contain the /usr/local libdirs. But processes spawned by autotest do not. Therefore, deqp worked when ran from a login shell; but it failed to dlopen libvulkan.so when ran by autotest. BUG= chromium:689177 TEST=Run `test_that ... graphics_dEQP.vk.info` on sentry. Inspect client.0.DEBUG to confirm that tests ran. Change-Id: Ib067acc76db1364328b45ef2c66a6a73a4e12c38 Reviewed-on: https://chromium-review.googlesource.com/444984 Commit-Ready: Ilja H. Friedel <ihf@chromium.org> Tested-by: Chad Versace <chadversary@chromium.org> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> Reviewed-by: Gurchetan Singh <gurchetansingh@chromium.org> [modify] https://crrev.com/0a37d8effca38bc9c7b36d5fa218f2561e3b24cc/client/site_tests/graphics_dEQP/graphics_dEQP.py
,
Feb 19 2017
Very well, we are getting close! Looks like vk-master is running 6h, finishing about 40k tests and then getting aborted. Sounds like we should run the hasty version instead or increase the timeout to 12+xh-ish, which I am reluctant to do for a daily test. https://wmatrix.googleplex.com/failures/unfiltered?hide_missing=True&tests=graphics_dEQP.vk-master&builds=R58-9296.0.0&releases=tot&platforms=cave https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/102146906-chromeos-test/chromeos2-row8-rack7-host13/debug/ 02/19 02:14:08.214 INFO | graphics_dEQP:0519| ChromeOS GPU family = skylake 02/19 02:14:08.214 INFO | graphics_dEQP:0529| dEQP test filter = vk-master 02/19 02:14:09.331 INFO | graphics_dEQP:0563| Running each test individually. 02/19 02:14:09.332 INFO | graphics_dEQP:0322| [1/81464] TestCase: dEQP-VK.info.build 02/19 02:14:09.461 INFO | graphics_dEQP:0398| Result: Pass 02/19 02:14:09.462 INFO | graphics_dEQP:0322| [2/81464] TestCase: dEQP-VK.info.device 02/19 02:14:09.573 INFO | graphics_dEQP:0398| Result: Pass 02/19 02:14:09.574 INFO | graphics_dEQP:0322| [3/81464] TestCase: dEQP-VK.info.platform [...] 02/19 08:16:23.943 INFO | graphics_dEQP:0398| Result: Pass 02/19 08:16:23.943 INFO | graphics_dEQP:0322| [44953/81464] TestCase: dEQP-VK.pipeline.sampler.view_type.cube_array.format.astc_4x4_unorm_block.mipmap.nearest.lod.equal_min_3_max_3 [eof]
,
Feb 21 2017
If I recall correctly the official Khronos procedures for the Vulkan CTS, vendors must pass all the tests in one run (that is, a single process). So, instead of increasing the timeout, we should switch to hasty. Last time I checked, the Apollo Lake platforms already passed graphics_dEQP.vk-master.hasty.
,
Feb 22 2017
The good news is hasty runs in 20 minutes. The bad news is there are 8 failures: test_that <sentry> graphics_dEQP.vk-master.hasty /tmp/test_that_results_nURsxC/results-1-graphics_dEQP.vk-master.hasty/graphics_dEQP.vk-master.hasty FAIL: Failed: on skylake 8/81464 tests failed. The failures are below and details are in the attachment. Are you sure everything is passing without hasty? I don't have 12h to run the test. dEQP-VK.glsl.builtin.precision.sin.highp_compute.scalar dEQP-VK.glsl.builtin.precision.sin.highp_compute.vec2 dEQP-VK.glsl.builtin.precision.sin.highp_compute.vec3 dEQP-VK.glsl.builtin.precision.sin.highp_compute.vec4 dEQP-VK.glsl.builtin.precision.cos.highp_compute.scalar dEQP-VK.glsl.builtin.precision.cos.highp_compute.vec2 dEQP-VK.glsl.builtin.precision.cos.highp_compute.vec3 dEQP-VK.glsl.builtin.precision.cos.highp_compute.vec4
,
Feb 22 2017
Ah, I guess apollo lake != skylake. https://chromium-review.googlesource.com/445928
,
Feb 22 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/16163e6a938fc945d05df60ce0d56ddc1ce8c830 commit 16163e6a938fc945d05df60ce0d56ddc1ce8c830 Author: Ilja H. Friedel <ihf@chromium.org> Date: Wed Feb 22 05:38:37 2017 graphics_dEQP: run vk-master.hasty in nightly suite. vk-master takes 12h to run. vk-master.hasty takes 0.3h to run. Unlike other -hasty variants vk-master.hasty is supposed to be allways passing without flakes. BUG= chromium:689177 TEST=Ran locally on sentry, saw 8 failures. Probably not due to hasty. Change-Id: I6b20eac8b6e7428bc9b5964cd0f686ea405ba4fb Reviewed-on: https://chromium-review.googlesource.com/445928 Tested-by: Ilja H. Friedel <ihf@chromium.org> Trybot-Ready: Ilja H. Friedel <ihf@chromium.org> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> [modify] https://crrev.com/16163e6a938fc945d05df60ce0d56ddc1ce8c830/client/site_tests/graphics_dEQP/control.vk-master.hasty [modify] https://crrev.com/16163e6a938fc945d05df60ce0d56ddc1ce8c830/client/site_tests/graphics_dEQP/control.vk-master
,
Mar 9 2017
Completed by commit above, in comment 20.
,
May 30 2017
,
Aug 1 2017
,
Jan 22 2018
|
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by chadversary@chromium.org
, Feb 6 2017