Driver upgrade for Windows and Linux NVIDIA GPU bots |
||||||||||||||||||||||||||||
Issue descriptionOn Windows, our GPU NVIDIA bots are currently on 388.16 which are circa 10/2017 (looking at https://chromium-swarm.appspot.com/task?id=400e46bc52416b10&refresh=10&show_raw=1) On Linux, the driver is 384.90. No date is specified but google search shows this as from 9/2017. On both systems the drivers seem to be about a year old. I think if we continue to use these configurations for a while we should invest in upgrading their drivers to the most recent public releases for Windows and Linux. Note that we don't currently have an experimental bot for Linux NVIDIA so it might be a good idea to add one before proceeding on Linux. Ken if you agree this is a good idea I'd be happy to help drive this. I'm hoping this can fix some of the Linux Vulkan flakiness we were seeing in issue 861814. Do you have a suggestion of how many bots we should initially migrate to the new versions to test on the experimental bots? ⛆ |
|
|
,
Sep 20
Good idea Jamie; yes, let's upgrade the drivers. As you suggested, could you please set up a Linux NVIDIA experimental bot? Looking at a recent upgrade of the driver on the Intel GPU bots ( Issue 838970 , Issue 879651) we upgraded 4 bots to the new driver version. I'd suggest the same for this, upgrading 4 Windows and 4 Linux bots to the new version. Please see e.g. Issue 879651 for the general template for filing requests with the Labs team. Thanks for driving this! P.S. for anyone else looking at this bug, here are instructions for performing an upgrade like this: https://chromium.googlesource.com/chromium/src/+/master/docs/gpu/gpu_testing_bot_details.md#How-to-test-and-deploy-a-driver-update
,
Sep 21
With RTX 2080 I a seeing some errors: Log Messages GpuProcessHostUIShim: [31824:26884:0920/200248.967:WARNING:angle_platform_impl.cc(59)] : compileToBinary(235): C:\fakepath(44,8-56): warning X3571: pow(f, e) will not work for negative f, use abs(f) or conditionally handle negative values if you expect them C:\fakepath(52,9-41): warning X3571: pow(f, e) will not work for negative f, use abs(f) or conditionally handle negative values if you expect them [31824:3664:0920/201634.926:ERROR:gl_surface_egl.cc(538)] : EGL Driver message (Error) eglSStreamConsumerReleaseKHR: Invalid stream state [31824:3664:0920/201634.926:ERROR:dxva_picture_buffer_win.cc(24)] : Error in dxva_picture_buffer_win.cc on line 493 [31824:26884:0920/210412.367:WARNING:angle_platform_impl.cc(59)] : compileToBinary(235): C:\fakepath(64,8-56): warning X3571: pow(f, e) will not work for negative f, use abs(f) or conditionally handle negative values if you expect them C:\fakepath(72,9-41): warning X3571: pow(f, e) will not work for negative f, use abs(f) or conditionally handle negative values if you expect them [31824:3664:0920/210650.570:WARNING:ipc_message_attachment_set.cc(49)] : MessageAttachmentSet destroyed with unconsumed attachments: 0/1 [31824:3664:0920/210650.570:WARNING:ipc_message_attachment_set.cc(49)] : MessageAttachmentSet destroyed with unconsumed attachments: 0/1 [31824:3664:0921/084401.006:ERROR:gl_surface_egl.cc(538)] : EGL Driver message (Error) eglSStreamConsumerReleaseKHR: Current GL context not associated with stream consumer [31824:3664:0921/084401.006:ERROR:dxva_picture_buffer_win.cc(24)] : Error in dxva_picture_buffer_win.cc on line 493 [31824:36308:0921/084402.519:WARNING:angle_platform_impl.cc(59)] : compileToBinary(235): C:\fakepath(55,8-56): warning X3571: pow(f, e) will not work for negative f, use abs(f) or conditionally handle negative values if you expect them C:\fakepath(63,9-41): warning X3571: pow(f, e) will not work for negative f, use abs(f) or conditionally handle negative values if you expect them [31824:3664:0921/084429.987:WARNING:ipc_message_attachment_set.cc(49)] : MessageAttachmentSet destroyed with unconsumed attachments: 0/1 [31824:41556:0921/084433.426:WARNING:angle_platform_impl.cc(59)] : compileToBinary(235): C:\fakepath(56,8-56): warning X3571: pow(f, e) will not work for negative f, use abs(f) or conditionally handle negative values if you expect them C:\fakepath(64,9-41): warning X3571: pow(f, e) will not work for negative f, use abs(f) or conditionally handle negative values if you expect them [31824:36308:0921/084433.437:WARNING:angle_platform_impl.cc(59)] : compileToBinary(235): C:\fakepath(56,8-56): warning X3571: pow(f, e) will not work for negative f, use abs(f) or conditionally handle negative values if you expect them C:\fakepath(64,9-41): warning X3571: pow(f, e) will not work for negative f, use abs(f) or conditionally handle negative values if you expect them [31824:41556:0921/084433.447:WARNING:angle_platform_impl.cc(59)] : compileToBinary(235): C:\fakepath(53,8-56): warning X3571: pow(f, e) will not work for negative f, use abs(f) or conditionally handle negative values if you expect them C:\fakepath(61,9-41): warning X3571: pow(f, e) will not work for negative f, use abs(f) or conditionally handle negative values if you expect them [31824:36308:0921/084433.458:WARNING:angle_platform_impl.cc(59)] : compileToBinary(235): C:\fakepath(53,8-56): warning X3571: pow(f, e) will not work for negative f, use abs(f) or conditionally handle negative values if you expect them C:\fakepath(61,9-41): warning X3571: pow(f, e) will not work for negative f, use abs(f) or conditionally handle negative values if you expect them [31824:41556:0921/084433.494:WARNING:angle_platform_impl.cc(59)] : compileToBinary(235): C:\fakepath(35,8-56): warning X3571: pow(f, e) will not work for negative f, use abs(f) or conditionally handle negative values if you expect them C:\fakepath(43,9-41): warning X3571: pow(f, e) will not work for negative f, use abs(f) or conditionally handle negative values if you expect them [31824:36308:0921/084433.504:WARNING:angle_platform_impl.cc(59)] : compileToBinary(235): C:\fakepath(54,8-56): warning X3571: pow(f, e) will not work for negative f, use abs(f) or conditionally handle negative values if you expect them C:\fakepath(62,9-41): warning X3571: pow(f, e) will not work for negative f, use abs(f) or conditionally handle negative values if you expect them [31824:36308:0921/084441.028:WARNING:angle_platform_impl.cc(59)] : compileToBinary(235): C:\fakepath(56,8-56): warning X3571: pow(f, e) will not work for negative f, use abs(f) or conditionally handle negative values if you expect them C:\fakepath(64,9-41): warning X3571: pow(f, e) will not work for negative f, use abs(f) or conditionally handle negative values if you expect them [31824:3664:0921/084441.213:ERROR:gles2_cmd_decoder_passthrough_doers.cc(4211)] : NOT IMPLEMENTED GpuProcessHostUIShim: Not sure if this is driver related issue. Thanks
,
Sep 21
Please file a new bug about that and provide a test case as well as about:gpu information.
,
Sep 23
I did open a new one: https://bugs.chromium.org/p/chromium/issues/detail?id=887931 Thanks
,
Sep 24
,
Sep 24
,
Oct 9
,
Oct 18
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/config/+/f22a9d0bc8de03b205301e4fdc5f79d0f0514199 commit f22a9d0bc8de03b205301e4fdc5f79d0f0514199 Author: Jamie Madill <jmadill@chromium.org> Date: Thu Oct 18 00:14:19 2018
,
Oct 31
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/0c93ce0c0eeba69b1a512f2ef810febed9098bfd commit 0c93ce0c0eeba69b1a512f2ef810febed9098bfd Author: Jamie Madill <jmadill@chromium.org> Date: Wed Oct 31 04:27:23 2018 Add Linux GPU FYI NV Experimental config. This config will be used to stage Linux Nvidia driver upgrades. Bug: 887241 Change-Id: I05a5d7ec3109c4892faf8453ba4733d03dc8242c Reviewed-on: https://chromium-review.googlesource.com/c/1286986 Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org> Cr-Commit-Position: refs/heads/master@{#604150} [modify] https://crrev.com/0c93ce0c0eeba69b1a512f2ef810febed9098bfd/infra/config/global/cr-buildbucket.cfg [modify] https://crrev.com/0c93ce0c0eeba69b1a512f2ef810febed9098bfd/infra/config/global/luci-milo.cfg [modify] https://crrev.com/0c93ce0c0eeba69b1a512f2ef810febed9098bfd/infra/config/global/luci-scheduler.cfg [modify] https://crrev.com/0c93ce0c0eeba69b1a512f2ef810febed9098bfd/testing/buildbot/chromium.gpu.fyi.json [modify] https://crrev.com/0c93ce0c0eeba69b1a512f2ef810febed9098bfd/testing/buildbot/waterfalls.pyl
,
Oct 31
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/8798565090420e2bc80cf95bffe733565d12a5d4 commit 8798565090420e2bc80cf95bffe733565d12a5d4 Author: Jamie Madill <jmadill@chromium.org> Date: Wed Oct 31 12:33:39 2018 Add Linux experimental GPU.FYI bot. Must be landed after src-side change. Bug: 887241 Change-Id: I824881c3cded82268fb8fec29b848c314a2e3323 Reviewed-on: https://chromium-review.googlesource.com/c/1305573 Reviewed-by: Kenneth Russell <kbr@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org> [modify] https://crrev.com/8798565090420e2bc80cf95bffe733565d12a5d4/scripts/slave/recipe_modules/chromium_tests/chromium_gpu_fyi.py
,
Oct 31
,
Nov 12
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/a5f88b7e35a567c40e25e21b285ec7e8c687e6d6 commit a5f88b7e35a567c40e25e21b285ec7e8c687e6d6 Author: James Darpinian <jdarpinian@chromium.org> Date: Mon Nov 12 23:09:19 2018 Cleaning up WebGL test failure expectations. Many of these are leftover failure expectations from fixed bugs. Some were fixed on some platforms but not others. There are some new failures that were hidden by previous broad suppressions. Also some tests are only failing because we haven't updated the drivers on the bots (http://crbug.com/887241). Bug: 757097 , 838133 , 672380 , 625738, angleproject:2325 Bug: 534697 , 849572, 625738, 703779, angleproject:2142 Bug: 757098, 644740, 705865, 602688, angleproject:2103 Bug: 903903, 740769 , 662644, 680720 , angleproject:1932 Bug: 709874 , 887241, 625365 , angleproject:2952 Change-Id: I4e39e5a09d31d788d8a8615ad06d1ec88f064f00 Reviewed-on: https://chromium-review.googlesource.com/c/1325197 Commit-Queue: James Darpinian <jdarpinian@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#607367} [modify] https://crrev.com/a5f88b7e35a567c40e25e21b285ec7e8c687e6d6/content/test/gpu/gpu_tests/webgl2_conformance_expectations.py [modify] https://crrev.com/a5f88b7e35a567c40e25e21b285ec7e8c687e6d6/content/test/gpu/gpu_tests/webgl_conformance_expectations.py [modify] https://crrev.com/a5f88b7e35a567c40e25e21b285ec7e8c687e6d6/content/test/gpu/gpu_tests/webgl_conformance_expectations_unittest.py
,
Nov 13
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/539ef7dd7cb70fc0fa684c44c0bfce18ced47e61 commit 539ef7dd7cb70fc0fa684c44c0bfce18ced47e61 Author: Corentin Wallez <cwallez@chromium.org> Date: Tue Nov 13 17:08:12 2018 Add back some WebGL test expectations A lot of expectations were removed but some of them were still necessary, this adds them back. TBR=kbr@chromium.org BUG= chromium:693135 BUG=chromium:772651 BUG= chromium:798117 BUG= chromium:874620 BUG=chromium:887241 Change-Id: Ib3bccdf2ff48a28a8f0a3ce428239980a58c5266 Reviewed-on: https://chromium-review.googlesource.com/c/1333650 Reviewed-by: Corentin Wallez <cwallez@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Commit-Queue: Corentin Wallez <cwallez@chromium.org> Cr-Commit-Position: refs/heads/master@{#607626} [modify] https://crrev.com/539ef7dd7cb70fc0fa684c44c0bfce18ced47e61/content/test/gpu/gpu_tests/webgl2_conformance_expectations.py [modify] https://crrev.com/539ef7dd7cb70fc0fa684c44c0bfce18ced47e61/content/test/gpu/gpu_tests/webgl_conformance_expectations.py
,
Nov 17
,
Nov 18
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/eb84fe2e5a8a5b2f15ab7edd9f7d4aac811cc66b commit eb84fe2e5a8a5b2f15ab7edd9f7d4aac811cc66b Author: Jamie Madill <jmadill@chromium.org> Date: Sun Nov 18 00:41:15 2018 Add tests to NVIDIA Windows experimental bot. These tests are targeting the driver "24.21.14.1195". Bug: 887241 Tbr: kbr@chromium.org Change-Id: I5d79254570d01fa7f42fcf6d020a737148a4bd4b Reviewed-on: https://chromium-review.googlesource.com/c/1341207 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Jamie Madill <jmadill@chromium.org> Cr-Commit-Position: refs/heads/master@{#609126} [modify] https://crrev.com/eb84fe2e5a8a5b2f15ab7edd9f7d4aac811cc66b/testing/buildbot/chromium.gpu.fyi.json [modify] https://crrev.com/eb84fe2e5a8a5b2f15ab7edd9f7d4aac811cc66b/testing/buildbot/mixins.pyl [modify] https://crrev.com/eb84fe2e5a8a5b2f15ab7edd9f7d4aac811cc66b/testing/buildbot/waterfalls.pyl
,
Nov 28
,
Nov 29
,
Dec 20
,
Dec 20
The Windows bot is looking pretty good with the new driver: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20FYI%20Exp%20Release%20(NVIDIA)
,
Dec 20
Note: saw some tasks expire in these two builds: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20FYI%20Exp%20Release%20%28NVIDIA%29/20669 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20FYI%20Exp%20Release%20%28NVIDIA%29/20670 Unclear why; maybe the 4 machines were overloaded at the time: https://chromium-swarm.appspot.com/botlist?c=id&c=task&c=os&c=status&d=asc&f=pool%3AChrome-GPU&f=os%3AWindows-10&f=gpu%3A10de%3A1cb3-24.21.14.1195&k=gpu&s=id
,
Dec 29
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/30f735cb4f5fc57390de3773b0dd21f9332e9774 commit 30f735cb4f5fc57390de3773b0dd21f9332e9774 Author: Jamie Madill <jmadill@chromium.org> Date: Sat Dec 29 18:04:30 2018 Add tests to Linux P400 experimental bot. This will allow us to test the driver before uprading the main fleet. Bug: 887241 Tbr: kbr@chromium.org Change-Id: I378e8af5afc96c980df55abd75856c20204167cc Reviewed-on: https://chromium-review.googlesource.com/c/1392267 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org> Cr-Commit-Position: refs/heads/master@{#619251} [modify] https://crrev.com/30f735cb4f5fc57390de3773b0dd21f9332e9774/testing/buildbot/chromium.gpu.fyi.json [modify] https://crrev.com/30f735cb4f5fc57390de3773b0dd21f9332e9774/testing/buildbot/mixins.pyl [modify] https://crrev.com/30f735cb4f5fc57390de3773b0dd21f9332e9774/testing/buildbot/test_suites.pyl [modify] https://crrev.com/30f735cb4f5fc57390de3773b0dd21f9332e9774/testing/buildbot/waterfalls.pyl
,
Dec 29
,
Dec 29
,
Jan 5
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/b31139b2fc32109c6af680a77b7998b84da5620b commit b31139b2fc32109c6af680a77b7998b84da5620b Author: Jamie Madill <jmadill@chromium.org> Date: Sat Jan 05 02:23:59 2019 Use either stable|experimental driver for NV Win bots. This is in prepraration for migration to the new driver version. Bug: 887241 Change-Id: I76c7244211003ca6e8b94d83103234c501d86da9 Reviewed-on: https://chromium-review.googlesource.com/c/1396604 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#620152} [modify] https://crrev.com/b31139b2fc32109c6af680a77b7998b84da5620b/testing/buildbot/chromium.gpu.fyi.json [modify] https://crrev.com/b31139b2fc32109c6af680a77b7998b84da5620b/testing/buildbot/chromium.gpu.json [modify] https://crrev.com/b31139b2fc32109c6af680a77b7998b84da5620b/testing/buildbot/client.v8.fyi.json [modify] https://crrev.com/b31139b2fc32109c6af680a77b7998b84da5620b/testing/buildbot/waterfalls.pyl
,
Jan 7
,
Jan 9
,
Jan 10
,
Jan 10
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/ab22c9209ed9932f116e85fb1cb0c21dc80b2278 commit ab22c9209ed9932f116e85fb1cb0c21dc80b2278 Author: Jamie Madill <jmadill@chromium.org> Date: Thu Jan 10 20:02:23 2019 Finish migration to new Win10/NV driver. * removes alternate swarming dimensions from Win10 NV bots * updates stable driver mixin * removes all tests from the experimental config Bug: 887241 No-Try: True Change-Id: I4d420a734c5935d3a5e04b7d1c6dbe53b4683401 Reviewed-on: https://chromium-review.googlesource.com/c/1405433 Commit-Queue: Kenneth Russell <kbr@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#621719} [modify] https://crrev.com/ab22c9209ed9932f116e85fb1cb0c21dc80b2278/testing/buildbot/chromium.gpu.fyi.json [modify] https://crrev.com/ab22c9209ed9932f116e85fb1cb0c21dc80b2278/testing/buildbot/chromium.gpu.json [modify] https://crrev.com/ab22c9209ed9932f116e85fb1cb0c21dc80b2278/testing/buildbot/client.v8.fyi.json [modify] https://crrev.com/ab22c9209ed9932f116e85fb1cb0c21dc80b2278/testing/buildbot/mixins.pyl [modify] https://crrev.com/ab22c9209ed9932f116e85fb1cb0c21dc80b2278/testing/buildbot/test_suites.pyl [modify] https://crrev.com/ab22c9209ed9932f116e85fb1cb0c21dc80b2278/testing/buildbot/waterfalls.pyl
,
Jan 11
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/606d523ae0443a5e062bf8ca6ef9932d09b94fdb commit 606d523ae0443a5e062bf8ca6ef9932d09b94fdb Author: Jamie Madill <jmadill@chromium.org> Date: Fri Jan 11 00:10:34 2019 Use alternate swarming dimensions for Win7 NV upgrade. Work is already completed for Windows 10. Using the alternate dimensions should allow for a smoother upgrade. Bug: 887241 Change-Id: If3d8f854f1d2e42ae5dc8385098800bb060c7000 Reviewed-on: https://chromium-review.googlesource.com/c/1406083 Reviewed-by: Kenneth Russell <kbr@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org> Cr-Commit-Position: refs/heads/master@{#621814} [modify] https://crrev.com/606d523ae0443a5e062bf8ca6ef9932d09b94fdb/testing/buildbot/chromium.gpu.fyi.json [modify] https://crrev.com/606d523ae0443a5e062bf8ca6ef9932d09b94fdb/testing/buildbot/waterfalls.pyl
,
Jan 11
,
Jan 11
,
Jan 14
,
Jan 14
,
Jan 14
,
Jan 14
,
Jan 14
,
Jan 18
(4 days ago)
,
Jan 18
(4 days ago)
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/427999cfc3d0da1500edf93898f497ad0e8a91bb commit 427999cfc3d0da1500edf93898f497ad0e8a91bb Author: Jamie Madill <jmadill@chromium.org> Date: Fri Jan 18 16:14:31 2019 Update Win7 NVIDIA bot driver dimensions. This removes the alternate dimensions now that the driver upgrade is complete. It also fixes the dimensions. The should have used the number 24.21.14.1195 instead of 23.21.14.1195. Bug: 923323 Bug: 887241 Change-Id: I946ee2a2e04b35f9437849162005859dd1eb6fb6 Tbr: kbr@chromium.org Reviewed-on: https://chromium-review.googlesource.com/c/1421877 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org> Cr-Commit-Position: refs/heads/master@{#624148} [modify] https://crrev.com/427999cfc3d0da1500edf93898f497ad0e8a91bb/testing/buildbot/chromium.gpu.fyi.json [modify] https://crrev.com/427999cfc3d0da1500edf93898f497ad0e8a91bb/testing/buildbot/mixins.pyl [modify] https://crrev.com/427999cfc3d0da1500edf93898f497ad0e8a91bb/testing/buildbot/waterfalls.pyl
,
Jan 18
(4 days ago)
,
Jan 18
(4 days ago)
,
Yesterday
(42 hours ago)
|
|||||||||||||||||||||||||
►
Sign in to add a comment |
||||||||||||||||||||||||||||
Comment 1 by jmad...@chromium.org
, Sep 20