New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 838970 link

Starred by 5 users

Issue metadata

Status: Fixed
Owner:
Closed: Oct 10
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 1
Type: Bug

Blocked on:
issue 838975
issue 839182
issue 839635
issue 841408
issue 841604
issue 841789
issue 843369
issue 879651
issue 885225
issue 886985

Blocking:
issue 597700
issue 854100
issue 894127



Sign in to add a comment

Update Windows Intel bot driver to 24.20.100.6286

Project Member Reported by zmo@chromium.org, May 2 2018

Issue description

This driver potentially fixes the top #1 Windows GPU crash

https://downloadcenter.intel.com/download/27680/Graphics-Intel-Graphics-Driver-for-Windows-10?product=80939

We should convert a VM to an experiment bot on GU waterfall, upgrade to this driver, make sure it's running green, and then update all other Win Intel bots.
 

Comment 1 by zmo@chromium.org, May 2 2018

Blocking: 597700

Comment 2 by kbr@chromium.org, May 2 2018

Blockedon: 838975

Comment 3 by kbr@chromium.org, May 2 2018

Owner: zmo@chromium.org
Status: Assigned (was: Available)
Mo: let me assign this to you and I'll walk you through the sub-steps of doing this upgrade.

Project Member

Comment 4 by bugdroid1@chromium.org, May 2 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c2913b9855db00283818286085a19ba122d09c9b

commit c2913b9855db00283818286085a19ba122d09c9b
Author: Zhenyao Mo <zmo@chromium.org>
Date: Wed May 02 19:28:31 2018

Add a GPU driver version to Win Intel GPU job dimension

This is for updating Win Intel GPU bot for a newer driver

BUG= 838970 
TRY=optional GPU bots
TBR=kbr@chromium.org

Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel
Change-Id: Iea98808dabcf24ce2f4e14b421c11a7f203d194e
Reviewed-on: https://chromium-review.googlesource.com/1040246
Commit-Queue: Zhenyao Mo <zmo@chromium.org>
Reviewed-by: Zhenyao Mo <zmo@chromium.org>
Cr-Commit-Position: refs/heads/master@{#555502}
[modify] https://crrev.com/c2913b9855db00283818286085a19ba122d09c9b/content/test/gpu/generate_buildbot_json.py
[modify] https://crrev.com/c2913b9855db00283818286085a19ba122d09c9b/testing/buildbot/chromium.gpu.fyi.json

Project Member

Comment 5 by bugdroid1@chromium.org, May 2 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/d80b8933ac0844341a211aa72883839f686eda32

commit d80b8933ac0844341a211aa72883839f686eda32
Author: Zhenyao Mo <zmo@chromium.org>
Date: Wed May 02 23:28:43 2018

Project Member

Comment 6 by bugdroid1@chromium.org, May 2 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/7225b5e672b4af3aad516d952ef6f73e35d8159a

commit 7225b5e672b4af3aad516d952ef6f73e35d8159a
Author: Zhenyao Mo <zmo@chromium.org>
Date: Wed May 02 23:45:11 2018

Add Win10 Intel Experimental Release bot to chromium.gpu.fyi

BUG= 838970 
R=kbr@chromium.org

Change-Id: Iabfd3eb97d28323e9f622d92e7c1dffff3f27090
Reviewed-on: https://chromium-review.googlesource.com/1041145
Commit-Queue: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>

[modify] https://crrev.com/7225b5e672b4af3aad516d952ef6f73e35d8159a/scripts/slave/recipe_modules/chromium_tests/chromium_gpu_fyi.py

Comment 7 by kbr@chromium.org, May 3 2018

Blockedon: 839182
Project Member

Comment 8 by bugdroid1@chromium.org, May 3 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9618adde3b505443776bd35ab818ba0943b6b783

commit 9618adde3b505443776bd35ab818ba0943b6b783
Author: Kenneth Russell <kbr@chromium.org>
Date: Thu May 03 03:16:05 2018

Update documentation for deploying new GPU FYI test bots.

Worked with zmo@ to develop this documentation along with the example
CLs.

Also added top-level links to the GPU docs and cleaned up some other
now-incorrect documentation. More cleanups are still needed.

Bug:  838970 
Tbr: zmo@chromium.org
Change-Id: I40e07adf864974a645490627f5a055ce686237a3
Reviewed-on: https://chromium-review.googlesource.com/1041273
Commit-Queue: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#555661}
[modify] https://crrev.com/9618adde3b505443776bd35ab818ba0943b6b783/docs/README.md
[modify] https://crrev.com/9618adde3b505443776bd35ab818ba0943b6b783/docs/gpu/gpu_testing_bot_details.md

Project Member

Comment 9 by bugdroid1@chromium.org, May 3 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/6b388edeed6b2bec30ad29872b23d9319d5d9c8a

commit 6b388edeed6b2bec30ad29872b23d9319d5d9c8a
Author: Zhenyao Mo <zmo@chromium.org>
Date: Thu May 03 23:25:45 2018

Revert "Add Win10 Intel Experimental Release bot to chromium.gpu.fyi"

This reverts commit 7225b5e672b4af3aad516d952ef6f73e35d8159a.

Reason for revert:  crbug.com/839635 

Original change's description:
> Add Win10 Intel Experimental Release bot to chromium.gpu.fyi
> 
> BUG= 838970 
> R=​kbr@chromium.org
> 
> Change-Id: Iabfd3eb97d28323e9f622d92e7c1dffff3f27090
> Reviewed-on: https://chromium-review.googlesource.com/1041145
> Commit-Queue: Kenneth Russell <kbr@chromium.org>
> Reviewed-by: Kenneth Russell <kbr@chromium.org>

TBR=zmo@chromium.org,kbr@chromium.org

Change-Id: I9413563c7ef2fe1ab24fb85ad4b011069c602ede
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  838970 
Reviewed-on: https://chromium-review.googlesource.com/1043365
Reviewed-by: Zhenyao Mo <zmo@chromium.org>
Commit-Queue: Zhenyao Mo <zmo@chromium.org>

[modify] https://crrev.com/6b388edeed6b2bec30ad29872b23d9319d5d9c8a/scripts/slave/recipe_modules/chromium_tests/chromium_gpu_fyi.py

Comment 10 by kbr@chromium.org, May 4 2018

Blockedon: 839635
Project Member

Comment 11 by bugdroid1@chromium.org, May 4 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/139881be4cefe1e70bfe5dca0340c2a0f6a890d0

commit 139881be4cefe1e70bfe5dca0340c2a0f6a890d0
Author: Kenneth Russell <kbr@chromium.org>
Date: Fri May 04 00:45:20 2018

Fix ordering problem in documentation of GPU bot deployment.

The chromium/src CL needs to land before the tools/build CL does, or
the builder will break completely.

Bug:  838970 
Change-Id: I3d788eac5564ad90bdd67f6e387a309b8718185b
Tbr: zmo@chromium.org
Tbr: cwallez@chromium.org
Reviewed-on: https://chromium-review.googlesource.com/1043489
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#555934}
[modify] https://crrev.com/139881be4cefe1e70bfe5dca0340c2a0f6a890d0/docs/gpu/gpu_testing_bot_details.md

Components: -Infra Infra>Client>Chrome
Project Member

Comment 13 by bugdroid1@chromium.org, May 9 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/eca3d8749122fc51150db1f75b5179a6cb644acd

commit eca3d8749122fc51150db1f75b5179a6cb644acd
Author: Zhenyao Mo <zmo@chromium.org>
Date: Wed May 09 16:37:19 2018

Enable Win10 Experimental Release Intel bot on GPU.FYI waterfall

BUG= 838970 

Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel
Change-Id: I28a0a55d4fb1d248a53ced7f721df128d470c34f
Reviewed-on: https://chromium-review.googlesource.com/1041164
Commit-Queue: Zhenyao Mo <zmo@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Cr-Commit-Position: refs/heads/master@{#557216}
[modify] https://crrev.com/eca3d8749122fc51150db1f75b5179a6cb644acd/content/test/gpu/generate_buildbot_json.py
[modify] https://crrev.com/eca3d8749122fc51150db1f75b5179a6cb644acd/infra/config/global/cr-buildbucket.cfg
[modify] https://crrev.com/eca3d8749122fc51150db1f75b5179a6cb644acd/infra/config/global/luci-milo.cfg
[modify] https://crrev.com/eca3d8749122fc51150db1f75b5179a6cb644acd/infra/config/global/luci-scheduler.cfg
[modify] https://crrev.com/eca3d8749122fc51150db1f75b5179a6cb644acd/testing/buildbot/chromium.gpu.fyi.json

Project Member

Comment 14 by bugdroid1@chromium.org, May 9 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/d118e1d7141e9f1c2b95e1ce1f5480d4af310484

commit d118e1d7141e9f1c2b95e1ce1f5480d4af310484
Author: Kenneth Russell <kbr@chromium.org>
Date: Wed May 09 18:37:21 2018

Reland "Add Win10 Intel Experimental Release bot to chromium.gpu.fyi"

This reverts commit 6b388edeed6b2bec30ad29872b23d9319d5d9c8a.

Reason for revert: bot is deployed; start running tests on it.

Original change's description:
> Revert "Add Win10 Intel Experimental Release bot to chromium.gpu.fyi"
> 
> This reverts commit 7225b5e672b4af3aad516d952ef6f73e35d8159a.
> 
> Reason for revert:  crbug.com/839635 
> 
> Original change's description:
> > Add Win10 Intel Experimental Release bot to chromium.gpu.fyi
> > 
> > BUG= 838970 
> > R=​kbr@chromium.org
> > 
> > Change-Id: Iabfd3eb97d28323e9f622d92e7c1dffff3f27090
> > Reviewed-on: https://chromium-review.googlesource.com/1041145
> > Commit-Queue: Kenneth Russell <kbr@chromium.org>
> > Reviewed-by: Kenneth Russell <kbr@chromium.org>
> 
> TBR=zmo@chromium.org,kbr@chromium.org
> 
> Change-Id: I9413563c7ef2fe1ab24fb85ad4b011069c602ede
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Bug:  838970 
> Reviewed-on: https://chromium-review.googlesource.com/1043365
> Reviewed-by: Zhenyao Mo <zmo@chromium.org>
> Commit-Queue: Zhenyao Mo <zmo@chromium.org>

TBR=zmo@chromium.org,kbr@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug:  838970 
Change-Id: Ia3ab6afc7302e504882a0b7e8e1870794dfb9903
Reviewed-on: https://chromium-review.googlesource.com/1052679
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Kenneth Russell <kbr@chromium.org>

[modify] https://crrev.com/d118e1d7141e9f1c2b95e1ce1f5480d4af310484/scripts/slave/recipe_modules/chromium_tests/chromium_gpu_fyi.py

Comment 15 by kbr@chromium.org, May 9 2018

Blocking: 841604

Comment 16 by kbr@chromium.org, May 9 2018

Blockedon: 841408
Project Member

Comment 17 by bugdroid1@chromium.org, May 9 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4d1bb448ed20272310bf769ee1b25dca3af3dff4

commit 4d1bb448ed20272310bf769ee1b25dca3af3dff4
Author: Kenneth Russell <kbr@chromium.org>
Date: Wed May 09 23:36:37 2018

Add LUCI migration app step to GPU bot deployment docs.

Tbr: nodir@chromium.org
No-Try: True
Bug:  838970 
Change-Id: I3593ef2201592358bb625bc115a6c4009da283a1
Reviewed-on: https://chromium-review.googlesource.com/1053353
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#557377}
[modify] https://crrev.com/4d1bb448ed20272310bf769ee1b25dca3af3dff4/docs/gpu/gpu_testing_bot_details.md

Comment 18 by kbr@chromium.org, May 10 2018

Cc: yunchao...@intel.com
Status: Started (was: Assigned)
Unfortunately the new driver is failing a lot of video-related WebGL conformance tests on ANGLE's D3D9 backend, and also one on the D3D11 backend. See this first fully working build:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20FYI%20Exp%20Release%20%28Intel%20HD%20630%29/14

Looking at this log:
https://chromium-swarm.appspot.com/task?id=3d60653b554c9710&refresh=10&show_raw=1

we see crashes inside the DXVA implementation:

	Last event: a14.3c8: Invalid handle - code c0000008 (first/second chance not available)
	  debugger time: Wed May  9 18:06:44.482 2018 (UTC - 7:00)
	ChildEBP RetAddr  Args to Child              
	WARNING: Stack unwind information not available. Following frames may be wrong.
	11cdd2d0 6578a976 08dd1e28 11cdd2e4 119a9bc8 win32u!NtGdiDdDDIWaitForSynchronizationObjectFromCpu+0xc
	11cdd2fc 65784a70 119a9c60 119a9c88 705f6e65 igd9dxva32!MOS_GetMemNinjaCounterGfx+0x8766
	11cdd5b4 65786ce3 08de3120 00000001 00000008 igd9dxva32!MOS_GetMemNinjaCounterGfx+0x2860
	11cdd7d8 657ace6f 00000001 00000000 00000000 igd9dxva32!MOS_GetMemNinjaCounterGfx+0x4ad3
	11cdd804 657a4ab8 00000001 00000001 00000000 igd9dxva32!MOS_GetMemNinjaCounterGfx+0x2ac5f
	11cdd824 65bda1f5 117e9908 11cdd900 00000000 igd9dxva32!MOS_GetMemNinjaCounterGfx+0x228a8
	11cddfe8 65ab1f05 1343f7c8 65a99c80 134421d8 igd9dxva32!MOS_GetMemNinjaCounterGfx+0x457fe5
	11cde014 65aadda4 1362a3a0 1362b390 1362b390 igd9dxva32!MOS_GetMemNinjaCounterGfx+0x32fcf5
	11cde05c 65ab4b41 11cde7ec 11cde778 13420980 igd9dxva32!MOS_GetMemNinjaCounterGfx+0x32bb94
	11cde634 65894639 13420980 11cde7ec 11cde778 igd9dxva32!MOS_GetMemNinjaCounterGfx+0x332931
	11cde730 65895a8a 11cde7ec 11cde778 11cde7ec igd9dxva32!MOS_GetMemNinjaCounterGfx+0x112429
	11cde754 65894d5e 11cde7ec 11cde778 cb1e2488 igd9dxva32!MOS_GetMemNinjaCounterGfx+0x11387a
	11cde7d4 65894572 11cde7ec 13420980 658942c0 igd9dxva32!MOS_GetMemNinjaCounterGfx+0x112b4e
	11cde964 657d5a52 11cde97c 133f4380 657d5940 igd9dxva32!MOS_GetMemNinjaCounterGfx+0x112362
	11cdea5c 6579b3ec 11a196b0 00000000 11cdeba0 igd9dxva32!MOS_GetMemNinjaCounterGfx+0x53842
	*** WARNING: Unable to verify checksum for d3d9.dll
	*** ERROR: Symbol file could not be found.  Defaulted to export symbols for d3d9.dll - 
	11cdea8c 691a8b88 08dd3d40 11cdead0 08dd1e28 igd9dxva32!MOS_GetMemNinjaCounterGfx+0x191dc
	11cdeaa8 690dc611 0ff60040 11cdead0 11cdecd8 d3d9!Direct3DShaderValidatorCreate9+0x46af8
	11cdeb64 690bd852 11cdeba0 690bd750 15af3018 d3d9!Direct3DCreate9Ex+0x30be1
	*** WARNING: Unable to verify checksum for dxva2.dll
	*** ERROR: Symbol file could not be found.  Defaulted to export symbols for dxva2.dll - 
	11cdec98 69b74111 118ddd80 11cdecd8 69b71038 d3d9!Direct3DCreate9Ex+0x11e22
	*** WARNING: Unable to verify checksum for chrome_child.dll
	No .natvis files found at c:\b\s\w\ir\out\Release\cdb\Visualizers.
	No .natvis files found at C:\Users\chrome-bot\AppData\Local\Dbg\Visualizers.
	11cded60 6d8e8992 15af3014 117cf920 11cdedc0 dxva2!DXVA2CreateVideoService+0x491
	11cdefc8 6d8eb81c 11801320 117cf920 00000000 chrome_child!media::DXVAVideoDecodeAccelerator::CopySurface+0x280
	11cdefe4 6bba8784 0984e6b0 6bb5f267 000003c8 chrome_child!base::internal::Invoker<base::internal::BindState<void (__thiscall media::DXVAVideoDecodeAccelerator::*)(IDirect3DSurface9 *,IDirect3DSurface9 *,int,int,gfx::ColorSpace const &),base::internal::UnretainedWrapper<media::DXVAVideoDecodeAccelerator>,base::internal::UnretainedWrapper<IDirect3DSurface9>,base::internal::UnretainedWrapper<IDirect3DSurface9>,int,int,gfx::ColorSpace>,void __cdecl(void)>::RunOnce+0x22
	11cdf0f8 6bbd5689 6e8ea86b 11cdf3d8 00000000 chrome_child!base::debug::TaskAnnotator::RunTask+0x134
	11cdf1cc 6bb8c8b7 11cdf3d8 098627d8 6f0b1a42 chrome_child!base::internal::IncomingTaskQueue::RunTask+0x69
	11cdf2f4 6bb8cc0d 11cdf3d8 6bb5ee57 0985fb88 chrome_child!base::MessageLoop::RunTask+0x207
	11cdf3d0 6bb8ce3a 00000000 6ea8c840 6f0b1a42 chrome_child!base::MessageLoop::DeferOrRunPendingTask+0x9d
	11cdf530 6bbd7ccc 0985fab0 0985faa8 00000301 chrome_child!base::MessageLoop::DoWork+0x1fa
	11cdf554 6bb8c3d4 098627d8 11cdf7e0 6a805aaf chrome_child!base::MessagePumpDefault::Run+0x9c
	11cdf624 6bb52e1c 00000001 11cdf7e0 778f1679 chrome_child!base::MessageLoop::Run+0x74
	11cdf6f8 6bb66e04 000003c8 000003c8 000003c8 chrome_child!base::RunLoop::Run+0xcc
	11cdf7c4 6bb67097 11cdf7e0 00000004 098663c8 chrome_child!base::Thread::Run+0xa4
	11cdf8a8 6bb5ac45 09831054 0000081c 0000081c chrome_child!base::Thread::ThreadMain+0x277
	
Intel folks, can you please investigate these failures in-house? We would really appreciate it if you would run the WebGL conformance suite in-house in all of the configurations that we do, in particular with ANGLE's D3D9 backend, in order to catch these failures up front.

We'll need to pause this driver upgrade until a new driver can be produced which fixes these issues.

Cc: fjhenigman@chromium.org
 Issue 841776  has been merged into this issue.

Comment 20 by kbr@chromium.org, May 10 2018

Blockedon: 841789

Comment 21 by kbr@chromium.org, May 11 2018

Blockedon: 841604
Blocking: -841604
Project Member

Comment 22 by bugdroid1@chromium.org, May 11 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/13ae25f1b218c05341a1c6d9f0b25811c0242d6b

commit 13ae25f1b218c05341a1c6d9f0b25811c0242d6b
Author: Kenneth Russell <kbr@chromium.org>
Date: Fri May 11 01:55:51 2018

Remove LUCI migration app step from GPU bot deployment docs.

The app was fixed and it's no longer necessary.

Bug:  838970 , 841604
Change-Id: I1bafa500ca3ad2845b15110e3e7bd0effc9fe471
Tbr: nodir@chromium.org
Reviewed-on: https://chromium-review.googlesource.com/1054357
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#557765}
[modify] https://crrev.com/13ae25f1b218c05341a1c6d9f0b25811c0242d6b/docs/gpu/gpu_testing_bot_details.md

Blocking: 843369

Comment 24 by kbr@chromium.org, Jun 23 2018

Blocking: 854100
Blockedon: 879651
Per https://bugs.chromium.org/p/chromium/issues/detail?id=841789#c14 : we are now going to target driver version 24.20.100.6286, which fixes the D3D9 crash issue seen earlier.

Summary: Update Windows Intel bot driver to 24.20.100.6286 (was: Update Windows Intel bot driver to 24.20.100.6025)
Project Member

Comment 28 by bugdroid1@chromium.org, Sep 5

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/212d5c5369080305b74a1bb65fab755b2dec2a72

commit 212d5c5369080305b74a1bb65fab755b2dec2a72
Author: Zhenyao Mo <zmo@chromium.org>
Date: Wed Sep 05 01:57:30 2018

Updating GPU/FYI Win/Intel experimental bot to 24.20.100.6286

BUG= 838970 
TEST=waterfall
R=kbr@chromium.org

Change-Id: Ic4f9fba1b7b858ee17b8898a4457810c7a167f90
Reviewed-on: https://chromium-review.googlesource.com/1200464
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#588740}
[modify] https://crrev.com/212d5c5369080305b74a1bb65fab755b2dec2a72/testing/buildbot/chromium.gpu.fyi.json
[modify] https://crrev.com/212d5c5369080305b74a1bb65fab755b2dec2a72/testing/buildbot/waterfalls.pyl

The experimental bot is green after the upgrade to the new driver and after re-enabling the previously failing test suites in https://chromium-review.googlesource.com/1207652 .

Link to the bot:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20FYI%20Exp%20Release%20%28Intel%20HD%20630%29?limit=200

and the first passing build with the previously-failing test suites re-enabled:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20FYI%20Exp%20Release%20%28Intel%20HD%20630%29/1233

Let's proceed with upgrading the rest of the bots - allowing jobs to run on both driver versions, asking the Labs team to upgrade the rest of them, and then targeting the jobs to the new driver version.

Blocking: -843369
Blockedon: 843369
Labels: -Pri-2 Pri-1
Raising to P1 so this gets attention when zmo is available.

Project Member

Comment 33 by bugdroid1@chromium.org, Sep 17

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5f6c76d8090b709c550eb62697fba6e741d8199f

commit 5f6c76d8090b709c550eb62697fba6e741d8199f
Author: Zhenyao Mo <zmo@chromium.org>
Date: Mon Sep 17 23:03:14 2018

Allow Win Intel bots on GPU FYI waterfalls to run with both older/newer drivers

older driver: 23.20.16.4877
newer driver: 24.20.100.6286

BUG= 838970 
TEST=waterfall
R=kbr@chromium.org

Change-Id: Ie7a92e8a8641930a4b1daa3ef9a3fa38e6d17bc6
Reviewed-on: https://chromium-review.googlesource.com/1225609
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Zhenyao Mo <zmo@chromium.org>
Cr-Commit-Position: refs/heads/master@{#591861}
[modify] https://crrev.com/5f6c76d8090b709c550eb62697fba6e741d8199f/testing/buildbot/chromium.gpu.fyi.json
[modify] https://crrev.com/5f6c76d8090b709c550eb62697fba6e741d8199f/testing/buildbot/waterfalls.pyl

Blockedon: 885225
Project Member

Comment 35 by bugdroid1@chromium.org, Sep 18

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7dede209712feb22e96826d1dd2c56fe3f6f1570

commit 7dede209712feb22e96826d1dd2c56fe3f6f1570
Author: Zhenyao Mo <zmo@chromium.org>
Date: Tue Sep 18 17:57:21 2018

Update the GPU bot driver update steps sample CL.

The original sample is outdated so replace with a newer one.

BUG= 838970 

Change-Id: Id257239e3fdb6424bd7bb9b3bdaea563dd1743dc
Reviewed-on: https://chromium-review.googlesource.com/1228783
Commit-Queue: Zhenyao Mo <zmo@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#592106}
[modify] https://crrev.com/7dede209712feb22e96826d1dd2c56fe3f6f1570/docs/gpu/gpu_testing_bot_details.md

Blockedon: 886985
Project Member

Comment 37 by bugdroid1@chromium.org, Sep 28

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ef5f45aebcc6b09dca18c841d43c85498ea68e5e

commit ef5f45aebcc6b09dca18c841d43c85498ea68e5e
Author: Zhenyao Mo <zmo@chromium.org>
Date: Fri Sep 28 23:52:52 2018

Lock Win Intel GPU bots on GPU FYI to driver 24.20.100.6286

BUG= 838970 
TEST=GPU FYI waterfall
R=kbr@chromium.org

Change-Id: Idf1b701539c427a63ed5648396a3c6588d305733
Reviewed-on: https://chromium-review.googlesource.com/1252630
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Zhenyao Mo <zmo@chromium.org>
Cr-Commit-Position: refs/heads/master@{#595257}
[modify] https://crrev.com/ef5f45aebcc6b09dca18c841d43c85498ea68e5e/testing/buildbot/chromium.gpu.fyi.json
[modify] https://crrev.com/ef5f45aebcc6b09dca18c841d43c85498ea68e5e/testing/buildbot/waterfalls.pyl

Status: Fixed (was: Started)
Project Member

Comment 39 by bugdroid1@chromium.org, Sep 29

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/07fc9d97256cbdeee6307a5b89cddba57413e9ed

commit 07fc9d97256cbdeee6307a5b89cddba57413e9ed
Author: John Budorick <jbudorick@chromium.org>
Date: Sat Sep 29 04:50:11 2018

Revert "Lock Win Intel GPU bots on GPU FYI to driver 24.20.100.6286"

This reverts commit ef5f45aebcc6b09dca18c841d43c85498ea68e5e.

Reason for revert: see comments on CL.

Original change's description:
> Lock Win Intel GPU bots on GPU FYI to driver 24.20.100.6286
> 
> BUG= 838970 
> TEST=GPU FYI waterfall
> R=​kbr@chromium.org
> 
> Change-Id: Idf1b701539c427a63ed5648396a3c6588d305733
> Reviewed-on: https://chromium-review.googlesource.com/1252630
> Reviewed-by: Kenneth Russell <kbr@chromium.org>
> Commit-Queue: Zhenyao Mo <zmo@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#595257}

TBR=zmo@chromium.org,kbr@chromium.org

Change-Id: Ie682295188ce2a834124a3abbf02c1e59fe786ba
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  838970 
Reviewed-on: https://chromium-review.googlesource.com/1253075
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#595300}
[modify] https://crrev.com/07fc9d97256cbdeee6307a5b89cddba57413e9ed/testing/buildbot/chromium.gpu.fyi.json
[modify] https://crrev.com/07fc9d97256cbdeee6307a5b89cddba57413e9ed/testing/buildbot/waterfalls.pyl

Status: Assigned (was: Fixed)
Thanks for reverting John. Updating status.
Cc: jmad...@chromium.org
I think the swarming Intel fleet might have inadvertently been upgraded to the wrong driver version.

Tasks are still failing after the revert in #39, e.g.:

https://ci.chromium.org/p/chromium/builders/luci.chromium.try/win_optional_gpu_tests_rel/8495

Looking at the fleet they seem to be on driver version 23.20.16.4973:

https://chromium-swarm.appspot.com/botlist?c=id&c=os&c=task&c=status&c=gpu&f=os%3AWindows-10&f=cpu%3Ax86-64&f=pool%3AChrome-GPU&f=gpu%3AIntel%20(8086)&l=100&s=id%3Aasc

We should probably change the settings in the waterfalls.pyl script to use the driver version the bots have now to get try jobs running again.
https://chromium-review.googlesource.com/c/chromium/src/+/1252466 updates the driver version in //testing/buildbot. Will land once today's maintenance is over.
Some updates here. https://bugs.chromium.org/p/chromium/issues/detail?id=885225#c14

Windows forced an update and we're re-pushing 100.6286


Do you disable auto updates on Windows 10 bots? I think for the stability of testing, it's better to disable it. 
These are all back on 100.6286

We now have a way to block updates and will be rolling that out to the lab shortly.

CL from #43 updated given #46.
Big thank you for the speedy action on this!
Project Member

Comment 49 by bugdroid1@chromium.org, Sep 29

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e819b49af8f9011bec234b64662002bb78617f27

commit e819b49af8f9011bec234b64662002bb78617f27
Author: John Budorick <jbudorick@chromium.org>
Date: Sat Sep 29 20:26:08 2018

chromium.gpu.fyi: update kaby lake driver version to 24.20.100.6286.

This is a reland of crrev.com/c/1252630

TBR=kbr@chromium.org,zmo@chromium.org

Cq-Include-Trybots: luci.chromium.try:win_optional_gpu_tests_rel
Bug:  838970 
Change-Id: I6590724352d1458bbd1c2a1fb050968ad1af025d
Reviewed-on: https://chromium-review.googlesource.com/1252466
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#595328}
[modify] https://crrev.com/e819b49af8f9011bec234b64662002bb78617f27/testing/buildbot/chromium.gpu.fyi.json
[modify] https://crrev.com/e819b49af8f9011bec234b64662002bb78617f27/testing/buildbot/waterfalls.pyl

Thank you John B. and John W. for your great help on this upgrade!

Status: Fixed (was: Assigned)
Closing this as fixed and linking to new bugs.

Blocking: 894127

Sign in to add a comment