The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/73c5accdecf29b1afba76a09f4a6a2ca964eaa1c

commit 73c5accdecf29b1afba76a09f4a6a2ca964eaa1c
Author: Jamie Madill <jmadill@chromium.org>
Date: Thu Oct 05 01:10:46 2017

Specify Win/NVIDIA GPU bots driver version.

Explicitly specifying the GPU driver version in the config will allow
us to safely test new driver versions. Because the tests will pick
machines of a particular driver version specified in the json, we
can ask infra to upgrade a few machines, then run a try job using the
new driver version.

This modifies the generate script to allow use of wildcards in test
dimension set specifiers. Wildcards are only supported as the last
character in the dimension. This allows us to more easily match
ranges of driver versions.

The current NVIDIA driver version used by the bots is 372.54.

R=kbr@chromium.org
BUG= 771654 

Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
Change-Id: I5875b9e91fbe8d2d626a8af20005161bc3f082bb
Reviewed-on: https://chromium-review.googlesource.com/699761
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Cr-Commit-Position: refs/heads/master@{#506600}
[modify] https://crrev.com/73c5accdecf29b1afba76a09f4a6a2ca964eaa1c/content/test/gpu/generate_buildbot_json.py
[modify] https://crrev.com/73c5accdecf29b1afba76a09f4a6a2ca964eaa1c/testing/buildbot/chromium.gpu.fyi.json
[modify] https://crrev.com/73c5accdecf29b1afba76a09f4a6a2ca964eaa1c/testing/buildbot/chromium.gpu.json
[modify] https://crrev.com/73c5accdecf29b1afba76a09f4a6a2ca964eaa1c/testing/buildbot/client.v8.fyi.json

Hey John, could you help me find someone to work with to test an NVIDIA driver upgrade before we ask for a deployment? Currently the NVIDIA Windows swarm runs driver version 372.54, could you maybe upgrade 3-4 of them to run the newest available driver? Currently this is 385.69 (http://us.download.nvidia.com/Windows/385.69/385.69-desktop-win8-win7-64bit-international-whql.exe) - there is a different download for Windows 10, but I think for now we should target Windows 7/Server 2008.

I've also filed issue 771840 for setting up a testing machine on the GPU.FYI waterfall.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1ebe145d35acada0f26e854fc623d1d62c17606c

commit 1ebe145d35acada0f26e854fc623d1d62c17606c
Author: Corentin Wallez <cwallez@chromium.org>
Date: Thu Oct 05 17:28:31 2017

Revert "Specify Win/NVIDIA GPU bots driver version."

This reverts commit 73c5accdecf29b1afba76a09f4a6a2ca964eaa1c.

Reason for revert: Breaks the Win 10 GPU FYI bots that can't find swarming bots with the required configuration anymore.

Original change's description:
> Specify Win/NVIDIA GPU bots driver version.
> 
> Explicitly specifying the GPU driver version in the config will allow
> us to safely test new driver versions. Because the tests will pick
> machines of a particular driver version specified in the json, we
> can ask infra to upgrade a few machines, then run a try job using the
> new driver version.
> 
> This modifies the generate script to allow use of wildcards in test
> dimension set specifiers. Wildcards are only supported as the last
> character in the dimension. This allows us to more easily match
> ranges of driver versions.
> 
> The current NVIDIA driver version used by the bots is 372.54.
> 
> R=​kbr@chromium.org
> BUG= 771654 
> 
> Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
> Change-Id: I5875b9e91fbe8d2d626a8af20005161bc3f082bb
> Reviewed-on: https://chromium-review.googlesource.com/699761
> Reviewed-by: Kenneth Russell <kbr@chromium.org>
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#506600}

TBR=jmadill@chromium.org,kbr@chromium.org

Change-Id: Id8c6f60074994e7a7ae9a62a9e0f6658ba43fc6d
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  771654 
Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
Reviewed-on: https://chromium-review.googlesource.com/702734
Reviewed-by: Corentin Wallez <cwallez@chromium.org>
Commit-Queue: Corentin Wallez <cwallez@chromium.org>
Cr-Commit-Position: refs/heads/master@{#506778}
[modify] https://crrev.com/1ebe145d35acada0f26e854fc623d1d62c17606c/content/test/gpu/generate_buildbot_json.py
[modify] https://crrev.com/1ebe145d35acada0f26e854fc623d1d62c17606c/testing/buildbot/chromium.gpu.fyi.json
[modify] https://crrev.com/1ebe145d35acada0f26e854fc623d1d62c17606c/testing/buildbot/chromium.gpu.json
[modify] https://crrev.com/1ebe145d35acada0f26e854fc623d1d62c17606c/testing/buildbot/client.v8.fyi.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c3b2de66608d423717a75d2ed946d5b3446d99e5

commit c3b2de66608d423717a75d2ed946d5b3446d99e5
Author: Jamie Madill <jmadill@chromium.org>
Date: Thu Oct 05 19:30:31 2017

Re-land "Specify Win/NVIDIA GPU bots driver version."

Explicitly specifying the GPU driver version in the config will allow
us to safely test new driver versions. Because the tests will pick
machines of a particular driver version specified in the json, we
can ask infra to upgrade a few machines, then run a try job using the
new driver version.

This modifies the generate script to allow use of wildcards in test
dimension set specifiers. Wildcards are only supported as the last
character in the dimension. This allows us to more easily match
ranges of driver versions.

Re-land fixes the driver version for Windows 10 bots.

The current NVIDIA driver version used by the bots is 372.54 for the
Windows 7 bots and 376.19 for Windows 10.

R=kbr@chromium.org
BUG= 771654 

Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
Change-Id: Ie06cedefa0e4c4afc598ab7148d192c3f60eca34
Reviewed-on: https://chromium-review.googlesource.com/702855
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Cr-Commit-Position: refs/heads/master@{#506824}
[modify] https://crrev.com/c3b2de66608d423717a75d2ed946d5b3446d99e5/content/test/gpu/generate_buildbot_json.py
[modify] https://crrev.com/c3b2de66608d423717a75d2ed946d5b3446d99e5/testing/buildbot/chromium.gpu.fyi.json
[modify] https://crrev.com/c3b2de66608d423717a75d2ed946d5b3446d99e5/testing/buildbot/chromium.gpu.json
[modify] https://crrev.com/c3b2de66608d423717a75d2ed946d5b3446d99e5/testing/buildbot/client.v8.fyi.json

Ken, now that we have vm345-m1, what work needs to happen to get it ready to test the new driver version?

The rough instructions are on https://www.chromium.org/developers/testing/gpu-testing/gpu-bot-details#TOC-How-to-add-a-new-swarmed-bot-to-the-chromium.gpu.fyi-waterfall but I'll help with the tools/build and initial Chromium-side CLs. https://chromium-review.googlesource.com/706141 is up for review to add the bot to the chromium.gpu.fyi waterfall.

Once the new bot is running tests with the current driver version, we can ask the Labs team to upgrade a few Swarming bots, and target the new bot at the new driver version.

Assigning to you Jamie as owner of the overall task.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/414a0539b7a7111587ed55ff7be392069a11e7be

commit 414a0539b7a7111587ed55ff7be392069a11e7be
Author: Kenneth Russell <kbr@chromium.org>
Date: Sat Oct 07 17:39:44 2017

Deploy "Win7 Experimental Release (NVIDIA)" on chromium.gpu.fyi.

BUG= 771654 
TBR=jmadill@chromium.org
NOTRY=true

Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
Change-Id: Ib9b6f177c1b2574fb282baaed9fd8d68107685a6
Reviewed-on: https://chromium-review.googlesource.com/706854
Commit-Queue: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#507298}
[modify] https://crrev.com/414a0539b7a7111587ed55ff7be392069a11e7be/content/test/gpu/generate_buildbot_json.py
[modify] https://crrev.com/414a0539b7a7111587ed55ff7be392069a11e7be/testing/buildbot/chromium.gpu.fyi.json

After the CL above, need:

1) A fix to Issue 772660 to un-break the presubmit checks in the tools/build workspace.

2) https://chromium-review.googlesource.com/706141
to deploy the new VM on chromium.gpu.fyi.

3) A restart of the chromium.gpu.fyi waterfall and a reboot of the new VM.

4) Upgrading a few (4?) of the Win7 NVIDIA bots in the Swarming pool to the new driver version.

5) Fixing the TODO in generate_buildbot_json.py to target the experimental builder to the new driver version.

Jamie, not sure if Issue 772660 will be addressed today, but if not then please feel free to take over https://chromium-review.googlesource.com/706141 from me (create a new CL if necessary and abandon mine).

Great, thanks for your help Ken.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/3bce2b4e847a0da8e0fc101b55e17b4f1c45265e

commit 3bce2b4e847a0da8e0fc101b55e17b4f1c45265e
Author: Kenneth Russell <kbr@chromium.org>
Date: Sat Oct 07 18:50:21 2017

Deploy vm345-m1 as "Win7 Experimental Release (NVIDIA)".

BUG= 771654 
TBR=dpranke@chromium.org
NOTRY=true

Change-Id: Ia8244dbacb4d3285fcfba3495c8a25d6d0abac66
Reviewed-on: https://chromium-review.googlesource.com/706141
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>

[modify] https://crrev.com/3bce2b4e847a0da8e0fc101b55e17b4f1c45265e/scripts/slave/recipe_modules/chromium_tests/chromium_gpu_fyi.py
[modify] https://crrev.com/3bce2b4e847a0da8e0fc101b55e17b4f1c45265e/masters/master.chromium.gpu.fyi/builders.pyl

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/master-manager/+/139033aadb5b055ece0668b84aa346adea60b210

commit 139033aadb5b055ece0668b84aa346adea60b210
Author: Kenneth Russell <kbr@google.com>
Date: Sat Oct 07 20:35:58 2017

I can do this for you.  Double checking you wanted 4 slaves updated?  If so assign this bug to me.

Also there is now a more recent driver that has been released. 
 Want this one installed?  https://us.download.nvidia.com/Windows/387.92/387.92-desktop-win8-win7-64bit-international-whql.exe    

Thanks Peter. Yes please go ahead and upgrade 4 slaves from the Windows 7 NVIDIA GPU pool. Please do use the newest driver version you linked (387.92).

build92{0..3}-m4 have 387.92 installed.

pschmidt@:~/txt$ swarming.py bots -d gpu "10de:104a-23.21.13.8792" --swarming chromium-swarm.appspot.com
build920-m4
  {"cores": ["8"], "cpu": ["x86", "x86-64", "x86-64-E3-1230_v5"], "gpu": ["10de", "10de:104a", "10de:104a-23.21.13.8792"], "id": ["build920-m4"], "integrity": ["high"], "locale": ["en_US.cp1252"], "machine_type": ["n1-standard-8"], "os": ["Windows", "Windows-2008ServerR2", "Windows-2008ServerR2-SP1"], "pool": ["Chrome-GPU"], "python": ["2.7.13"]}
build921-m4
  {"cores": ["8"], "cpu": ["x86", "x86-64", "x86-64-E3-1230_v5"], "gpu": ["10de", "10de:104a", "10de:104a-23.21.13.8792"], "id": ["build921-m4"], "integrity": ["high"], "locale": ["en_US.cp1252"], "machine_type": ["n1-standard-8"], "os": ["Windows", "Windows-2008ServerR2", "Windows-2008ServerR2-SP1"], "pool": ["Chrome-GPU"], "python": ["2.7.13"]}
build922-m4
  {"cores": ["8"], "cpu": ["x86", "x86-64", "x86-64-E3-1230_v5"], "gpu": ["10de", "10de:104a", "10de:104a-23.21.13.8792"], "id": ["build922-m4"], "integrity": ["high"], "locale": ["en_US.cp1252"], "machine_type": ["n1-standard-8"], "os": ["Windows", "Windows-2008ServerR2", "Windows-2008ServerR2-SP1"], "pool": ["Chrome-GPU"], "python": ["2.7.13"]}
build923-m4
  {"cores": ["8"], "cpu": ["x86", "x86-64", "x86-64-E3-1230_v5"], "gpu": ["10de", "10de:104a", "10de:104a-23.21.13.8792"], "id": ["build923-m4"], "integrity": ["high"], "locale": ["en_US.cp1252"], "machine_type": ["n1-standard-8"], "os": ["Windows", "Windows-2008ServerR2", "Windows-2008ServerR2-SP1"], "pool": ["Chrome-GPU"], "python": ["2.7.13"]}

Thanks Peter. Going to steal this back to try them out.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e88ee7d6a870ede722e40a8e7abb1f0aa6c60013

commit e88ee7d6a870ede722e40a8e7abb1f0aa6c60013
Author: Jamie Madill <jmadill@chromium.org>
Date: Wed Oct 11 18:30:56 2017

Use Windows NVIDIA 387.82 driver on experimental bot.

This bot will be a testing grounds for new driver versions. We
have already upgraded a few swarming machines to use the new
driver version, in preparation for this change. Once we have
determined the new driver version is stable, we can proceed with
upgrading more machines in the swarm.

BUG= 771654 
R=kbr@chromium.org

Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
Change-Id: Icffeb74c80170ac832b124556bd4a06d781e612b
Reviewed-on: https://chromium-review.googlesource.com/713457
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Cr-Commit-Position: refs/heads/master@{#508042}
[modify] https://crrev.com/e88ee7d6a870ede722e40a8e7abb1f0aa6c60013/content/test/gpu/generate_buildbot_json.py
[modify] https://crrev.com/e88ee7d6a870ede722e40a8e7abb1f0aa6c60013/testing/buildbot/chromium.gpu.fyi.json

Ken, what are the next steps after confirming the driver version is stable on the experimental bot? Should we ask for an upgrade for 50% of the fleet, move them to the new driver version, then upgrade the rest?

Note the webgl2_conformance_gl_tests failures in build #32 are unrelated, and are being handled in  issue 773823 .

Yes, that's the correct procedure. Try to work with the Labs team to minimize the time that the fleet's split in two. Thanks.

Peter, could I work with you to upgrade the bots as described in comment 18? I'll be available most of today, but away from about 3pm PST to 7pm PST. Tomorrow I'll be available early in the morning.

We would like to upgrade 50% of the fleet to the new driver version, then I'll land a CL which would switch our tests to use the new driver, then upgrade the rest. This would also include 2/4 of the Windows 10 bots.

Let's do this on Monday?   

That sounds good.

Half the win7 pool has been upgraded.  Will work on build{122,163}-m1 next as a different installer is required.  These are two win10 slaves.  

[build108-m1] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build109-m1] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build118-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build119-m1] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build119-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build120-m1] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build120-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build121-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build122-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build123-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build124-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build125-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build126-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build127-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build128-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build129-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build130-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build131-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build132-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build133-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build134-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build135-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build136-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build137-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build138-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build139-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build140-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build141-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build142-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build143-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build144-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build500-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build501-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build502-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build503-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build504-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build505-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build506-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build507-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build508-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build509-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build510-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build511-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build512-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build513-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build514-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build515-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build516-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build517-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build518-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build519-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build521-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build523-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build83-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build84-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build85-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build86-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build87-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build88-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build89-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build90-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build91-m1] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build91-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build92-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build93-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build94-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build95-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792

The win10 ones:

[build122-m1] out: NVIDIA GeForce GT 610 23.21.13.8792
[build163-m1] out: NVIDIA GeForce GT 610 23.21.13.8792

Thanks Peter. Landing Jamie's CL which targets the Swarming jobs at the new driver version in:
https://chromium-review.googlesource.com/716658

The presubmit step seems to have gotten stuck launching https://chromium-review.googlesource.com/c/chromium/src/+/716658 . Ken can you -2 and +2 again, and use no-presubmit at your discretion?

Unfortunately it looks like the presubmit bot is stuck. Filed  Issue 775250 .

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ac76a0ac0933a3ad569997f2b71f69328f42b3e7

commit ac76a0ac0933a3ad569997f2b71f69328f42b3e7
Author: Jamie Madill <jmadill@chromium.org>
Date: Tue Oct 17 02:00:18 2017

Upgrade Windows NVIDIA GPU bot drivers.

This upgrades the driver version used in the tester bots from 372.54
to 387.92. This CL should land as the bots are being switched over to
the new driver version, and the upgrade should be finished after it
lands.

BUG= 771654 

NOTRY=true

Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
Change-Id: I9ed27aeeab48604e3c3ec862f58d474934f06a7f
Reviewed-on: https://chromium-review.googlesource.com/716658
Commit-Queue: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#509215}
[modify] https://crrev.com/ac76a0ac0933a3ad569997f2b71f69328f42b3e7/content/test/gpu/generate_buildbot_json.py
[modify] https://crrev.com/ac76a0ac0933a3ad569997f2b71f69328f42b3e7/testing/buildbot/chromium.gpu.fyi.json
[modify] https://crrev.com/ac76a0ac0933a3ad569997f2b71f69328f42b3e7/testing/buildbot/chromium.gpu.json
[modify] https://crrev.com/ac76a0ac0933a3ad569997f2b71f69328f42b3e7/testing/buildbot/client.v8.fyi.json

Apologies for the delay Peter. We're now ready to proceed with the upgrade of the rest of the bots.

Started upgrading the rest.

All done.
build638-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build639-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build640-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build641-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build642-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build643-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build644-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build645-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build646-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build647-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build648-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build649-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build650-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build844-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build846-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build847-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build848-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build849-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build850-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build851-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build852-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build853-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build854-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build855-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build856-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build857-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build858-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build859-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build860-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build861-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build862-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build863-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build864-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build865-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build866-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build867-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build868-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build869-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build870-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build871-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build872-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build873-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build874-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build875-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build876-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build877-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build878-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build879-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build880-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build881-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build882-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build883-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build899-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build900-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build901-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build902-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build903-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build904-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build905-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build906-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build907-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build908-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build909-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build910-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build911-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build912-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build913-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build914-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build915-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build916-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build917-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build918-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build919-m4] out: NVIDIA GeForce GT 610                   23.21.13.8792
[build123-m1] out: NVIDIA GeForce GT 610 23.21.13.8792  #win10
[build124-m1] out: NVIDIA GeForce GT 610 23.21.13.8792  #win10

Excellent, thanks very much Peter. There's a follow-on issue to upgrade the NVIDIA perf GPU bots drivers now that these have been verified in  issue 767411 .