would this be due to the name change?

Can you include a link to the failing log?

https://pinpoint-dot-chromeperf.appspot.com/job/14003769c40000

I thought only the perf testers names changed? These errors are about the isolate hash not being found from the builders.

So linux-perf bots do use the new private swarming server, but that was back on April 12.   I am not sure the state of pinpoint.  I was under the impression that they weren't linked, adding Ned for more insight there.  

You are right that only the name of the perf testers changed, but this is also now the perf-id  crbug.com/817823, so the value that we pass to the dashboard as the 'bot'.  

Hmm yeah this seems to be more recent, jobs well past April 12 have succeeded.

+ashleymarie

Tracked this down, believe this is down to a change in the target, telemetry_perf_tests/telemetry_perf_webview_tests -> performance_test_suite

Emily/Ashley can you confirm the name change, and which bots/when it's used.

Yes this is the new isolate for our new recipe which is currently just linux-perf and mac-10_12_laptop_low_end-perf, the rest are still running telemetry_perf_tests

So I chatted with Simon offline.

There appear to be 3 potential issues here: 

1) bot name has changed from linux-release to linux-perf.  We deprecated the notion of perf-id and are sending down the configuration name as you see it on the waterfall for the bot name now.

2) The isolate target, ie. telemetry_perf_tests, is hardcoded and pinpoint doesn't know about performance_test_suite at all right now.

3) Once we get the new isolate, we don't want to run the script that is part of the target (run_performance_test suite.py: https://cs.chromium.org/chromium/src/testing/buildbot/gn_isolate_map.pyl?q=run_telemetry_benchmark_as_googletest&sq=package:chromium&dr=C&l=1093) we want to continue to run run_telemetry_benchmark_as_googletest.py that runs it a benchmark at a time.  run_performance_test_suite.py will try and run the entire benchmark suite in one execution.

#2 is further complicated by the fact some bots are using the new target performance_test_suite, while others use the old telemetry_perf_test/telemetry_webview_perf_test. Easy short-term fix for Pinpoint will be to do a fallback lookup, but if we fix that right now we will theoretically just hit problem #3.

Also for Emily, is everything now under performance_test_suite (ie. both webview/non-webview)?

No, we haven't migrated a webview bot yet.  In the future we will need another isolate that will be performance_test_suite_webview, but we haven't needed to implement that yet.

Since the perf waterfall isolate will now run the entire benchmark suite and Pinpoint only runs one benchmark (and often one story), their code paths will need to diverge. I can think of three places for this:

1. They run separate isolates. The builders will have to build both.
2. They run the same isolate, but with different commands. Pinpoint overrides the command in the isolate to provide its own command.
3. They run the same isolate and command, but with different arguments. run_performance_tests.py will need to provide the ability to filter for and run a single benchmark.

#3 sounds to me that it would involve the least code divergence and would be least disruptive, both now and in the future.

I like #3 as well, separate isolates and/or target scripts seems brittle.

We might need #3 for the sharding work later & perf smoke testing on CQ as well. +1 to #3 from me as well

I agree that we shouldn't add another script.  I will have to take a look at this code closer and see how we can fork it.  We already have a wrapper script we planned on deprecating and we can just use that full time in the isolate: https://cs.chromium.org/chromium/src/testing/scripts/run_performance_tests_wrapper.py?q=run_performance&sq=package:chromium&l=1

Sounds like #3 makes the most sense.  I will take ownership.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8a0edf5c40b72414592833275788e161f41b8739

commit 8a0edf5c40b72414592833275788e161f41b8739
Author: Emily Hanley <eyaich@google.com>
Date: Sat Apr 28 01:12:58 2018

Adding ability to trigger one benchmark in the performance_test_suite

Bug:  836037 
Change-Id: I9a927fe4655261a77965ca27cdfcf1055a9a0f1d
Reviewed-on: https://chromium-review.googlesource.com/1030773
Commit-Queue: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Reviewed-by: David Tu <dtu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#554597}
[modify] https://crrev.com/8a0edf5c40b72414592833275788e161f41b8739/testing/scripts/run_performance_tests.py

Ok the new logic has landed for the ability to trigger just one benchmark at a time.  I am passing this off to Dave for implementation on the pinpoint side: 

Iff isolate==performance_test_suite:

1) No longer pre-pend the benchmark as an arg, instead append a flag with a comma separated list of benchmarks to run.  In your case it will just be one, --benchmarks=<benchmark_to_run>

2) The perf results will be located in a new directory structure path that will include the benchmark name.  is instead of <isolated_out_dir>/perf_results.json it is now <isolated_out_dir>/<benchmark_to_run>/perf_results.json

The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/4f42243e85c8ff1829a76127c039f67623f50e5b

commit 4f42243e85c8ff1829a76127c039f67623f50e5b
Author: Simon <simonhatch@chromium.org>
Date: Fri May 04 01:47:28 2018

Dashboard - Use new isolate target for linux-perf.

Bug:  chromium:836037 
Change-Id: I0a48bcced9328b00c11acfba4bf3c8afacb06b33
Reviewed-on: https://chromium-review.googlesource.com/1042814
Reviewed-by: David Tu <dtu@chromium.org>
Commit-Queue: Simon Hatch <simonhatch@chromium.org>

[modify] https://crrev.com/4f42243e85c8ff1829a76127c039f67623f50e5b/dashboard/dashboard/pinpoint_request.py
[modify] https://crrev.com/4f42243e85c8ff1829a76127c039f67623f50e5b/dashboard/dashboard/pinpoint_request_test.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c4b0f83e9230f7176a68ec7f690634b1b25e7a5c

commit c4b0f83e9230f7176a68ec7f690634b1b25e7a5c
Author: catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Fri May 04 06:10:23 2018

Roll src/third_party/catapult/ 2048c078d..4f42243e8 (1 commit)

https://chromium.googlesource.com/catapult.git/+log/2048c078d45e..4f42243e85c8

$ git log 2048c078d..4f42243e8 --date=short --no-merges --format='%ad %ae %s'
2018-05-03 simonhatch Dashboard - Use new isolate target for linux-perf.

Created with:
  roll-dep src/third_party/catapult
BUG= chromium:836037 


The AutoRoll server is located here: https://catapult-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=sullivan@chromium.org

Change-Id: I812ed7c6ee64248f9595c2590dd7682492042590
Reviewed-on: https://chromium-review.googlesource.com/1043605
Commit-Queue: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Reviewed-by: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#555998}
[modify] https://crrev.com/c4b0f83e9230f7176a68ec7f690634b1b25e7a5c/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/870dc3a4f6b3eab13a98eced9febbc518ba54928

commit 870dc3a4f6b3eab13a98eced9febbc518ba54928
Author: Dave Tu <dtu@chromium.org>
Date: Fri May 04 19:16:29 2018

[pinpoint] Add `--benchmarks` argument for performance_test_suite.

Bug:  chromium:836037 
Change-Id: I6830767995630f330b8b507eed452fc9dad8a002
Reviewed-on: https://chromium-review.googlesource.com/1044659
Reviewed-by: Annie Sullivan <sullivan@chromium.org>
Commit-Queue: David Tu <dtu@chromium.org>

[modify] https://crrev.com/870dc3a4f6b3eab13a98eced9febbc518ba54928/dashboard/dashboard/pinpoint/models/quest/run_telemetry_test_test.py
[modify] https://crrev.com/870dc3a4f6b3eab13a98eced9febbc518ba54928/dashboard/dashboard/pinpoint/models/quest/run_telemetry_test.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d2fd84eb7936727c830e567e6e11eb55d0b090ac

commit d2fd84eb7936727c830e567e6e11eb55d0b090ac
Author: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Fri May 04 21:34:18 2018

Roll src/third_party/catapult/ 53d9b3ced..870dc3a4f (1 commit)

https://chromium.googlesource.com/catapult.git/+log/53d9b3ced389..870dc3a4f6b3

$ git log 53d9b3ced..870dc3a4f --date=short --no-merges --format='%ad %ae %s'
2018-05-04 dtu [pinpoint] Add `--benchmarks` argument for performance_test_suite.

Created with:
  roll-dep src/third_party/catapult
BUG= chromium:836037 


The AutoRoll server is located here: https://catapult-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=sullivan@chromium.org

Change-Id: Ic8df92913ae7d940a624b339e5cf1048ce54c2ae
Reviewed-on: https://chromium-review.googlesource.com/1044778
Reviewed-by: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#556202}
[modify] https://crrev.com/d2fd84eb7936727c830e567e6e11eb55d0b090ac/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/c273e5a94ee5cb6a01f2b9ec5e55b2304c133c77

commit c273e5a94ee5cb6a01f2b9ec5e55b2304c133c77
Author: Simon <simonhatch@chromium.org>
Date: Mon May 07 15:22:07 2018

Dashboard - Use new isolate target for mac too.

Bug:  chromium:836037 
Change-Id: I98f2109d51c3734fce010b57fc5a933a7dbd9711
Reviewed-on: https://chromium-review.googlesource.com/1047195
Reviewed-by: Emily Hanley <eyaich@chromium.org>
Commit-Queue: Simon Hatch <simonhatch@chromium.org>

[modify] https://crrev.com/c273e5a94ee5cb6a01f2b9ec5e55b2304c133c77/dashboard/dashboard/pinpoint_request.py

The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/8ec2eb07d467b8bf34152c6bc5df0ab9c6f63d78

commit 8ec2eb07d467b8bf34152c6bc5df0ab9c6f63d78
Author: Dave Tu <dtu@chromium.org>
Date: Mon May 07 19:47:26 2018

[pinpoint] Look in new location for perf results file.

The new telemetry benchmark harness has the ability to run multiple
benchmarks. So for each one, it creates a separate output directory.
The new location is dependent on the benchmark name, so we need to
plumb that through.

Bug:  chromium:836037 
Change-Id: Ib6ce9ee03d53df7733f582f4b967021927c4377d
Reviewed-on: https://chromium-review.googlesource.com/1044951
Reviewed-by: Annie Sullivan <sullivan@chromium.org>
Reviewed-by: Simon Hatch <simonhatch@chromium.org>
Commit-Queue: David Tu <dtu@chromium.org>

[modify] https://crrev.com/8ec2eb07d467b8bf34152c6bc5df0ab9c6f63d78/dashboard/dashboard/pinpoint/models/quest/read_value_test.py
[modify] https://crrev.com/8ec2eb07d467b8bf34152c6bc5df0ab9c6f63d78/dashboard/dashboard/pinpoint/models/quest/read_value.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/68e06f112043d333ae0e8cfec895c7f5bee33a5f

commit 68e06f112043d333ae0e8cfec895c7f5bee33a5f
Author: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Mon May 07 20:25:25 2018

Roll src/third_party/catapult/ b9ff578f3..e54b6ffb6 (3 commits)

https://chromium.googlesource.com/catapult.git/+log/b9ff578f32e7..e54b6ffb6dad

$ git log b9ff578f3..e54b6ffb6 --date=short --no-merges --format='%ad %ae %s'
2018-05-07 bpastene devil: Update expected verity output.
2018-05-07 simonhatch Dashboard - Remove reference to bisect stats in nav_bar
2018-05-07 simonhatch Dashboard - Use new isolate target for mac too.

Created with:
  roll-dep src/third_party/catapult
BUG=chromium:839741, chromium:836037 


The AutoRoll server is located here: https://catapult-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=sullivan@chromium.org

Change-Id: I712bd8bef76f3f01ca6d262b72e51a989db254a1
Reviewed-on: https://chromium-review.googlesource.com/1048107
Reviewed-by: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#556552}
[modify] https://crrev.com/68e06f112043d333ae0e8cfec895c7f5bee33a5f/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a219f660877f87796879022f119d9ff84b12fdaa

commit a219f660877f87796879022f119d9ff84b12fdaa
Author: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Mon May 07 23:13:52 2018

Roll src/third_party/catapult/ e54b6ffb6..e85353176 (2 commits)

https://chromium.googlesource.com/catapult.git/+log/e54b6ffb6dad..e853531767d2

$ git log e54b6ffb6..e85353176 --date=short --no-merges --format='%ad %ae %s'
2018-05-07 nednguyen Remove deadcode related to old test disabling system in Telemetry story runner
2018-05-04 dtu [pinpoint] Look in new location for perf results file.

Created with:
  roll-dep src/third_party/catapult
BUG= chromium:796340 , chromium:836037 


The AutoRoll server is located here: https://catapult-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=sullivan@chromium.org

Change-Id: Id4ca3154cc36c121ee3a7b9347df5053decf30f3
Reviewed-on: https://chromium-review.googlesource.com/1048307
Commit-Queue: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Reviewed-by: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#556604}
[modify] https://crrev.com/a219f660877f87796879022f119d9ff84b12fdaa/DEPS

Fixed in dashboard in Pinpoint. Keep us updated on which configs are moving over so we can update the configs in dashboard.

There were two fixes on the Chromium side.
https://chromium-review.googlesource.com/c/chromium/src/+/1030773
https://chromium-review.googlesource.com/c/chromium/src/+/1057350

Any jobs that run on a Chromium range before those fixes will fail.
With the new "run job with patch" feature, we can re-run those jobs with a fix in, but we can't apply multiple patches (issue 820653), so we still can't run any jobs on ranges before r554597