All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=848551

(For debugging:) Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?sid=e12823e736504c48330606c8f3c073c5f447e79353fba8c5b476b980e8cf1db6


Bot(s) for this bug's original alert(s):

mojo-linux-perf

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8944960978620372688

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8944960937681560640

The job above returns the following error:
```
DATA:{
    "bucket": "master.tryserver.chromium.perf", 
    "canary_preference": "AUTO", 
    "completed_utc": "Fri Jun  1 01:45:21 2018", 
    "created_by": "user:425761728072-pa1bs18esuhp2cp2qfa1u9vb6p1v6kfu@developer.gserviceaccount.com", 
    "created_utc": "Fri Jun  1 01:45:14 2018", 
    "experimental": false, 
    "failure_reason": "INVALID_BUILD_DEFINITION", 
    "id": "8944960937681560640", 
    "parameters": {
        "builder_name": "linux_perf_bisect", 
        "properties": {
            "bisect_config": {
                "bad_revision": "560196", 
                "bug_id": 848551, 
                "builder_host": null, 
                "builder_port": null, 
                "command": "src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=Flickr.diff loading.desktop.network_service", 
                "good_revision": "560155", 
                "gs_bucket": "chrome-perf", 
                "max_time_minutes": "20", 
                "metric": "timeToFirstContentfulPaint:style_avg_diff/warm/Flickr_diff", 
                "recipe_tester_name": "linux_perf_bisect", 
                "repeat_count": "20", 
                "test_type": "perf", 
                "try_job_id": 5814493810786304
            }
        }
    }, 
    "project": "chromium", 
    "result": "FAILURE", 
    "result_details": {
        "error": {
            "message": "Builder linux_perf_bisect not found"
        }
    }, 
    "status": "COMPLETED", 
    "status_changed_utc": "Fri Jun  1 01:45:21 2018", 
    "tags": [
        "builder:linux_perf_bisect"
    ], 
    "updated_utc": "Fri Jun  1 01:45:21 2018", 
    "utcnow_utc": "Fri Jun  1 20:13:30 2018"
} 
```

Looks like they cannot find the bisect bot (linux_perf_bisect)...

+nednguyen@ Is there any special configurations I need to do to make bisect work?

Thanks!

Currently, there is no bisect support for fyi waterfall

Ah I see, thanks for the information!
Do we know how hard it is to add the support? Or are they actually using different infrastructures (other than different priorities)?

Thanks!

It's mostly different priorities. The infrastrucutre is essentially the same

To add support, we need to:
* Ensure that the perf builders build that isolate
* Ensure that there are bisector bots matching that FYI bot config, and add some if necessary
* Update some perf dashboard and Pinpoint configs

Re #c9: Thanks for the information! Since I'm not so familiar with the builder config here - Do you have a rough estimate on how many SWE hours required to add the support?

Or if it's a burden for your team, can you point me to related docs or patches so I can take a look and see if I can fix it?

Thanks!

II checked, and we should already have that benchmark available on the perf builders. We also have those Linux machines available already, so we just need to update some configs.

Uploaded a CL to add the configs.
https://chromium-review.googlesource.com/c/catapult/+/1091807

Nice, really appreciated!

The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/c9718952e5d02e7003820e47503e9f9335f4a8c4

commit c9718952e5d02e7003820e47503e9f9335f4a8c4
Author: Dave Tu <dtu@chromium.org>
Date: Thu Jun 07 23:41:08 2018

[dashboard] Add mojo-linux-perf to Pinpoint.

Bug:  chromium:848551 
Change-Id: Id5b975e5505ae5c2784add30a86bf44e9c100ef8
Reviewed-on: https://chromium-review.googlesource.com/1091807
Reviewed-by: Simon Hatch <simonhatch@chromium.org>
Commit-Queue: David Tu <dtu@chromium.org>

[modify] https://crrev.com/c9718952e5d02e7003820e47503e9f9335f4a8c4/dashboard/dashboard/pinpoint/elements/jobs-page/new-job-fab.html
[modify] https://crrev.com/c9718952e5d02e7003820e47503e9f9335f4a8c4/dashboard/dashboard/pinpoint_request.py
[modify] https://crrev.com/c9718952e5d02e7003820e47503e9f9335f4a8c4/dashboard/dashboard/static/pinpoint.html

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a3f039c765a3f71a1341e553518065394179d0ba

commit a3f039c765a3f71a1341e553518065394179d0ba
Author: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Fri Jun 08 04:56:57 2018

Roll src/third_party/catapult 09b82f4..c971895 (1 commits)

https://chromium.googlesource.com/catapult.git/+log/09b82f4..c971895


git log 09b82f4..c971895 --date=short --no-merges --format='%ad %ae %s'
2018-06-07 dtu@chromium.org [dashboard] Add mojo-linux-perf to Pinpoint.


Created with:
  gclient setdep -r src/third_party/catapult@c971895

The AutoRoll server is located here: https://catapult-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG= chromium:848551 
TBR=sullivan@chromium.org

Change-Id: Iaf08d38a02eb816bcc880cb602ed2592804fdc2d
Reviewed-on: https://chromium-review.googlesource.com/1092250
Reviewed-by: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#565543}
[modify] https://crrev.com/a3f039c765a3f71a1341e553518065394179d0ba/DEPS

Configs should be up! Let me know if anything is not working.

Awesome! Will give it a try and get back to you if I still have questions. Thanks!

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/12cb55c5240000

😿 Pinpoint job stopped with an error.
https://pinpoint-dot-chromeperf.appspot.com/job/12cb55c5240000

All of the runs failed. The most common error (5/20 runs) was:
  Run at /b/s/w/ir/third_party/catapult/telemetry/telemetry/benchmark_runner.pyTraceback (most recent call last):

Please ignore the failure above - The fix was landed earlier today, will give it some time to propagate.

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/13ba8825240000

The job failed because it's trying to use `histograms` as the output format, which is not supported by loading.desktop.network_service:
```From #c21
2621 2018-06-15 00:22:42.664 E: local_auth: the returned access token is invalid
Additional test environment:
    CHROME_DEVEL_SANDBOX=/opt/chromium/chrome_sandbox
    CHROME_HEADLESS=1
    LANG=en_US.UTF-8
Command: /usr/bin/python ../../testing/scripts/run_performance_tests.py ../../tools/perf/run_benchmark --benchmarks loading.desktop.network_service --story-filter Flickr.diff --pageset-repeat 1 --browser release -v --upload-results --output-format histograms --isolated-script-test-output /b/s/w/ioJNlyvN/output.json --isolated-script-test-chartjson-output /b/s/w/ioJNlyvN/chartjson-output.json --results-label chromium@aa324eb
```

However I did changed the output format back to `chartjson` for the benchmark on Jun 11 (bb7c4560ca39360eeb681248378bb7099ff28a00), or do we have a separate config for bisect?

Thanks!

😿 Pinpoint job stopped with an error.
https://pinpoint-dot-chromeperf.appspot.com/job/13ba8825240000

The swarming task expired. The bots are likely overloaded, dead, or misconfigured.

Ah sorry, Pinpoint has already dropped support for chartjson. Pinpoint contains its own logic for reading and interpreting the benchmark results, and supports histograms only.

Ah I see, guess I will have to run pinpoint manually then... Anyway thanks for the help!

There's no manual pinpoint, it's a service that runs on appengine.

Re #c26: Sorry for the confusion. I was thinking I might be able to run pinpoint try jobs with Rev# vs. Rev# + patch, but just find out that we only support HEAD vs. HEAD + patch.