Strange, offhand I don't know why it didn't fail. Someone will likely need to dig into this.

Is it supposed to de-apply the patch when it runs?

it's a trybot, so I'd assume so?

Well, yeah that's true. I guess i'm wondering if ned expected to happen in that build.

#2: Is that for detecting flakes? I think that's fine.

Re #5: Yes it is.

I saw this in the swarming stdout log:

Running ['/usr/bin/python', '/b/c/b/obbs_fyi/src/tools/perf/process_perf_results.py', '--build-properties', '{"attempt_start_ts": 1532457561000000, "blamelist": ["nednguyen@google.com"], "bot_id": "slave1584-c4", "branch": "", "build_data_dir": "/b/rr/tmp_Of51H", "build_id": "buildbot/tryserver.chromium.perf/obbs_fyi/409", "buildbotURL": "http://build.chromium.org/p/tryserver.chromium.perf/", "buildbucket": "{\\"build\\": {\\"bucket\\": \\"master.tryserver.chromium.perf\\", \\"created_by\\": \\"user:5071639625-1lppvbtck1morgivc6sq4dul7klu27sd@developer.gserviceaccount.com\\", \\"created_ts\\": \\"1532457600147810\\", \\"id\\": \\"8940095455426158080\\", \\"lease_key\\": \\"1861137390\\", \\"tags\\": [\\"builder:obbs_fyi\\", \\"buildset:patch/gerrit/chromium-review.googlesource.com/1148228/6\\", \\"cq_experimental:false\\", \\"master:master.tryserver.chromium.perf\\", \\"user_agent:cq\\"]}}", "builder_id": "master.tryserver.chromium.perf:obbs_fyi", "buildername": "obbs_fyi", "buildnumber": 409, "category": "cq", "got_angle_revision": "06235df9ff91acdf3cf50f2a6febb38b5e26cc6b", "got_buildtools_revision": "0dd5c6f980d22be96b728155249df2da355989d9", "got_nacl_revision": "aebc9e5205dcaad5d5954a18abc532316a954644", "got_revision": "6735d97d371148402ec51cfe0a1cb54b1809a59c", "got_revision_cp": "refs/heads/master@{#577621}", "got_swarming_client_revision": "486c9b53c4d54dd4b95bb6ce0e31160e600dfc11", "got_v8_revision": "696e3951665ffc632c6a154a1d2b14c3b3e92212", "got_v8_revision_cp": "refs/heads/7.0.21@{#1}", "got_webrtc_revision": "e09e6ef947d7bb6c29fa36c331bd855baacfafbc", "got_webrtc_revision_cp": "refs/heads/master@{#24084}", "luci_migration": {"status": "ok"}, "mastername": "tryserver.chromium.perf", "patch_gerrit_url": "https://chromium-review.googlesource.com", "patch_issue": 1148228, "patch_project": "chromium/src", "patch_ref": "refs/changes/28/1148228/6", "patch_repository_url": "https://chromium.googlesource.com/chromium/src", "patch_set": 6, "patch_storage": "gerrit", "path_config": "kitchen", "project": "", "reason": "CQ", "recipe": "chromium_trybot", "recipe_repository": "https://chromium.googlesource.com/chromium/tools/build.git", "repository": "https://chromium.googlesource.com/chromium/src", "requestedAt": 1532457612, "requester": "commit-bot@chromium.org", "revision": "", "slavename": "slave1584-c4", "workdir": "/b/build/slave/obbs_fyi"}', '--summary-json', '/tmp/tmpGXo0zQ/summary.json', '--task-output-dir', '/tmp/tmpGXo0zQ', u'--service-account-file', u'/creds/service_accounts/service-account-chromium-perf-histograms.json', '-o', '/tmp/tmpP08aGz.json'] in None (env: None)
Duration of Generating benchmark shard map stream: 0 seconds
Duration of Generating perf log streams: 0 seconds
Benchmark system_health.memory_mobile.reference disabled
Benchmark system_health.memory_mobile disabled
Benchmark memory.top_10_mobile disabled
Benchmark memory.top_10_mobile.reference disabled
Duration of Analyzing perf json test results: 0 seconds
Uploading perf results from dummy_benchmark.stable_benchmark_1.reference benchmark
Uploading perf results from rasterize_and_record_micro.partial_invalidation benchmark
Uploading perf results from dummy_benchmark.histogram_benchmark_1 benchmark
Uploading perf results from rasterize_and_record_micro.partial_invalidation.reference benchmark
Uploading perf results from dummy_benchmark.histogram_benchmark_1.reference benchmark
invalid value "30" for flag -lifetime: time: missing unit in duration 3
Uploading perf results from blink_perf.owp_storage.reference benchmark
Generates an access token if requested and prints it.
usage:  luci-auth token
  -json-output string
    	Destination file to print token and expiration time in JSON. "-" for standard output.
  -lifetime duration
    	Minimum token lifetime. If existing token expired and refresh token or service account is not present, returns nothing. (default 1m0s)
  -scopes string
    	space-separated OAuth 2.0 scopes (default "https://www.googleapis.com/auth/userinfo.email")
  -service-account-json string
    	Path to JSON file with service account credentials to use.
Duration of blink_perf.paint.reference results merging: 0 seconds
Uploading perf results from blink_perf.paint.reference benchmark
Duration of blink_perf.owp_storage results merging: 0 seconds
invalid value "30" for flag -lifetime: time: missing unit in duration 30
Uploading perf results from blink_perf.owp_storage benchmark
Generates an access token if requested and prints it.
usage:  luci-auth token
  -json-output string
    	Destination file to print token and expiration time in JSON. "-" for standard output.
  -lifetime duration
    	Minimum token lifetime. If existing token expired and refresh token or service account is not present, returns nothing. (default 1m0s)
  -scopes string
    	space-separated OAuth 2.0 scopes (default "https://www.googleapis.com/auth/userinfo.email")
  -service-account-json string
    	Path to JSON file with service account credentials to use.
ERROR:root:Error uploading perf result of dummy_benchmark.stable_benchmark_1.reference
Traceback (most recent call last):
  File "/b/c/b/obbs_fyi/src/tools/perf/process_perf_results.py", line 373, in _upload_individual_benchmark
    return _upload_individual(*params)
  File "/b/c/b/obbs_fyi/src/tools/perf/process_perf_results.py", line 357, in _upload_individual
    token_expiration_in_minutes=30) as oauth_file:
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/b/c/b/obbs_fyi/src/tools/perf/core/oauth_api.py", line 31, in with_access_token
    subprocess.check_call(args, stdout=fd)
  File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['luci-auth', 'token', '-service-account-json', '/creds/service_accounts/service-account-chromium-perf-histograms.json', '-lifetime', '30']' returned non-zero exit status 2
invalid value "30" for flag -lifetime: time: missing unit in duration 30

It looks like the merge script is calling luci-auth incorrectly somehow. A quick code search showed that the merge script failing doesn't fail the collection? https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/swarming/resources/collect_task.py?q=collect_task.py&sq=package:chromium&g=0&l=165

It could be that something is marking the step as failed later, but it's not ever raising an exception?