(Woops - forgot to add the attachments.)

Deleted: collection_time.png 39.7 KB

	collection_time.png 39.7 KB View Download

Deleted: duration.png 39.9 KB

	duration.png 39.9 KB View Download

This isn't really a Milo issue because the behavior is identical on buildbot.  Milo renders data directly from buildbot, and buildbot is seeing the same thing.

I'll explain what's going on.  Swarming works asynchronously to the build by first triggering a bunch of tasks, and then collecting the result one by one afterwards since recipes are single threaded.  The order that the results are collected doesn't change, so a job waiting to finish will block grabbing results further down the list.  This also means that if the first item takes a long time, but everything else has finished, the "step" to collect log results will look like they finish immediately regardless of how long the step actually take.

What we don't want to do is to "fake" the swarming log collection step, because (for example) what if the log collection actually takes a long time, we want to surface that.  But there are some solutions we can use, the simplest of which would be to add the swarming timing information into the step text, so I'm rerouting this bug to swarming.

Sounds good to me. Thanks hinoka@!

The NextAction date has arrived: 2017-08-25

iannucci@, do you have any idea who might be an appropriate swarming person to redirect this to? Basically, we just want swarming to show metadata next to each task in the Milo output page showing how long the swarming task took to run, similar to what we currently show for pending.

Doh - didn't CC iannucci@. Robbie, please see comment 6 :-)

It's a recipe change, anyone can do it.
https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/swarming/api.py?l=665

The problem isn't that the information isn't present on the page, it's that the information already present on the page looks like it's telling us swarming task duration, when it's not. 

We could add something to the text of the step, but most people would ignore that and read the duration to the right of the step title, which is wrong and misleading. Ideally we'd change that text to be what time swarming actually took.

I think this is a symptom of a slightly bigger problem, which is that the way we trigger/rely on swarming tasks from recipes is pretty far from ideal, coupled with the very limited control that recipes have over the presentation in Milo.

If recipes had full control over the presentation (i.e. they directly emit the annotation protobuf), we could make a much better UI for this by building swarming support directly into the recipe engine (an integration I think would actually be fine; recipes are a client of LUCI services, having direct support for those services is a good thing).

The direct support would probably be low-ish level, but present the swarming tasks in the UI correctly (the recipe engine could even manage multiple in-progress tasks in parallel, so they all appear yellow on trigger, and turn green as they actually finish, complete with their correct timing information, etc.). We'd still want to have a chromium-specific swarming module which consolidates all the peculiarities of how chromium uses swarming though (we wouldn't want those in the recipe engine itself).

I still think having it as text to the step, then a sum at the very bottom would still be useful right now and doesn't require refactoring the world.

I agree we should think about the larger scope but this smaller change has immediate benefit and isn't complex to do AFAIK.

Yes, adding some additional text would be fine (I have no objection), but I suspect that what martiniss said is also correct: no one will see it, and this bug will be filed repeatedly :)

I agree with comment 11 - while I also agree with iannucci@ and martiniss@ that another presentation would be optimal, my primary problem right now is that the information just isn't readily available at all.

iannucci@, is there any chance that you'd be willing to take on the recipe change above that hinoka@ described as easy? I suspect that in order to make it, I'd have to get acquainted with a whole bunch of code and the recipe infrastructure that I'm not currently.

So I just looked into it a bit, and this module seems to be made out of quite a bit of sphagetti. The 'pending time' text is only added when the collection step was for gtest results (apparently? why does the swarming module know about that?), and the actual collection step is overridable.

I'm not exactly sure what the feature request is... Which time do you want to see, and where do you want to see it?

Basically, on a line above the "swarming pending" # of seconds, it'd be great to have a "swarming duration" time. (Separately, it seems like both of these should be in hh:mm:ss format like on the swarming page, rather than just seconds, which is pretty hard to read for some of the perf-length tasks.)

The duration of what though? Each shard? The combination of shards? The maximum length shard?

Or should this be special cased only for single-shard collections?

Actually I'm not sure what the pending time operates on, either, since each shard has different pending times (all of this code is new to me :)).

I'm not sure: probably doing something that matches what we do for the pending time makes sense. It's hard for me to make a reasonable call for these situations, though: I can only suggest what would be very helpful in the single-shard perf cases I'm familiar with.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc

commit 1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc
Author: Robert Iannucci <iannucci@chromium.org>
Date: Wed Aug 30 17:41:35 2017

Add better stats on gtest swarming collection tasks.

This will indicate that the pending time is the time for the
longest-pending shard, and also indicates the maximum shard time of all
the shards that were collected.

If there was more than one shard, each time measurement says which shard
it belongs to, as well.

R=charliea@chromium.org, jbudorick@chromium.org, vadimsh@chromium.org

Bug: 758339
Change-Id: Ifbf4f1b30ac7825a9bdeba317946582b1b1b391f
Reviewed-on: https://chromium-review.googlesource.com/642491
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>

[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/chromium.expected/dynamic_swarmed_gtest_mac_gpu.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_win32_release.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/findit/chromium/test.expected/flaky_tests.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/findit/chromium/test.expected/remove_culprits_for_flaky_failures.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/findit/chromium/test.expected/all_test_failed.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_linux_rel_ng_triggered_triggered_by_cq.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_mac64_debug.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/tryserver_webrtc_win_experimental.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/ios/examples/full.expected/errors.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/swarming/examples/full.expected/show_shards_in_collect_step.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/chromium.expected/isolated_script_test_custom_merge_script_with_args.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/tryserver_webrtc_linux_experimental.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/tryserver_webrtc_linux_tsan2.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/swarming/examples/full.expected/rietveld_trybot.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/tryserver_webrtc_android_dbg.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_win32_debug.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/chromium_tests/tests/steps/swarming_gtest_test.expected/basic.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/swarming/examples/full.expected/basic.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_win64_release.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_passed_isolated_script_perf_test_disabled.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/dart/vm_kernel_swarming.expected/vm-kernel-linux-debug-x64-try.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/ios/try.expected/basic.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/chromium.expected/isolated_script_test_custom_merge_script.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_android32__m_nexus5x_.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/chromium.expected/dynamic_swarmed_passed_with_bad_retcode_isolated_script_test.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/swarming/examples/full.expected/show_isolated_out_in_collect_step.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/dart/dart2js_swarming.expected/dart2js-win7-ie10-debug-try.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/tryserver_webrtc_linux_asan.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/ios/unified_builder_tester.expected/basic.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/chromium_trybot.expected/dynamic_swarmed_isolated_script_test_failure_no_result_json.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/findit/chromium/test.expected/findit_culprit_in_first_sub_range.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_fyi_win64_debug__win8_.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/chromium_trybot.expected/swarmed_webkit_tests_interrupted.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/chromium_tests/tests/api/run_tests_on_tryserver.expected/basic.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_passed_isolated_script_perf_test.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/swarming/examples/full.expected/isolated_script_expired_new.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_passed_isolated_script_perf_test_empty.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/dart/vm_kernel_swarming.expected/vm-kernel-mac-release-x64-be.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_win64_debug.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/tryserver_webrtc_linux_ubsan_vptr.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/dart/dart2js_swarming.expected/dart2js-linux-d8-hostchecked-csp-unittest-try.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/v8.expected/full_client_v8_V8_Random_Deopt_Fuzzer___debug.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/chromium_trybot.expected/swarming_test_failure.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/findit/chromium/test.expected/use_analyze_set_to_False_for_non_linear_try_job.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/chromium_tests/tests/api/run_tests_on_tryserver.expected/disable_deapply_patch_recipes.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/v8.expected/full_client_v8_V8_Fuzzer.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/findit/chromium/test.expected/swarming_tests.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_fyi_android_asan__swarming_.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/swarming/examples/full.expected/swarming_timeout_new.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/swarming/staging.expected/one_fails.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_isolated_chartjson_test_missing_all_shards.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/tryserver_webrtc_win_x64_win8.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/chromium.expected/dynamic_swarmed_isolated_script_perf_test_ignore_task_failure.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/chromium_tests/tests/steps/swarming_isolated_script_test.expected/basic.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/chromium_trybot.expected/swarmed_webkit_tests_minimal_pass_continues.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_linux64_rel_ng_triggered.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_linux_memcheck.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_linux_rel_ng_triggered_failures.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/tryserver_webrtc_mac_rel.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/findit/chromium/test.expected/use_build_parameter_for_tests_non_json_buildbucket.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/chromium_tests/tests/steps/swarming_isolated_script_test.expected/dimensions_windows.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_linux_rel_ng_triggered.json
[add] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/swarming/examples/full.expected/gtest_with_long_task.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/tryserver_webrtc_win_asan.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/chromium_trybot.expected/add_swarming_layout_tests_via_manual_diff_inspection.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/tryserver_webrtc_win_x64_dbg.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/v8.expected/full_client_v8_V8_Deopt_Fuzzer.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/swarming/examples/full.expected/swarming_expired_old.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/chromium.expected/dynamic_swarmed_failed_isolated_script_test.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/chromium_tests/tests/steps/swarming_gtest_test.expected/android.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_fyi_win64_debug__win10_.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/tryserver_webrtc_android_arm64_rel.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/tryserver_webrtc_linux_dbg_gerrit.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/v8.expected/full_client_v8_V8_Linux.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_win32_asan.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/v8.expected/full_client_v8_V8_Linux64.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/findit/chromium/test.expected/use_abbreviated_revision_in_step_name.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/tryserver_webrtc_linux_ubsan.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/v8.expected/full_client_v8_V8_Linux64_one_failure_build_env_not_supported.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_android64__m_nexus5x__dbg_.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_linux_tsan_v2.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/findit/chromium/test.expected/use_build_parameter_for_tests.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/tryserver_webrtc_mac_dbg.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/tryserver_webrtc_linux_dbg.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/findit/chromium/test.expected/unaffected_test_skipped_by_analyze.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/chromium_trybot.expected/dynamic_isolated_script_test_with_args_on_trybot.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/chromium_tests/tests/steps/swarming_isolated_script_test.expected/dimensions_mac.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/tryserver_webrtc_win_x64_rel.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/findit/chromium/test.expected/all_test_passed.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_linux64_debug_forced.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/swarming/examples/full.expected/isolated_script_expired_old.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/README.recipes.md
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/swarming/examples/full.expected/isolated_script_with_null_shard.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_linux_msan.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/chromium_tests/tests/steps/swarming_isolated_script_test.expected/chartjson_max_failures.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_linux_asan.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/chromium_tests/tests/steps/generate_gtest.expected/gsutil_upload_swarming.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/chromium.expected/isolated_script_test_custom_results_handler.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_linux_rel_ng_triggered_slow_tests.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_linux_rel_ng_triggered_triggered_by_ts.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/swarming/examples/full.expected/swarming_timeout_old.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/chromium_tests/tests/api/run_tests_on_tryserver.expected/nonzero_exit_code_no_gtest_output.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/chromium_trybot.expected/dynamic_isolated_script_test_on_trybot_passing.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_isolated_chartjson_test_missing_shard.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_corrupt_json_isolated_script_test.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/tryserver_webrtc_win_x64_win10.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/chromium_tests/tests/steps/swarming_isolated_script_test.expected/override_compile_targets.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/chromium.expected/dynamic_swarmed_isolated_script_test_mac_gpu.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_linux_rel_ng_triggered_flakes.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/findit/chromium/test.expected/findit_consecutive_culprits.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_linux_ubsan.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/swarming/examples/full.expected/isolated_script_timeout_old.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/tryserver_webrtc_mac_asan.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/findit/chromium/test.expected/findit_culprit_in_last_sub_range.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/ios/examples/full.expected/basic.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_android64__m_nexus5x_.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/tryserver_webrtc_android_rel.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/ios/try.expected/gn.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/swarming/examples/full.py
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/swarming/examples/full.expected/gtest_with_outputs_ref.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/chromium_trybot.expected/dynamic_isolated_script_test_on_trybot_failing.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/v8.expected/full_client_v8_V8_Linux64___debug.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_isolated_chartjson_test_harness_failure.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/swarming/staging.expected/windows.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_linux_ubsan_vptr.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/chromium.expected/gtest_custom_merge_script.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_passed_isolated_script_test.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/chromium_trybot.expected/swarmed_webkit_tests_compile_without_patch_fails.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/tryserver_webrtc_win_dbg.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/chromium.expected/dynamic_swarmed_isolated_script_test_linux_gpu.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/tryserver_webrtc_linux32_rel.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/chromium_trybot.expected/swarmed_layout_tests_too_many_failures_for_retcode.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_failed_isolated_script_test.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/chromium_tests/tests/api/run_tests_on_tryserver.expected/disable_deapply_patch_affected_files.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/findit/chromium/test.expected/findit_tests_multiple_culprits.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/findit/chromium/flake.expected/flakiness_swarming_tests.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/chromium_tests/tests/steps/swarming_isolated_script_test.expected/dimensions_mac_hidpi.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_fyi_webrtc_perf_tests__linux_trusty_swarming_.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/tryserver_webrtc_linux_rel.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_linux64_release.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/tryserver_webrtc_linux32_dbg.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/chromium_trybot.expected/add_swarming_layout_tests_via_manual_diff_inspection_that_fails.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/chromium_tests/tests/steps/swarming_isolated_script_test.expected/chartjson_ignore_task_failure.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_linux_rel_ng_triggered_test_filter.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_linux64_debug_failing_test.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/chromium_tests/tests/steps/swarming_isolated_script_test.expected/chartjson_invalid.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/webrtc/standalone.expected/tryserver_webrtc_win_rel.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/dart/examples/example.expected/basic.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipes/findit/chromium/flake.expected/use_build_parameter_for_tests.json
[modify] https://crrev.com/1d5d72caca7f1e6ee62b8b5be300b2e76f1029fc/scripts/slave/recipe_modules/chromium_tests/tests/steps/swarming_isolated_script_test.expected/chartjson_no_results.json
[modify] https://crrev.com/

Is there any chance that we could get units after each of the fields? With different swarming tasks taking different orders of magnitude of time, I feel it might make it more readable and less error-prone when analyzing.

e.g. 

Rather than

00:24:33.2342

0h 24m 33.2342s

Deleted: units.png 169 KB

	units.png 169 KB View Download

Rounding to the second would help readability. Then it's less a big deal IMHO.