Or simpler @@@CURRENT_TIMESTAMP@@@<timestamp>@@@. Timestamp implies UTC.

Known issue AFAIK. hinoka can say more

Any objections to introduce new @@@CURRENT_TIMESTAMP@@@ annotation and emit it in recipe engine? It will also require modifying AnnotationObserver in buildbot so it ignores the annotation without complaining.

It'll also require modifying annotee to recognize the annotation and act accordingly.  It might be better to just skip over to integrating logdog output into recipes maybe?

Currently logdog available is p3. Blocking on logdog would make it p1 and put pressure on dnj to complete logdog as a service and its integration with recipes/kitchen and milo. I don't know how much time will it take.

I am less hesitant to introduce less systems in one release. So far next release (get public infra off of buildbot) requires first prod use of swarmbucket, milo-with-swarmbucket, and swarming-with-cipd, recipes-on-swarming and luci-cron-with-swarmbucket (note that luci-cron is not ready yet ATM).

RE #3: We've talked about a timestamp annotation for Milo. Modifying Annotee to parse that is trivial, so I wouldn't consider that a blocker for the idea.

I started working on this. STEP_DURATION would be more precise and simpler than generic STEP_TIMESTAMP

FTR STEP_DURATION won't work because we need time of step start and end
CLs:
https://codereview.chromium.org/2050703003/
https://codereview.chromium.org/2052543003/

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools.git/+/d139d55fee54417a67dfd7f14b2b266ceb3ce4a9

commit d139d55fee54417a67dfd7f14b2b266ceb3ce4a9
Author: recipe-roller <recipe-roller@chromium.org>
Date: Fri Jun 10 01:55:32 2016

Roll recipe dependencies (trivial).

This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug
(or complain)

recipe_engine:
  https://crrev.com/02bd5642786bd00b6060404bf8c459e6cc146c4c Emit CURRENT_TIMESTAMP annotation (nodir@chromium.org)

R=iannucci@chromium.org,nodir@chromium.org,martiniss@chromium.org
BUG= 613584 

TBR=martiniss@chromium.org,phajdan.jr@chromium.org

Review-Url: https://codereview.chromium.org/2056113002

[modify] https://crrev.com/d139d55fee54417a67dfd7f14b2b266ceb3ce4a9/infra/config/recipes.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/5dd960d6a237067b0f36abe22c36f928a2d7d02c

commit 5dd960d6a237067b0f36abe22c36f928a2d7d02c
Author: recipe-roller <recipe-roller@chromium.org>
Date: Fri Jun 10 02:06:30 2016

Roll recipe dependencies (trivial).

This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug
(or complain)

depot_tools:
  https://crrev.com/d139d55fee54417a67dfd7f14b2b266ceb3ce4a9 Roll recipe dependencies (trivial). (recipe-roller@chromium.org)
recipe_engine:
  https://crrev.com/02bd5642786bd00b6060404bf8c459e6cc146c4c Emit CURRENT_TIMESTAMP annotation (nodir@chromium.org)

R=phajdan.jr@chromium.org,iannucci@chromium.org,nodir@chromium.org,martiniss@chromium.org,recipe-roller@chromium.org
BUG= 613584 

TBR=martiniss@chromium.org,phajdan.jr@chromium.org

Review-Url: https://codereview.chromium.org/2056923003

[modify] https://crrev.com/5dd960d6a237067b0f36abe22c36f928a2d7d02c/infra/config/recipes.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools.git/+/97e8e55968769e9e455ee8f28abe1a036b32837b

commit 97e8e55968769e9e455ee8f28abe1a036b32837b
Author: recipe-roller <recipe-roller@chromium.org>
Date: Fri Jun 10 02:15:27 2016

Roll recipe dependencies (trivial).

This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug
(or complain)

recipe_engine:
  https://crrev.com/da26a351b7034d37e55acd468fa90c8ee3a1ada0 Fix --timestamps help (nodir@chromium.org)

R=nodir@chromium.org,martiniss@chromium.org
BUG= 613584 

TBR=martiniss@chromium.org,phajdan.jr@chromium.org

Review-Url: https://codereview.chromium.org/2059593002

[modify] https://crrev.com/97e8e55968769e9e455ee8f28abe1a036b32837b/infra/config/recipes.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/bed9cca76ea5e0670a300ef170682a7ceed4334d

commit bed9cca76ea5e0670a300ef170682a7ceed4334d
Author: recipe-roller <recipe-roller@chromium.org>
Date: Fri Jun 10 02:26:20 2016

Roll recipe dependencies (trivial).

This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug
(or complain)

depot_tools:
  https://crrev.com/97e8e55968769e9e455ee8f28abe1a036b32837b Roll recipe dependencies (trivial). (recipe-roller@chromium.org)
recipe_engine:
  https://crrev.com/da26a351b7034d37e55acd468fa90c8ee3a1ada0 Fix --timestamps help (nodir@chromium.org)

R=phajdan.jr@chromium.org,nodir@chromium.org,martiniss@chromium.org,recipe-roller@chromium.org
BUG= 613584 

TBR=martiniss@chromium.org,phajdan.jr@chromium.org

Review-Url: https://codereview.chromium.org/2051363002

[modify] https://crrev.com/bed9cca76ea5e0670a300ef170682a7ceed4334d/infra/config/recipes.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra.git/+/72cddc434d92b2a0e8a06982eb3c5b223f7f478f

commit 72cddc434d92b2a0e8a06982eb3c5b223f7f478f
Author: recipe-roller <recipe-roller@chromium.org>
Date: Fri Jun 10 03:21:44 2016

Roll recipe dependencies (trivial).

This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug
(or complain)

build:
  https://crrev.com/38191f487dee3f1dd1d4733684686141cdbfcd1f Roll recipe dependencies (trivial). (recipe-roller@chromium.org)
  https://crrev.com/5dd960d6a237067b0f36abe22c36f928a2d7d02c Roll recipe dependencies (trivial). (recipe-roller@chromium.org)
depot_tools:
  https://crrev.com/7aafc3f4adca9d61017a95b9a8e53e7c4ac4ce4a git cl upload: commit subject is default patchset title (nodir@chromium.org)
  https://crrev.com/d139d55fee54417a67dfd7f14b2b266ceb3ce4a9 Roll recipe dependencies (trivial). (recipe-roller@chromium.org)
recipe_engine:
  https://crrev.com/02bd5642786bd00b6060404bf8c459e6cc146c4c Emit CURRENT_TIMESTAMP annotation (nodir@chromium.org)

R=vadimsh@chromium.org,iannucci@chromium.org,recipe-roller@chromium.org,nodir@chromium.org,martiniss@chromium.org,phajdan.jr@chromium.org,dnj@chromium.org
BUG= 582419 , 613584 

TBR=martiniss@chromium.org,phajdan.jr@chromium.org

Review-Url: https://codereview.chromium.org/2055163002

[modify] https://crrev.com/72cddc434d92b2a0e8a06982eb3c5b223f7f478f/infra/config/recipes.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/16134c57b96067a549a014e8c3ae3addf390285c

commit 16134c57b96067a549a014e8c3ae3addf390285c
Author: nodir <nodir@chromium.org>
Date: Mon Jun 13 16:20:00 2016

kitchen: CURRENT_TIMESTAMP support

Add -timestamps flag: if set, emit CURRENT_TIMESTAMP and pass
--timestamps to recipes.py

Also fix bug in cook: final annotations were not printed in case of
failure.

R=martiniss@chromium.org
BUG= 613584 

Review-Url: https://codereview.chromium.org/2055283002

[modify] https://crrev.com/16134c57b96067a549a014e8c3ae3addf390285c/client/cmd/kitchen/cook.go
[modify] https://crrev.com/16134c57b96067a549a014e8c3ae3addf390285c/client/cmd/kitchen/recipe.go

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/3f5844f8dddb36cd14bd3f050916f0aa7878aec8

commit 3f5844f8dddb36cd14bd3f050916f0aa7878aec8
Author: nodir <nodir@chromium.org>
Date: Thu Jun 16 17:30:47 2016

milo: fix step duration

annottee: support CURRENT_TIMESTAMP, which specifies build start time,
build end time and time of the following annotation.

milo: compute step duration from step start/end if availalbe.

BUG= 613584 

Review-Url: https://codereview.chromium.org/2069283002

[modify] https://crrev.com/3f5844f8dddb36cd14bd3f050916f0aa7878aec8/appengine/cmd/milo/logdog/logDogBuild.go
[modify] https://crrev.com/3f5844f8dddb36cd14bd3f050916f0aa7878aec8/appengine/cmd/milo/swarming/build.go
[modify] https://crrev.com/3f5844f8dddb36cd14bd3f050916f0aa7878aec8/appengine/cmd/milo/swarming/expectations/build-exception.json
[modify] https://crrev.com/3f5844f8dddb36cd14bd3f050916f0aa7878aec8/appengine/cmd/milo/swarming/expectations/build-patch-failure.json
[modify] https://crrev.com/3f5844f8dddb36cd14bd3f050916f0aa7878aec8/appengine/cmd/milo/swarming/expectations/build-timeout.json
[modify] https://crrev.com/3f5844f8dddb36cd14bd3f050916f0aa7878aec8/appengine/cmd/milo/swarming/testdata/build-exception
[modify] https://crrev.com/3f5844f8dddb36cd14bd3f050916f0aa7878aec8/appengine/cmd/milo/swarming/testdata/build-patch-failure
[modify] https://crrev.com/3f5844f8dddb36cd14bd3f050916f0aa7878aec8/appengine/cmd/milo/swarming/testdata/build-timeout
[modify] https://crrev.com/3f5844f8dddb36cd14bd3f050916f0aa7878aec8/client/logdog/annotee/annotation/annotation.go
[modify] https://crrev.com/3f5844f8dddb36cd14bd3f050916f0aa7878aec8/client/logdog/annotee/annotation/annotation_test.go
[add] https://crrev.com/3f5844f8dddb36cd14bd3f050916f0aa7878aec8/client/logdog/annotee/annotation/test_data/timestamps.annotations.txt
[add] https://crrev.com/3f5844f8dddb36cd14bd3f050916f0aa7878aec8/client/logdog/annotee/annotation/test_expectations/timestamps_steps.proto.txt
[add] https://crrev.com/3f5844f8dddb36cd14bd3f050916f0aa7878aec8/client/logdog/annotee/annotation/test_expectations/timestamps_steps_one.proto.txt
[add] https://crrev.com/3f5844f8dddb36cd14bd3f050916f0aa7878aec8/client/logdog/annotee/annotation/test_expectations/timestamps_steps_two.proto.txt
[modify] https://crrev.com/3f5844f8dddb36cd14bd3f050916f0aa7878aec8/client/logdog/annotee/processor.go

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/30e6ec55c034c0c327a38c25a040727a07276c61

commit 30e6ec55c034c0c327a38c25a040727a07276c61
Author: nodir <nodir@chromium.org>
Date: Thu Jun 16 21:52:03 2016

milo: fix running steps

milo displays running steps as succeeded because annotation processor incorrectly closes all steps in the end. Fix that.

Milo did not have css rule for running builds. Fix that.

Also do not add empty property groups to milo build.

Also fix build duration for running builds.

Refactor some code related to changes above.

R=hinoka@chromium.org, estaab@chromium.org, dnj@chromium.org
BUG= 613584 

Review-Url: https://codereview.chromium.org/2078603002

[modify] https://crrev.com/30e6ec55c034c0c327a38c25a040727a07276c61/appengine/cmd/milo/frontend/static/buildbot/css/default.css
[modify] https://crrev.com/30e6ec55c034c0c327a38c25a040727a07276c61/appengine/cmd/milo/logdog/logDogStream.go
[modify] https://crrev.com/30e6ec55c034c0c327a38c25a040727a07276c61/appengine/cmd/milo/swarming/build.go
[modify] https://crrev.com/30e6ec55c034c0c327a38c25a040727a07276c61/appengine/cmd/milo/swarming/build_test.go
[modify] https://crrev.com/30e6ec55c034c0c327a38c25a040727a07276c61/appengine/cmd/milo/swarming/expectations/build-pending.json
[add] https://crrev.com/30e6ec55c034c0c327a38c25a040727a07276c61/appengine/cmd/milo/swarming/expectations/build-running.json
[add] https://crrev.com/30e6ec55c034c0c327a38c25a040727a07276c61/appengine/cmd/milo/swarming/testdata/build-running
[add] https://crrev.com/30e6ec55c034c0c327a38c25a040727a07276c61/appengine/cmd/milo/swarming/testdata/build-running.swarm
[modify] https://crrev.com/30e6ec55c034c0c327a38c25a040727a07276c61/client/cmd/logdog_annotee/main.go
[modify] https://crrev.com/30e6ec55c034c0c327a38c25a040727a07276c61/client/logdog/annotee/annotation/annotation.go
[modify] https://crrev.com/30e6ec55c034c0c327a38c25a040727a07276c61/client/logdog/annotee/processor.go
[modify] https://crrev.com/30e6ec55c034c0c327a38c25a040727a07276c61/common/proto/milo/annotations.pb.go
[modify] https://crrev.com/30e6ec55c034c0c327a38c25a040727a07276c61/common/proto/milo/annotations.proto

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/5bb3c22e0c674555fbc03424aad46fe5a85e7d58

commit 5bb3c22e0c674555fbc03424aad46fe5a85e7d58
Author: recipe-roller <recipe-roller@chromium.org>
Date: Fri Jun 10 02:18:40 2016