Hmm.. looks like both "histogram" and "diagnostics" return null on /api/timeseries2 for internal bots (I assume because they do not upload histograms?).

Is there another way I could retrieve the trace URLs for those?

e.g. I want to be able to get some example traces from:
https://chromeperf.appspot.com/report?sid=397ff6aace03427d3d17e8475bb72a9c9db57f05990c524092a7b48e3940f7c2&start_rev=1539346032&end_rev=1540893828

Ping, Ben?

I can add ?columns=annotations to get all a_* properties ala revisions:
https://github.com/catapult-project/catapult/blob/master/dashboard/dashboard/api/timeseries2.py#L261

Would that work?

Eventually, I'd like to try to simplify both the datastore models and the API to ignore Row entities and their expando properties, and only support Histograms and their statistics and Diagnostics, but I think we'll need a few cleanups before we can do that.

Yeah, I think that would work as a short-term workaround.

Ben, assigning to you for the API change. When that's done please assign back to me. Thanks!

The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/f22b70c5e5a7ec75bfbad1f3022027af764f9240

commit f22b70c5e5a7ec75bfbad1f3022027af764f9240
Author: Ben Hayden <benjhayden@chromium.org>
Date: Fri Nov 09 19:24:00 2018

Expose annotations via /api/timeseries2

Bug: chromium:900218
Change-Id: Idc0bddd469e9f8f47a8a3d663298e37570c57248
Reviewed-on: https://chromium-review.googlesource.com/c/1324150
Reviewed-by: Dave Tu <dtu@chromium.org>
Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org>
Commit-Queue: Ben Hayden <benjhayden@chromium.org>

[modify] https://crrev.com/f22b70c5e5a7ec75bfbad1f3022027af764f9240/dashboard/dashboard/api/timeseries2.py
[modify] https://crrev.com/f22b70c5e5a7ec75bfbad1f3022027af764f9240/dashboard/dashboard/api/timeseries2_test.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/33c10c78715457aac9937c0a5b71b477233cc9da

commit 33c10c78715457aac9937c0a5b71b477233cc9da
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Sat Nov 10 09:22:50 2018

Roll src/third_party/catapult 5d5091665700..f2c3502f7788 (9 commits)

https://chromium.googlesource.com/catapult.git/+log/5d5091665700..f2c3502f7788


git log 5d5091665700..f2c3502f7788 --date=short --no-merges --format='%ad %ae %s'
2018-11-10 nednguyen@google.com [browser_test_runner] Ensure that skipped tests have skipped expectation if typ expectation file is not used
2018-11-09 benjhayden@chromium.org Add BatchIterator for v2spa
2018-11-09 benjhayden@chromium.org Merge test suite descriptors in v2spa
2018-11-09 benjhayden@chromium.org Add RecommendedOptions to v2spa
2018-11-09 benjhayden@chromium.org Support uploading to dev_appserver.py
2018-11-09 benjhayden@chromium.org Generalize complex test cases in Descriptor.
2018-11-09 benjhayden@chromium.org Expose annotations via /api/timeseries2
2018-11-09 benjhayden@chromium.org Add tag-filter for v2spa
2018-11-08 dpranke@chromium.org Integrate test expectation-parsing code into TYP.


Created with:
  gclient setdep -r src/third_party/catapult@f2c3502f7788

The AutoRoll server is located here: https://autoroll.skia.org/r/catapult-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG= chromium:904019 ,chromium:898552,chromium:900218, chromium:835690 
TBR=sullivan@chromium.org

Change-Id: Ic8d2d5b5c618bef91ecd2c6051a9e85ab340dab9
Reviewed-on: https://chromium-review.googlesource.com/c/1330771
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#607117}
[modify] https://crrev.com/33c10c78715457aac9937c0a5b71b477233cc9da/DEPS

I'm getting "null" when requesting the 'annotations' column, is the change in #7 live?

Should be live now.

Awesome! Thanks.

The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/d69ae20edf9486cf53e4ee007c89a1518b03abab

commit d69ae20edf9486cf53e4ee007c89a1518b03abab
Author: Juan Antonio Navarro Perez <perezju@chromium.org>
Date: Wed Nov 21 17:21:43 2018

[soundwave] Add timeseries2 API

Add dashboard_service.Timeseries2() function, and add support to it
from soundwave.

As part of v2 benefits, we will now get units and trace urls.

Since not all soundwave commands can be switched to v2 (e.g. the -b
option to get sheriffed timeseries from a benchmark, crbug.com/907121),
for now we keep support of both v1 and v2 APIs.

To distinguish between both, old queries require a full test_path
string (as before), while new queries require building a timeseries.Key
object to uniquely identify a particular timeseries.

In this CL v8_study is migrated to v2. Other studies and the -i
option to read from a file will be done on follow up CLs.

Bug: chromium:900218
Change-Id: If6914b01b9239c5f336009345f06afc8084bb2d7
Reviewed-on: https://chromium-review.googlesource.com/c/1333818
Commit-Queue: Juan Antonio Navarro Pérez <perezju@chromium.org>
Reviewed-by: Sami Kyöstilä <skyostil@chromium.org>
Reviewed-by: Ben Hayden <benjhayden@chromium.org>

[modify] https://crrev.com/d69ae20edf9486cf53e4ee007c89a1518b03abab/experimental/soundwave/soundwave/tables/timeseries.py
[modify] https://crrev.com/d69ae20edf9486cf53e4ee007c89a1518b03abab/experimental/soundwave/soundwave/tables/timeseries_test.py
[modify] https://crrev.com/d69ae20edf9486cf53e4ee007c89a1518b03abab/experimental/soundwave/soundwave/studies/v8_study.py
[modify] https://crrev.com/d69ae20edf9486cf53e4ee007c89a1518b03abab/experimental/soundwave/services/dashboard_service.py
[modify] https://crrev.com/d69ae20edf9486cf53e4ee007c89a1518b03abab/experimental/soundwave/soundwave/commands.py
[modify] https://crrev.com/d69ae20edf9486cf53e4ee007c89a1518b03abab/experimental/soundwave/soundwave/dashboard_api.py
[modify] https://crrev.com/d69ae20edf9486cf53e4ee007c89a1518b03abab/experimental/soundwave/common/utils.py
[modify] https://crrev.com/d69ae20edf9486cf53e4ee007c89a1518b03abab/experimental/soundwave/soundwave/dashboard_api_test.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e6338ead9687097f56800c5dcce84f365d6df911

commit e6338ead9687097f56800c5dcce84f365d6df911
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Wed Nov 21 19:27:57 2018

Roll src/third_party/catapult fd17a12ec7ac..d69ae20edf94 (1 commits)

https://chromium.googlesource.com/catapult.git/+log/fd17a12ec7ac..d69ae20edf94


git log fd17a12ec7ac..d69ae20edf94 --date=short --no-merges --format='%ad %ae %s'
2018-11-21 perezju@chromium.org [soundwave] Add timeseries2 API


Created with:
  gclient setdep -r src/third_party/catapult@d69ae20edf94

The AutoRoll server is located here: https://autoroll.skia.org/r/catapult-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG=chromium:900218
TBR=sullivan@chromium.org

Change-Id: I15708dc6e61810aa55530c6e1713f11350b2f545
Reviewed-on: https://chromium-review.googlesource.com/c/1347071
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#610151}
[modify] https://crrev.com/e6338ead9687097f56800c5dcce84f365d6df911/DEPS

Hmm, is there a way for /timeseres2 to give me a timeseries for a group of test_case's?

For example in the old v1 I could ask for '{bot}/memory.top_10_mobile/{measurement}/foreground' to get the average of all foreground stories. Is something like that possible? Or do I have to fetch timeseries for all stories and average them myself?

The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/43e8ebcaf28498c50c27fcf3b63b6e189ff12fb4

commit 43e8ebcaf28498c50c27fcf3b63b6e189ff12fb4
Author: Juan Antonio Navarro Perez <perezju@chromium.org>
Date: Fri Nov 23 11:59:09 2018

[soundwave] Use /timeseries2 to fetch data on -i option

Now, instead of a text file with test_paths, the -i option will read a
json file with a list of timeseries key descriptions to read.

In addition to give more structure to the input, the data obtained
will now also include trace_url and correct unit information.

Bug: chromium:900218
Change-Id: Ia98b39050201e8a09e32eea6378c977ccc3d76ab
Reviewed-on: https://chromium-review.googlesource.com/c/1349352
Commit-Queue: Juan Antonio Navarro Pérez <perezju@chromium.org>
Reviewed-by: Sami Kyöstilä <skyostil@chromium.org>

[modify] https://crrev.com/43e8ebcaf28498c50c27fcf3b63b6e189ff12fb4/experimental/soundwave/bin/soundwave
[modify] https://crrev.com/43e8ebcaf28498c50c27fcf3b63b6e189ff12fb4/experimental/soundwave/soundwave/tables/timeseries.py
[add] https://crrev.com/43e8ebcaf28498c50c27fcf3b63b6e189ff12fb4/experimental/soundwave/soundwave_examples/startup_timeseries.json
[modify] https://crrev.com/43e8ebcaf28498c50c27fcf3b63b6e189ff12fb4/experimental/soundwave/soundwave/tables/timeseries_test.py
[modify] https://crrev.com/43e8ebcaf28498c50c27fcf3b63b6e189ff12fb4/experimental/soundwave/soundwave/commands.py

No longer blocking needs of scheduler team ( issue 900216 ).

Remaining work is to also switch the health report to /timeseries2, but that is now blocked on issue 907883.

Btw +chiniforooshan, now you should be able to get traces for any timeseries by doing e.g.:

catapult/experimental/soundwave $ ./bin/soundwave timeseries -i soundwave_examples/startup_timeseries.json --output-csv output.csv

Awesome, thanks Juan!

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b778d86c12f35f8fef5285009f67978960af2837

commit b778d86c12f35f8fef5285009f67978960af2837
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Fri Nov 23 14:26:19 2018

Roll src/third_party/catapult 4ed4737007d8..43e8ebcaf284 (2 commits)

https://chromium.googlesource.com/catapult.git/+log/4ed4737007d8..43e8ebcaf284


git log 4ed4737007d8..43e8ebcaf284 --date=short --no-merges --format='%ad %ae %s'
2018-11-23 perezju@chromium.org [soundwave] Use /timeseries2 to fetch data on -i option
2018-11-23 perezju@chromium.org [soundwave] On finalization terminate pool, don't close.


Created with:
  gclient setdep -r src/third_party/catapult@43e8ebcaf284

The AutoRoll server is located here: https://autoroll.skia.org/r/catapult-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG=chromium:900218, chromium:907885 
TBR=sullivan@chromium.org

Change-Id: Id2475cd9a73b69a2231b508deb72467107836bd6
Reviewed-on: https://chromium-review.googlesource.com/c/1349434
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#610602}
[modify] https://crrev.com/b778d86c12f35f8fef5285009f67978960af2837/DEPS