Mojo Linux Perf: Cannot see perf graph on the Perf Dashboard |
||||||||||||
Issue description(Not sure if this is my benchmark's bug or Perf Dashboard's bug) Background: I'm running loading.desktop.network_service[1] on the Mojo Linux Perf[2] bot, which basically: 1. Run loading.desktop twice w/ and w/o feature; 2. Compute the difference on (lists of) values, rename subtests, and upload. e.g. 'timeToFirstContentfulPaint_avg' becomes * 'timeToFirstContentfulPaint_avg_control' * 'timeToFirstContentfulPaint_avg_enabled' * 'timeToFirstContentfulPaint_avg_diff' Problem: The benchmark has been running green for one week (since May 1st), but I couldn't get expected graph from the Perf Dashboard[3]. More specifically: * 'timeToFirstContentfulPaint_avg_diff' and 'timeToFirstContentfulPaint_avg_diff/cold' is empty; * 'timeToFirstContentfulPaint_avg_diff/warm' only has some old data. Can I have some hints on where I should be looking at? Thanks! (e.g. Does the Perf Dashboard filter names, or do they apply some restrictions to 'foo_avg'?) --- Please see attachments for screenshot of the Perf Dashboard as well as sample JSON from the bot. [1] https://cs.chromium.org/chromium/src/tools/perf/contrib/network_service/loading.py?l=141&rcl=a2ccc9338c77902ad18006d9a79e739b4de60aa9 [2] https://ci.chromium.org/buildbot/chromium.perf.fyi/Mojo%20Linux%20Perf/ [3] https://chromeperf.appspot.com/report?sid=479240c958c5e024a6c45e6c20acd454c8d3fc34cebdfc9eb29d3891e5035fce
,
May 8 2018
Locally debugging seems fine, metric is added as expected. So there seem to be some failures in add_point/add_point_queue trying to save the TestMetadata, specifically the monitored property on the suite level entity is too large. Will have to dig into that a bit more. I think this output is considerably larger than anything we've had to deal with up until now.
,
May 8 2018
I see, thanks for the help! I will check about size constrains in the dashboard code.
,
May 9 2018
It seems that the output will spawn 19973 tasks, and each post to add_point_queue handles 32 tasks. Not sure if this is causing problems but it never finishes (or took a very long time) on my local. Going to reduce the size by only record *_diff and see if that helps.
,
May 9 2018
That should help. The 20k shouldn't be an issue, I think it's a bottleneck in how we stored "monitored" tests. See https://github.com/catapult-project/catapult/issues/3930
,
May 10 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/0b962604c44e6d7ffeb845b9576bca6dece340bf commit 0b962604c44e6d7ffeb845b9576bca6dece340bf Author: Chong Zhang <chongz@chromium.org> Date: Thu May 10 02:58:39 2018 Mojo Linux Perf: Only upload _diff charts to Perf Dashboard Perf Dashboard has some issues handling large test results (see bug). This patch tentatively changed the benchmark to only upload _diff charts (which is consider most useful) to get the graph running, and we should revert the benchmark back after the issue has been fixed. Bug: 840524 Change-Id: I4e87da8405bf102f16c76ccad28107b3c9b87be7 Reviewed-on: https://chromium-review.googlesource.com/1052674 Reviewed-by: Jun Cai <juncai@chromium.org> Commit-Queue: Chong Zhang <chongz@chromium.org> Cr-Commit-Position: refs/heads/master@{#557437} [modify] https://crrev.com/0b962604c44e6d7ffeb845b9576bca6dece340bf/tools/perf/contrib/network_service/loading.py
,
May 10 2018
Re #c5: Thanks for the explanation and link to the bug! Now the size should be similar to the normal loading benchmark (5.1MB), will see if it works.
,
May 14 2018
,
May 15 2018
FYI I did some work last week and this week to get rid of this bottleneck, this might work now without changes to the benchmark.
,
May 15 2018
That’s really awesome, thanks! (The graph was still a little bit weird since my last patch, but hopefully the issue could go away now)
,
May 16 2018
Re #c9: I'm getting expected data on the Perf Dashboard now (data starts from 2018-05-15) and it look great! Thanks a lot for the help! Will let it run for a few days and add back original charts.
,
May 18 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/a68350f4606b9b37314fa4e795f17b0905727733 commit a68350f4606b9b37314fa4e795f17b0905727733 Author: Chong Zhang <chongz@chromium.org> Date: Fri May 18 23:42:59 2018 Mojo Linux Perf: Add back _enabled and _control charts The charts was previously removed in: https://chromium-review.googlesource.com/c/chromium/src/+/1052674 We can add them back now since the Perf Dashboard has been fixed in https://github.com/catapult-project/catapult/issues/3930. Bug: 840524 Change-Id: Id4b199573af658a5aa27cfde85dae77f47f82b03 Reviewed-on: https://chromium-review.googlesource.com/1066779 Commit-Queue: Chong Zhang <chongz@chromium.org> Reviewed-by: Jun Cai <juncai@chromium.org> Cr-Commit-Position: refs/heads/master@{#560102} [modify] https://crrev.com/a68350f4606b9b37314fa4e795f17b0905727733/tools/perf/contrib/network_service/loading.py
,
May 21 2018
|
||||||||||||
►
Sign in to add a comment |
||||||||||||
Comment 1 by simonhatch@chromium.org
, May 7 2018