New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 14 users

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Feature



Sign in to add a comment

DevTools: Flame Graph / Flame Chart toggle

Reported by bgr...@netflix.com, Jan 27 2015

Issue description

UserAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36

Steps to reproduce the problem:
1. View->Developer->Developer Tools
2. Profiles->Collect JavaScript CPU Profile->Start ... Stop
3. Select "Chart" (aka "Flame Chart")

What is the expected behavior?
A flame chart is shown.

What went wrong?
This feature request is to provide another profile view, flame graph, in addition to the current "Chart" (flame chart) view. Both flame charts and flame graphs visualize the same profile data, with the only difference the x-axis:

- Flame Charts: x-axis shows the passage of time. Time-based patterns can be identified. For long profiles with short and changing code paths, the stack "flames" can appear thin and hard to read.
- Flame Graphs: x-axis sorts the stacks alphabetically, from origin to leaf. This maximizes merging of frames, allowing the big picture of CPU usage to be visualized. These can be useful in other cases, for example, non-regression testing, where a pair of flame graphs can be visually compared and small differences seen.

An explanation for flame graphs is: http://www.slideshare.net/brendangregg/blazing-performance-with-flame-graphs/35. It would be awesome to have a tool that can toggle between both types of visualization for the same profile data.

Did this work before? N/A 

Chrome version: 39.0.2171.95  Channel: n/a
OS Version: OS X 10.9.2
Flash Version: Shockwave Flash 16.0 r0
 
Cc: paulir...@chromium.org
Labels: -Type-Bug -OS-Mac Type-Feature OS-All
Status: Untriaged

Comment 2 by eustas@chromium.org, Jan 28 2015

Cc: yu...@chromium.org
Labels: -Via-Wizard -OS-All
Owner: loislo@chromium.org
Status: Assigned

Comment 3 by yu...@chromium.org, Jan 28 2015

Cc: a...@chromium.org

Comment 4 by yu...@chromium.org, Jan 28 2015

@bgregg: am I right that what you'd like to see is basically graphical representation of "Top Down" view. I.e. the side along x-axis would reflect "Total" column from "Top Down" view?

Comment 5 by bgr...@netflix.com, Jan 28 2015

@yurys: Yes, that's basically it. With the current top down view, one must expand code paths and read the percentages. With the flame graph, wider means more, and one can visually compare the widths of frames and code paths to understand the make up of the profile.

It's becoming a widespread profile visualization (google image search for "flame graphs"), so there's the benefit of users already having experience using these.
There's been some other discussion around this area recently, so it's great that it's been on your mind as well.
Btw thanks for dropping by the bug tracker; was a surprise to see the reporter on this one. :)

The time-based flamechart definitely works well for clientside apps and helps tell the narrative of how a user interacted and the system responded. That said, a weakness is definitely that it's up to the human to identify patterns and collate the various costs into a mental picture of the overall bottlenecks.  This is where we need some help. 

Below my current braindump on this visualization, hope to hear from other folks too.


The flame visualization has worked well, but it isn't without a few flaws.

* As a first-time flame graph user, intuitively long "flames" feel bad, but they're not. 
* The primary user  action is finding wide frames towards the "top" of the flame. But that's not obvious. 
* Frameworks can make stacks REALLY deep but may have little actual overhead 

These issues have a few potential solutions. Some ideas:

* Chop off flames if the total-time isn't really significant. Fade them too.
* Side-by-side "bottom-up view" with synchronized hover
* Filter out call frames in the middle that have insignificant self-time



Also recently, a few teams have been experimenting with Sunburst charts.
They're pretty much the same as a flame graph, but wrapped into a donut.

Basic sunburst demo:  http://etsukata.com/d3js_trace/sample1/ from https://github.com/Etsukata/d3js_trace#readme

Much more interactive & complete sunburst: https://dl.dropboxusercontent.com/u/39519/sunburst.html (3MB)

Sunburst feels like it has some advantages. It has more visual resolution at the leaf nodes and less at the base, which seems preferable for the task. To me, there's something about the curved surface area of high self-time frames that attracts my attention. The curve, to me, makes it feel just as interesting as a long skinny flame jutting out next to it.

However I have heard that some users are confused when first introduced to a sunburst chart. I'm interested in doing some user-testing to explore this some more.

Comment 7 by yu...@chromium.org, Jan 29 2015

I tried to adopt https://github.com/Etsukata/d3js_trace to show CPU profile in DevTools. If you want to play with that, just apply the patch from https://codereview.chromium.org/887673002/. I only checked it in debug_devtools=1 mode. The result should look like the attached screenshot.
sfgate.png
376 KB View Download

Comment 8 by yu...@chromium.org, Jan 29 2015

The chart build by the patch from #7 doesn't seem to include self time of non-leaf nodes so it gives somewhat distorted picture.

Comment 9 by bgr...@netflix.com, Jan 29 2015

@paul: Good to hear from you. Yes, I've been meaning to try sunburst since I saw them in http://homes.cs.washington.edu/~jheer/files/zoo/. It's great to have some examples of sunburst stack profiles to look at.

Sunburst is visually interesting, and I suspect we'll find some use cases given many real profiles. One value is that it creates a distinctive visual representation of software (the shape and coloring). Imagine collecting profiles for an entire datacenter or cloud, then visualizing each server as a sunburst. Like these: http://www.brendangregg.com/ColonyGraphs/cloud.html#Datacenter -- see the outlier servers? That's just ptree data; a sunburst stack profile should reveal new patterns and outliers. This might be a way for seriously large clouds (like ours at Netflix) detect odd software behavior, by looking for the outlier mini-sunbursts. (For the amount of profile data involved, I might need to write a sunburst implementation that can emit sunburst PNGs for each server, and then stitch the PNGs together.)

For single server analysis, and Chrome JavaScript, I'd generally want to use flame graphs for the overall view. Comparisons of different frame rectangle widths I think is a bit easier than comparing radians, especially at different radii. Plus there's the distortions at distant radii.

With a flame graph, you should still be able to print the function names in the rectangles that matter -- the biggest. This lends itself to the interpretation: the most important functions are the ones you can read. The flame graph can also be screen-shotted, and included in presentations, papers, books, and still be useful & readable.

As for thin wispy towers -- I elide them beneath a configurable threshold (eg, 0.1%). For really tall thin towers bigger than the threshold, then it does make the flame graph alarmingly tall. As you said, that can grab attention when it shouldn't. Perhaps the solution is to fade them, or truncate them with a visual clue (a zig-zag cut).

I'm not sure about the value of sunburst emphasizing the deep stack functions. There are cases where studying the stack leafs are of most interest, but for those I flip the stack order and draw an inverted flame graph, so they are merged. (I also draw them as an icicle plot, to emphasize that it's different from the flame graph.) I haven't got around to automating both: you can imagine a "butterfly plot", with a flame graph on top (merging from origin to leaf), and an icicle plot on the bottom (merging from leaf to origin).

For interpreting flame graphs, there's a few rules to follow, which I showed on slides 39 to 48 of http://www.slideshare.net/brendangregg/blazing-performance-with-flame-graphs/39 .

Yes, it's the top edge that matters -- showing what's on-CPU. But for a lot of my profiles, the on-CPU function is not interesting: bcopy(), strlen(), mutex_trylock(), etc. I need to look down the ancestry to understand the profile. Finding branches, like I showed on slide 48, is a quick and visual way to see the logical choices in the code -- and the functions that matter the most.

Also regarding the top edge: you can color the rectangles to show the real time they were on-CPU (self time?). I've done that before, and it can help. It's possible that different palettes can selected to highlight different details.

You could filter out frames you knew were insignificant, especially if a visual clue was included to indicate this happened. For the recent JVM hotspot profiling I'm doing, this happens anyway, since the JVM loves to inline functions like crazy. So the flame graphs we're looking at may only show 1/3rd of the frames -- the rest are inlined. They still make a lot of sense, though, so losing so many frames isn't a big problem. I'd prefer the full stacks if possible, but that can get rather tall.

Anyway, you have all the profile data, so having different views is possible. Flame charts for time-ordered problems, and flame graphs as well as sunbursts for summaries. It will be the best software profiler out there!
Brendan, thanks for the really valuable input.

We've been discussing performance this week and have some ideas on how best to use flame graphs help the user out. We've also been exploring combo'ing it with this sort of UX: http://mbostock.github.io/d3/talk/20111018/partition.html


Comment 11 by bgr...@netflix.com, Feb 12 2015

G'Day Paul,

Yes, the interactive d3 plot would be great. Rotated 90 degrees to the left (flame graph).

In the perl flame graph implementation, click-to-zoom happens on one dimension only (eg, http://www.brendangregg.com/FlameGraphs/cpu-mixedmode-vertx.svg), which I find preferable. The d3 implementation zooms both x & y, so that ancestor frames can no longer be seen. When you click 20 times on a complex flame graph, it's easy to forget where you are, so it's nice to keep them on the screen as a reference. So, for the d3 version, imagine it zoomed vertically only. Also, since visible ancestor frames are now partial (their size has been truncated), they can be changed visibly to indicate that (eg, in the perl version, they are faded).

The vertical flame graph allows deep stacks to fit on the screen (eg, 40 levels), with the function names still readable.

The current d3 orientation might may it difficult to include many stacks, especially if the visualization had a small area of screen real estate.

The transitions in the d3 version are very nice, and provide a good feeling for how far you are zooming in or out. The current perl version lacks that.
Owner: yu...@chromium.org

Comment 13 by yu...@chromium.org, Jun 10 2015

Owner: ----
Status: Available
Labels: Cr-Platform-DevTools-Performance
Summary: DevTools: Flame Graph / Flame Chart toggle (was: Flame Graph / Flame Chart toggle)
Owner: paulir...@chromium.org
Status: Assigned (was: Available)
Hi all, I have an idea on this issue.

Because Chrome 58 had change the CPU profile workflow.

https://developers.google.com/web/updates/2016/12/devtools-javascript-cpu-profile-migration

So, @bgregg original UI suggest is outdated.

But I found out in new profile workflow, that is in "Performance" panel, It is easier to implement this feature.

In "Performance" panel, the top-view we can treat it as a time-range picker for sample data, and the bottom-view, the sub-tabs are the difference statics information of selected sample data, when change the selection of sample data, all the sub-tabs will update. 

The data in "Call Tree" tab actually the data for drawing a flame graph which @bgregg said. So why not add a new tab "Flame Graph", reuse the data model in "Call Tree", but draw them in a flame graph, we can reuse the code which devtools bundle.

It only take about 100 lines to implement a minimum demo. Please take a look the attachments.

It is worth to implement the feature like this?

whole_profile_data.png
211 KB View Download
Another screenshot, select part of sample data, the "Flame Graph" will update.
part_profile_data.png
167 KB View Download

Comment 18 by bgr...@netflix.com, May 29 2017

Thanks, I think that looks good. Whole picture view. Are we able to zoom in on the flame graph tab? Eg, to select "Function Call".

It might be misleading to decorate it with an x-axis in milliseconds, since if it's showing the flame graph properly, it's not the passage of time. It might be less confusing to people to have percentages on that axis.

Sign in to add a comment