Provide debug symbols for Mac
Reported by
bmau...@fb.com,
Feb 17 2017
|
|||||||||||
Issue descriptionWhat steps will reproduce the problem? (1) attempt to use a debugger or profiler that requires debug symbols on a production macOS build What is the expected result? On windows, Chrome provides downloadable symbols for production builds. In cases where a FB employee has been able to reproduce a hang or excessive CPU usage on a windows machine we've been able to track down the cause. What happens instead? Most of our employees use Mac and this greatly reduces our ability to use this technique.
,
Feb 18 2017
<dumb-question>Is the cost of providing symbols to release builds just binary size?</dumb-question> Beyond symbolizing traces, I'd really like to be able to hook up Instruments to an arbitrary Chrome release build and get useful data out of it. I think we'd see a lot more effective measurement of perf in the wild if that were easy. Whether that's through bundling symbols with the production build or some other mechanism, I don't really care as long as it's trivial to do. For example, could we add an about:symbols page that has a button you can click to download symbols for the Chrome binary you're on? That way we don't bloat the binary for normal users but anyone who wants to do perf testing just needs to click a button once per release.
,
Feb 23 2017
Mike and I discussed publishing the symbols at the convergence in January. There's no reason why we can't publish the dSYMS for Chrome like we do PDBs for Windows. The only complication at the moment is that the symbols live in a storage bucket with the unsigned official build products, which I don't think we want to distribute. We should consult the TPMs and ask if we can set up a separate GCS bucket just for the symbols and store them there, instead. Ojan: Internally you can already grab symbols for official builds through the GCS bucket (go/chrome-symbols).
,
Mar 6 2017
Friendly ping, any chance someone has managed to speak to the TPMs about this?
,
Mar 14 2017
Gonna give this another ping :-). This is really important for us to be able to debug issues that are reported internally
,
Mar 14 2017
Sorry, this bug is not a high priority at the moment.
,
Mar 15 2017
rsesek: Are there instructions on how to use Instruments with those symbols on a production Chrome? I wasn't able to find anything Chrome specific, but I found https://developer.apple.com/library/content/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/LocatingSymbolsforYourData.html. I assume that would work? Also, who maintains the infrastructure for the symbol archiving? Is that the Ops team? I'm trying to understand the steps needed to get what I want and who would do it. Whether it's a priority to do them now is another question, but I at least want to know who to nag. :) The extra hurdle of finding the right symbols for my current Chrome release, downloading it, hooking it up to Instruments is enough that I won't bother in many cases. I think that if we make this dirt simple, we'll see more Chrome devs doing perf profiling of Chrome Mac on real content. So, if the GCS bucket were publicly available, I think this would be really simple to build: 1. about:symbols page where you click a button and it downloads the right symbols to a well known location. 2. We can tell people to put that location in their Instruments dSYM search path That way, there's a setup step (#2) you do once and after that you only ever need to go to the symbols page and click a button.
,
Mar 15 2017
ojan: Yes, that would work. Generally tools on the system (e.g., lldb) find symbols via Spotlight indexing the .dSYMs based on their UUID (which match the LC_UUID embedded in the stripped binary). As for the pipeline: mmoss@ I believe would be a good place to start with asking about changes to the official build process.
,
Mar 15 2017
Given a UUID, is there a way for someone like me to locate the archived dSYM somewhere in Google?
,
Mar 15 2017
This Dremel query will work: SELECT Product.Version FROM crash.prod.latest WHERE Product.Name = 'Chrome_Mac' AND CrashedStackTrace.StackFrame.Module.DebugFile = 'Google Chrome Framework' AND CrashedStackTrace.StackFrame.Module.DebugIdentifier = '7A8C35E7141C37389AAC0C1F809438E30' GROUP BY Product.Version You need to remove the dashes from the UUID and add a trailing 0 (which Breakpad adds as a "generation number").
,
Mar 15 2017
And if you want to look up multiple UUIDs:
SELECT Product.Version, CrashedStackTrace.StackFrame.Module.DebugIdentifier AS UUID FROM crash.prod.latest
WHERE Product.Name = 'Chrome_Mac'
AND CrashedStackTrace.StackFrame.Module.DebugFile = 'Google Chrome Framework'
AND CrashedStackTrace.StackFrame.Module.DebugIdentifier IN ('7A8C35E7141C37389AAC0C1F809438E30', '824C7E6383843F16BD487E5BFA58E2510')
GROUP BY Product.Version, UUID
,
Mar 15 2017
Thank you rsesek@. rsesek@ also points out that this bug is about changing the build process to provide easy access to symbols while my interest in it is a little different and not so related to that overall goal, so I'm changing the priority back to p3.
,
Mar 23 2017
I spoke with +waffles, and he thinks it wouldn't be too difficult to incorporate this into the Omaha push for each release. Basically, when the TPMs trigger a push, Omaha would do all its normal stuff, and then also grab the corresponding symbol tarball from the internal build archive and copy them to a public bucket. The public location would follow a well-defined format, something like gs://chrome-mac-symbols/<version>/<builder>/<tarball> (e.g. gs//chrome-mac-symbols/1.2.3.4/mac64/Google Chrome.dSYM.tar.bz2), so it would be easy to find the symbols for any released build. Would something like that work?
,
Mar 23 2017
Yeah, something like this would work. If it became popular we could perhaps come up with a way to automate some of this (eg a shell script that would fetch symbols) but just getting the basics working here would be great and it would be good to prioritize providing the symbols
,
Mar 23 2017
Ah, I was thinking it would be very easy to extend the same system that we use for publishing other release-associated artifacts at push time (e.g. the domain policy templates, enterprise installers, etc); there is one wrinkle though, that for all of those the channel appears in the path - we could maybe do something clever to remove it. Also, I was thinking the public path would probably be something like https://redirector.gvt1.com/edgedl/chrome/mac-symbols/<channel?>/<version>/<builder>/<tarball>. Is it OK to unpublish symbol files for versions less than [the version of stable 14 days ago]?
,
Apr 3 2017
Yeah, for our use case I think having symbols for any build that's not more than 14 days out of date is fine
,
Apr 10 2017
Just checking in on this -- we had another case today where a user had a hang that symbols would have allowed us to debug
,
Apr 10 2017
,
Apr 10 2017
,
Apr 10 2017
Do you still have the hang? We could probably provide you with symbols for that version of Chrome.
,
Apr 10 2017
Yes, his tab is still hanging and he's able to reproduce it pretty often. He's running Chrome 57.0.2987.133 on OS X
,
Apr 10 2017
rsesek@ - what does it take to find these symbols? Alternatively, if you can send the hang we can quickly symbolize it and send it back.
,
Apr 10 2017
I filed issue 710209 for the video player hang
,
Apr 21 2017
waffles@, are you the right owner for exposing the symbols at a public path? Unpublishing symbols for releases older than the current stable version seems fine to me as well. As an aside, is there a straightforward way to get the symbols into Instruments? I got it to work in the end, but I had to rename the package to remove the file extension so Instruments would treat the package as a directory and recurse into it.
,
Apr 24 2017
> waffles@, are you the right owner for exposing the symbols at a public path? Yes.
,
Aug 9 2017
,
Aug 29 2017
Can we get this symbolicated? It comes from an employee hang on Chrome 60 (Release) that's waiting on some condition variable. Symbol would tell us what they block on. If the results are interesting I'll file a issue for it.
,
Nov 2 2017
Here you go.
,
Jun 7 2018
|
|||||||||||
►
Sign in to add a comment |
|||||||||||
Comment 1 by shrike@chromium.org
, Feb 18 2017Status: Assigned (was: Unconfirmed)