Also a nice feature would be that the signature would be small enough to fit into an UMA stability report.

Something like this might work:
Translate all machine words on the stack to {module,offset} (where possible), which has the effect of normalizing them with respect to load address. These pairs will make a superset of all return addresses on the stack. It's possible that e.g. constraining to only .text segments will help.

Now hash these pairs to an integer, each of which is used to set a bit in a bit in a bit vector. You now possibly have a crude "proximity" measure for two vectors by counting the number of common bits in them.

What we do in SyzyASAN:

- Calculate a heuristic crash stack.
- Map each frame to a (module, offset), when possible. Frames that don't match this get mapped to a dummy module and zero offset.
- Frames that are in non-Chrome code get converted to a normalized module name, and no offset. (Deals with different versions of third party libraries, etc)
- Consecutive non-Chrome frames are collapsed to a single frame (taking on the name of the first module in the stack). Again, helps to deal with differing implementations across different platform/library versions.
- The entire thing is hashed into a single 32-bit value.

For example:

0x00007ff9c4dbb3be chrome.dll + offset5
0x00007ff9c4d04875 chrome.dll + offset4
0x00007ff7ede55b25 chrome.exe + offset3
0x00007ff9ef278363 KERNEL32.DLL + offset2
0x00007ff9ef278363 KERNEL32.DLL + offset1
0x00007ff9f16f5e90 ntdll.dll + offset0

becomes

(chrome.dll, offset5)
(chrome.dll, offset4)
(chrome.exe, offset3)
(kernel32.dll, offset2)
(kernel32.dll, offset1)
(ntdll.dll, offset0)

becomes:

(chrome.dll, offset5)
(chrome.dll, offset4)
(chrome.exe, offset3)
(ntdll.dll)

Which is then hashed. No need to hash Chrome version numbers, as those are implicit in the crash report. So the signature is unique for a given stack with a given version of "client" binaries (Chrome, in this case).