Note to self, the set of benchmarks run by the Windows PGO bot is here: https://cs.chromium.org/chromium/src/build/win/run_pgo_profiling_benchmarks.py

I'm somewhat unconvinced by how pgo went on Windows -- it's slow enough that we can't use it on the perf bots, and so we've been flying blind on Windows since launching pgo. (Clang/win fixes this cause the order files change less often and are generated asynchronously to the build.)

If we do this, profile generation should be done asynchronously to the build.

Priority wise, I'd say this is after using lld by default in linux, and maybe Android.

I'm unfamiliar with Windows PGO. By asynchronous do you mean that collecting profile data and applying profile data don't have to use the exact same codebase? If so, I believe that this is how profile-guided section layout works as well (e.g. if a symbol is missing, lld will just ignore that part of the profile data).

On Windows, we currently do
1 build
2 run some benchmarks to collect profiles
3 build again, using profiles

This takes 6h. With asynchronously, I mean:
Profile collection bot builds, collects profiles, uploads them somewhere, repeat. Actual builder downloads recent profiles, builds. (Which sounds like what you're saying too)

I believe profile-guided section layout has potential.

Quick thoughts about Android:
* AutoFDO is designed for taking slightly stale profiles
* One unfortunate constraint on ARM is that LBR is not available, which severely limits the optimization power of AutoFDO
* the current orderfile slows down execution somewhat, but improves startup, to push something new we would need to evaluate both of these dimensions
* In the android land we are also becoming increasingly worried about the binary footprint. Which means, the optimization must be tuned to not expand code. Also, if we could gauge this optimization to group unused code to a special section of the binary that is supposedly unused, this would save some resident memory. If it turns out to be >1MiB of savings, that would be good, if it's >10MiB, I'll finance a beer party for everyone involved :)

> * In the android land we are also becoming increasingly worried about the binary footprint. Which means, the optimization must be tuned to not expand code. Also, if we could gauge this optimization to group unused code to a special section of the binary that is supposedly unused, this would save some resident memory. If it turns out to be >1MiB of savings, that would be good, if it's >10MiB, I'll finance a beer party for everyone involved :)

I'm curiously why you want to pack rarely used code (not unused, right?) to a special section. My understanding is that, if you correctly identify rarely used functions and aggregate them into one place in .text, the kernel won't map the pages for the functions because they are not used at runtime. So you can save memory, no?

that's what I am thinking about - extract rarely used functions out of .text, aggregate them to a contiguous region, say at the end of .text. I did not mean to make a special _section_ in the binary beyond .text. Pardon if mentioning "section" confuses the matter.

The kernel will of course have the byte range mapped to virtual memory, but the hope is that if the pages stay mostly unused, this mapping won't have pages of physical RAM attached.

> One unfortunate constraint on ARM is that LBR is not available, which severely limits the optimization power of AutoFDO

We might consider using profiles generated on x86 when targeting ARM. Hopefully the differences between the profiles will be minor enough not to matter in practice.

> In the android land we are also becoming increasingly worried about the binary footprint. Which means, the optimization must be tuned to not expand code.

Since this is purely a reordering optimization I would not expect it to increase code size, in fact I would expect there to be a slight decrease in code size on ARM because the linker would need to create fewer range extension thunks as a result of functions being placed nearer to their callers.

> > One unfortunate constraint on ARM is that LBR is not available, which
> > severely limits the optimization power of AutoFDO

> We might consider using profiles generated on x86 when targeting ARM.
> Hopefully the differences between the profiles will be minor enough not to
> matter in practice.

Agreed. Most likely for laying out sections in the code the LBR should not
matter. But since nobody considered LBR-less AutoFDO before, I don't have
confidence in how it affects the inline heuristics.

> > In the android land we are also becoming increasingly worried about the
> > binary footprint. Which means, the optimization must be tuned to not expand
> > code.
>
> Since this is purely a reordering optimization I would not expect it to
> increase code size, in fact I would expect there to be a slight decrease in
> code size on ARM because the linker would need to create fewer range extension
> thunks as a result of functions being placed nearer to their callers.

Fingers crossed :)

In my old experiment AutoFDO produced more compact code compared to -O2 (3 MiB
saving), but produced bigger code compared to the "partial O2" that we ship
these days (300KiB more). Nobody knows the reasons, but I speculate that "it is
all the inliner", which was not tuned for our situation, I guess.

List of benchmarks used for AutoFDO on Chrome OS: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/master/server/site_tests/telemetry_AFDOGenerate/telemetry_AFDOGenerate.py#44

I haven't had a chance to compare that list with the one used on Windows.

I collected a presumably more realistic profile using the Windows benchmark list linked above, together with blink_perf.layout. That gave me a median 5% improvement across the benchmarks in blink_perf.layout as compared to no layout. I then made some improvements to the lld patch linked above:
- if a section is replaced with an identical section using ICF, apply the profile to the replacement section
- create a separate symbol table for looking up sections that also includes local symbols.
With those changes I get a median 7.5% improvement across blink_perf.layout.

pcc: nice!

Would it be hard to get numbers for non-microbenchmarks from the list above?

Since the sunset of octane, the most important benchmarks from the list seem to be:
1. page_cycler_v2.* (but --pageset-repeat=1 may be low for sampling profiler)
2. speedometer

This is probably a silly question, but the numbers look too good to me. 5% to 7.5% improvements look huge. Is it because they are micro-benchmarks?

#11: I will take a look.

#12: Yes, they are microbenchmarks. The numbers do seem in line with those reported in the Facebook paper, though.

Upon re-reading, this is maybe more similar to the orderfile stuff we do in win/clang builds than full PGO (?). Hans knows all about that.

The Windows orderfile stuff is even documented nowadays: https://chromium.googlesource.com/chromium/src/+/master/docs/win_order_files.md

Archiving P3s older than 1 year with no owner or component.