All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=862380

(For debugging:) Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?sid=03201729ec95f880cd83e3defd399087284123738421b448f25d6d802c56b067


Bot(s) for this bug's original alert(s):

linux

Aaand the AFDO-induced size turbulence continues. Pinging the maintainer of the pipeline we use to generate these profiles now, to see if there've been any recent changes. 7 out of our last 10 profile rolls have caused these alerts. :/

 Issue 865994  has been merged into this issue.

Bots stopped complaining, so I assumed this fixed itself. Turns out that lack of complaints was likely caused by lack of new size data, so I'm hopping back on this.

The AFDO pipeline maintainer said that there's been no new changes recently, so I'm suspecting that some massive function (of which I've seen a few in Chromium) is hovering very closely to some arbitrary coldness/hotness threshold. Should be interesting. :)

So, I've tried to attack this in a number of different ways. The summary is that profile flakes sometimes happen and are painful when they do, but the serious flakes appear to be rareish. I don't have many tools to debug flakes. Combining that with their apparent rarity, it's hard to say what there is to do about them. If they start to become a serious issue, we can set up some presubmit testing, but like said, they don't appear to happen often (monthly or so?)

Ignoring massive flakes, the more consistent binary size fluctuations on Linux (of 100-300KB) and on Android (of 10-100KB) aren't something that we can do much about. We'll be switching to a new profile source soonish, but only time will tell if that helps stability. 

Linux can swap to using -fprofile-sample-accurate, which should dampen noise quite a bit, but that comes at a performance cost. It's unclear what we'd end up paying if we went down this route. Android already does it, but they're far more size-conscious than Linux. Using -fprofile-sample-accurate would help because clang, by default, assumes that pieces of code with zero samples are 'unknowably hot' and optimizes them as such. With -fprofile-sample-accurate, clang starts to assume that zero samples = cold = optimize for size. This is relevant, since if we happen to get "lucky" and vector::push_back<Foo *> gets one sample, it'll be considered really cold. If the previous profile had zero samples, it'll be considered not-cold. Cold code is inlined less aggressively, optimized more for size, etc.

I have a stream-of-consciousness doc with my full notes available here: https://docs.google.com/document/d/1cVcCAjNYEPrFiEYfqW6wb1VQzyZqI5E_Qmoa2UDvszI/edit

...So, I'm closing this up, since it's hard to see what there is to do here. Like said, hopefully the new hotness (pun intended ;) ) in profiling will give us more consistent profiles. If not, we can potentially start looking into more creative reporting options (size builders with AFDO disabled + that bug me if the size gap between AFDO/non-AFDO remains above N% for M days, ...). Or we can see what tradeoffs -fprofile-sample-accurate brings. These all sound like fundamentally separate things, though.