Do you have an example of when it worked before? Ideally the most recent example? I don't think anything in the build script or diff triggering has changed recently.

Sorry, it didn't work for any of the Win64 PGO builds. However diffs are getting created for Win64 (Non-PGO) builds w/o fail.

I'm still not sure I understand what we should be seeing but aren't. I agree 2913 is messed up, but I'm not sure it's a general issue with PGO diffs. For instance, looking at the previous build, I see a couple diff installers:

https://pantheon.corp.google.com/storage/browser/chrome-signed/desktop-5c0tCh/56.0.2912.0/win64-pgo/

56.0.2912.0_56.0.2910.0_chrome_updater.exe
56.0.2912.0_56.0.2906.0_chrome_updater.exe

It looks like that second one is a diff against the current dev channel (56.0.2906.0). I assume that's what you're expecting to see?

Yes Correct. However '56.0.2912.0' & '56.0.2911.0' diffs generation seems like exception case to me. We have never seen the correct diffs got generated for 64-PGO builds except for 2911 & 2912.

I think there's some conflict between the 64-bit and the 32-bit diff creation. Probably the reason the 2912 and 2911 diffs worked for win64-pgo is that those builds both happened to fail on on win-pgo, so there was no conflict. I'm not sure exactly what the conflict is, but I seem to recall that this was also an issue with the old win builders, so might not be so new, nor specific to the PGO builders.

Yeah, I think this is an old problem, possibly exacerbated by the PGO builds taking a lot longer to run than the non-PGO builds. It looks the scripts only ever query Omaha for the 32-bit releases, so if the 32-bit build goes live before the 64-bit build finishes, then the 64-bit build thinks there is no older build, and you end up with meaningless diffs like:

56.0.2913.0_56.0.2913.0_chrome_updater.exe

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build/+/d51e12077dd2f577a9b43ff8f331f51c0a274664

commit d51e12077dd2f577a9b43ff8f331f51c0a274664
Author: mmoss <mmoss@google.com>
Date: Wed Nov 09 05:07:16 2016

The latest win64-pgo build properly identified the previous 64-bit release:

[11:40:00] Channel canary: Version: 56.0.2914.0

which matches what omahaproxy shows for win64, whereas the previous win canary shows 56.0.2913.0.

For some reason chrome#56.0.2914.3 has failed to generate the 64-bit PGO diff (56.0.2906.0 --> 56.0.2914.3) and we did it manually. Is something still fishy? or we fell under some exception bucket again?

Thank you!

The build definitely detected all the previous versions properly:

https://uberchromegw.corp.google.com/i/official.desktop/builders/win64-pgo/builds/479/steps/official%20signing/logs/stdio

but it only chose to create a diff against the previous canary (56.0.2914.1). I'm not that familiar with the background of diff creation, so I don't know if that logic is intentional or not.

Hmm. do we know someone else who can take a look?

Oh, well, I can look at the code (the file changed in #8) and figure out the logic it's using, I just don't necessarily know if that logic fits expectations, or who defines those expectations (+kerz). But, for instance, there's this line and comment:

  break  # We don't want to match more than one channel.

which sounds a lot like it's WAI, because it matched the canary channel, so it shouldn't try to diff anything else.

Why would we diff from a Canary to a Dev?

Reg c#14: Since we are picking latest Canary as a qualifying build for Chrome Dev releases.