> If it's possible, we should verify that GN and GYP outputs equivalent 
> ninja files for all devices.

That is not generally possible; there are (intentionally) significant differences between GN and GYP due to the way flags propagate and the way we have tried to reduce the number of global flags.

It might be possible to verify the architecture/code-gen flags in most cases, though.

Can't we verify at least:
- The same files are included in the build.
- The same compiler flags (e.g. the same set of -D options, -W options, -f options, and tuning options like -march) are used for each file.
- The same linker flags are used.
- No significant output binary size difference.
- No significant performance difference.

in comment #2, hashimoto@ wrote:
> Can't we verify at least:
> - The same files are included in the build.

This is difficult if not impossible due to the 
use of source_sets() instead of static_libraries in many places.

> - The same compiler flags (e.g. the same set of -D options, -W options,
> -f options, and tuning options like -march) are used for each file.
> - The same linker flags are used.

The sets of defines (-D) will be different as I tried to explain above. The
others will hopefully match on most files, but I doubt they'll match for
everything across the board. Many of the -W flags will be different as
well because GN defaults to stricter flags.

> - No significant output binary size difference.
> - No significant performance difference.

Yes, these are doable and we've done them on the other platforms.

What we try to verify at least are the last two, and also that the list of executables
are the same (i.e., every executable that was built in a GYP build is also built
in a GN build), and we try to at least roughly compare the code generation flags
(what you call the -f options and the tuning options and the linker flags), and that
the test binaries run the same lists of tests.

> This is difficult if not impossible due to the use of source_sets() instead of static_libraries in many places.
Sorry, I still don't understand the logic.
Can't we, say, run "ninja -v -n -C out/Release chrome" to get the list of compiled C++ files and verify that the list (roughly) matches that of GYP?

> The sets of defines (-D) will be different
I understand the defines don't necessarily match, but can't we at least check USE/ENABLE flags like -DUSE_LIBJPEG_TURBO=1 and -DENABLE_SUPERVISED_USERS=1?
The same goes for other flags, they don't have to match, but I want to make sure there are no critical differences.

Unfortunately, many parts of Chrome OS code are not tested by unit_tests nor browser_tests.
(browser_tests w/ target_os="chromeos" runs on a Linux workstation, but not on a real Chrome OS devices: issue 371286)
So if we can detect problems just by inspecting ninja files of all devices, I think we should do that before resorting to manual tests.

> we try to at least roughly compare the code generation flags
That sounds nice!
It should catch issues like https://codereview.chromium.org/1807163002.

I was trying to make it clear that there's limits on what we can do and challenges, but yes, we can probably do those things. 

@stevenjb - I'm going to assign this to you, and you can decide if there's more to be done here before we ship a binary or drop GYP support or both. Feel free to delegate to anyone other than me :).

I don't think there is anything more we can really do here. In the process of fixing GN issues we have found more than one place where the GYP configuration was wrong. I think we have to rely on tests and Dev / Canary / Beta testing.