SQLite's expected distribution is the amalgamation, a thing where a TCL script takes all the necessary source code and generates a giant sqlite3.c file, currently almost 7MB in size.  There are various reasons for this, which I will consider irrelevant to us (if SQLite wants to work that way, that's fine).  https://www.sqlite.org/amalgamation.html

This has an impact on landing an import, though, because our infrastructure cannot handle a 7MB source file at all, and even if it could it's plausible that having thousands of patches to it during an import would break something.  Likewise, a SQLite import can be expected to have binary changes, which the our infrastructure cares not for ( issue 579778 ).  Also, simply posting the complete CL takes a very long time, which is unfortunate when you're trying to figure out if you're going to break the build.

Attached is a Perl script I wrote to let me run things against trybots.  You run it in a clean branch.  It takes another branch as parameter, representing the branch which you want to pull things from.  It patches over the sqlite/BUILD.gn changes and sqlite/amalgamation/sqlite3.h changes.  Then it bursts the sqlite3.c file out into .NN.c files each a bit over 880k, segmented at logical boundaries, with an #include chain.  Then it patches BUILD.gn to build sqlite3.00.c instead of sqlite3.c.

I agree, this is insane.  The link above references "The Split Amalgamation", which sounds like it's the same basic idea as this, except different.  Maybe it would make sense to leverage that going forward.  One thing to note is that our infrastructure breaks down around 1MB, which is where the 880k above came from.  "The Split Amalgamation" sounds like it might have fewer files, so might get closer to edge.

---

Q: Why Perl instead of Python?

A: I know Perl really well, and Python hardly at all.

Q: ???

A: Yeah, I know.

Q: How do you land changes, then?

A: You land manually using git cl land from the real branch (don't land the trybot hack!), maybe with --bypass-hooks.  In fact, since you need a stable origin/master for this, so you need something more like `git fetch origin && git rebase origin/master && git cl land --bypass-hooks`, which you maybe have to run four or five times.  Or just do it early or late MTV time.

Q: Can I ask you questions about this?

A: Sure, but I don't think it's much more complicated than it sounds, nor that it can be made much simpler.
 

Deleted: split.pl 2.7 KB

split.pl 2.7 KB View Download