It is not the paygen timeout. It is the build timeout, which killed the Paygen Stage. Check the bottom of quawks-release build, the build elapsed 7 hrs, 44 mins, 23 secs. And the BuildPackage Stage took 5 hrs, 22 mins, 1 secs.

I checked a green build for quawks-release, the BuildPackage Stage only took 3 hrs, 8 mins, 57 secs, which is almost 2 and half hours shorter.

So, the question is why the BuildPackage stage took so long for this run?

Don, Is this related to new GCE instances change?

The failure in the -294 build on quawks looks a lot like  bug 610536 .

I discovered that the logs for build_package now include the output
from build processes every 60 minutes or so.  That means it's possible
to see output snapshots from the longest running builds.  Most interesting
(to me) was the output from the gdb build.  That showed the system spent
more than an hour in "configure"(!).  The configure script shouldn't
really take that long.  At all.

If I had to guess at this point, I'd say the builder system is overloaded,
possibly from memory pressure from high levels of parallelism.

We have a problem with BuildPackages with AFDO enabled, they are randomly running super late.