This PFQ master run:
https://uberchromegw.corp.google.com/i/chromeos/builders/master-chromium-pfq/builds/2853

Killed one of the slaves after 4.5 hours:

MasterSlaveSyncCompletion:
  start:    0:47:02 median 0:50:02 mean 0:33:37 min 0:12:27 max 0:53:21
  duration: 3:42:21 median 2:34:02 mean 1:36:16 min 0:12:09 max 3:09:49
  finish:   4:29:23 median 3:25:01 mean 2:09:54 min 0:24:37 max 4:00:50

The slave:
https://uberchromegw.corp.google.com/i/chromeos/builders/x86-alex-chrome-pfq/builds/551

Started 40 minutes after the master (reason unknown), then took 51 minutes to complete the SyncChrome stage instead of the usual 9:

SyncChrome:
  start:    0:25:26 median 0:13:35 mean 0:14:19 min 0:11:41 max 0:25:26
  duration: 0:51:02 median 0:05:52 mean 0:08:38 min 0:05:03 max 0:51:02
  finish:   1:16:28 median 0:19:38 mean 0:22:58 min 0:16:44 max 1:16:28

(Looking at the SyncChrome log there is no obvious cause except likely overall system load)

The combined additional 80 minutes caused the slave to be killed before it completed.

I would like to suggest that we are too aggressive in killing slaves. There have been other similar cases where a non fatal load related slowness has caused PFQ slaves to be killed prematurely.

Padding the maximum timeout would likely improve overall reliability with the following caveats:
* We need to ensure that aborting a hung slave will cause the master to complete.
* We need to identify slow builders (i.e. total time > 120% of the median). This would be useful anyway to help identify regressions in the time taken for various stages. Builders that were unusually slow should be reported in the master.