Upon further investigation, job-state can stay in 'processing' forever.  I need to examine 'job-state-reasons' to find out why it hasn't printed yet.  Unfortunately, this isn't included in the GetJobs2 call >:( 

Performing a survey of observed failure modes: go/printer-error-matrix

It looks like timing out network requests and checking for jobs that complete without printing any pages covers most errors.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d440b845e1f0f93c6637cb86994ec69faf60cba6

commit d440b845e1f0f93c6637cb86994ec69faf60cba6
Author: skau <skau@chromium.org>
Date: Wed Feb 15 17:29:29 2017

Handle retained printing history.

In order to accurately report errors, CUPS on Chrome OS is going to
retain history.  Previously, the notification code assumed that
completed jobs were removed from history but this prevents the accurate
reporting of jobs that failed due to errors.

BUG= 684853 

Review-Url: https://codereview.chromium.org/2694123002
Cr-Commit-Position: refs/heads/master@{#450738}

[modify] https://crrev.com/d440b845e1f0f93c6637cb86994ec69faf60cba6/chrome/browser/chromeos/printing/cups_print_job_manager_impl.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3fadb49c64f97201f9a328052a11ad07d01e9cbd

commit 3fadb49c64f97201f9a328052a11ad07d01e9cbd
Author: skau <skau@chromium.org>
Date: Wed Feb 15 19:23:37 2017

Add job id to JobEventDetails.

Adding job id to JobEventDetails makes it easier to track jobs after
they've been queued.

BUG= 684853 

Review-Url: https://codereview.chromium.org/2692923006
Cr-Commit-Position: refs/heads/master@{#450766}

[modify] https://crrev.com/3fadb49c64f97201f9a328052a11ad07d01e9cbd/chrome/browser/printing/print_job.cc
[modify] https://crrev.com/3fadb49c64f97201f9a328052a11ad07d01e9cbd/chrome/browser/printing/print_job.h
[modify] https://crrev.com/3fadb49c64f97201f9a328052a11ad07d01e9cbd/chrome/browser/printing/print_job_worker.cc
[modify] https://crrev.com/3fadb49c64f97201f9a328052a11ad07d01e9cbd/printing/printing_context.h
[modify] https://crrev.com/3fadb49c64f97201f9a328052a11ad07d01e9cbd/printing/printing_context_chromeos.h

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5f12e1b60b6deec1caafc5a361e7950345f39d18

commit 5f12e1b60b6deec1caafc5a361e7950345f39d18
Author: skau <skau@chromium.org>
Date: Fri Feb 17 22:38:45 2017

Show notifications for waiting jobs.

If more than one job is queued for a printer, the inactive jobs will
be shown in the paused state.

BUG= 684853 

Review-Url: https://codereview.chromium.org/2696763002
Cr-Commit-Position: refs/heads/master@{#451388}

[modify] https://crrev.com/5f12e1b60b6deec1caafc5a361e7950345f39d18/chrome/browser/chromeos/printing/cups_print_job_manager_impl.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/3d1b3bc9501e79c9f13299a67597531bc3f8e27e

commit 3d1b3bc9501e79c9f13299a67597531bc3f8e27e
Author: Sean Kau <skau@chromium.org>
Date: Tue Feb 21 22:07:45 2017

net-print/cups: Enable job history for CUPS

In order to accurately report status for completed jobs, we need to
retain job history for a period of time.  History is still cleaned up
on logout when the CUPS directories are removed.

BUG= chromium:684853 
TEST=Run 'lpstat -t' after a print job is complete and note that it is
still listed.

Change-Id: Id4931cabeff7efcaad9944040a3fbb366aaf75b0
Reviewed-on: https://chromium-review.googlesource.com/441990
Commit-Ready: Sean Kau <skau@chromium.org>
Tested-by: Sean Kau <skau@chromium.org>
Reviewed-by: Sean Kau <skau@chromium.org>

[rename] https://crrev.com/3d1b3bc9501e79c9f13299a67597531bc3f8e27e/net-print/cups/cups-2.1.4-r12.ebuild
[modify] https://crrev.com/3d1b3bc9501e79c9f13299a67597531bc3f8e27e/net-print/cups/files/cupsd.conf

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6157021217dd1922d6f5dcc8fd00b6e36bbb3a32

commit 6157021217dd1922d6f5dcc8fd00b6e36bbb3a32
Author: skau <skau@chromium.org>
Date: Wed Feb 22 23:51:45 2017

Use job id from JobEventDetails to create CupsPrintJobs.

By using job id from JobEventDetails, we can avoid querying CUPS to
display the initial notification and we don't have to guess if we've
identified the correct job.  This wasn't done originally, because job id
was not present in JobEventDetails.

Additionally, add some rate limiting for queries and check for jobs which
we lose track of.

Speculatively, this fixes 682853 as cupsd can become overwhelmed and stop responding to queries.

BUG= 684853 , 682853 

Review-Url: https://codereview.chromium.org/2699273002
Cr-Commit-Position: refs/heads/master@{#452281}

[modify] https://crrev.com/6157021217dd1922d6f5dcc8fd00b6e36bbb3a32/chrome/browser/chromeos/printing/cups_print_job_manager_impl.cc
[modify] https://crrev.com/6157021217dd1922d6f5dcc8fd00b6e36bbb3a32/chrome/browser/chromeos/printing/cups_print_job_manager_impl.h

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b5d49127cc30716341b904fd9cfc6e26e68b6d5d

commit b5d49127cc30716341b904fd9cfc6e26e68b6d5d
Author: skau <skau@chromium.org>
Date: Sat Feb 25 00:48:24 2017

Report print jobs that are cancelled by CUPS as errors.

If a print job encounters a problem, it can sometimes be cancelled by
CUPS or the printer.  We will now notify the user of an error in this
situation.  If a user cancels a print job, the behavior remains the same.

BUG= 682853 , 684853 

Review-Url: https://codereview.chromium.org/2708233006
Cr-Commit-Position: refs/heads/master@{#453025}

[modify] https://crrev.com/b5d49127cc30716341b904fd9cfc6e26e68b6d5d/chrome/browser/chromeos/printing/cups_print_job_notification.cc
[modify] https://crrev.com/b5d49127cc30716341b904fd9cfc6e26e68b6d5d/chrome/browser/chromeos/printing/cups_print_job_notification.h
[modify] https://crrev.com/b5d49127cc30716341b904fd9cfc6e26e68b6d5d/chrome/browser/chromeos/printing/cups_print_job_notification_manager.cc
[modify] https://crrev.com/b5d49127cc30716341b904fd9cfc6e26e68b6d5d/chrome/browser/chromeos/printing/cups_print_job_notification_manager.h

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a26877dfeb89300f2da958471d2ab9f72766cf38

commit a26877dfeb89300f2da958471d2ab9f72766cf38
Author: skau <skau@chromium.org>
Date: Sat Mar 11 00:12:29 2017

Implement IPP Get-Jobs and Get-Printer-Attributes requests.

CUPS provides cupsGetJobs2 but it doesn't provide the necessary fields
to report status accurately.  Notably, it doesn't provide
job-impressions-completed or printer-state-reasons both of which are
necessary to differentiate errors.

BUG= 684853 

Review-Url: https://codereview.chromium.org/2691093006
Cr-Commit-Position: refs/heads/master@{#456225}

[modify] https://crrev.com/a26877dfeb89300f2da958471d2ab9f72766cf38/chrome/browser/chromeos/printing/cups_print_job_manager_impl.cc
[modify] https://crrev.com/a26877dfeb89300f2da958471d2ab9f72766cf38/chrome/browser/chromeos/printing/cups_print_job_manager_impl.h
[modify] https://crrev.com/a26877dfeb89300f2da958471d2ab9f72766cf38/printing/BUILD.gn
[modify] https://crrev.com/a26877dfeb89300f2da958471d2ab9f72766cf38/printing/backend/cups_connection.cc
[modify] https://crrev.com/a26877dfeb89300f2da958471d2ab9f72766cf38/printing/backend/cups_connection.h
[modify] https://crrev.com/a26877dfeb89300f2da958471d2ab9f72766cf38/printing/backend/cups_ipp_util.h
[add] https://crrev.com/a26877dfeb89300f2da958471d2ab9f72766cf38/printing/backend/cups_jobs.cc
[add] https://crrev.com/a26877dfeb89300f2da958471d2ab9f72766cf38/printing/backend/cups_jobs.h

One last CL to clean up the print queues then we're done.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/def538a8cd9e4d982d150d4308e44a2aa4c504ff

commit def538a8cd9e4d982d150d4308e44a2aa4c504ff
Author: skau <skau@chromium.org>
Date: Fri Mar 17 00:37:01 2017

Cleanup jobs which have failed.

We're stricter about declaring failed jobs than CUPS.  Be sure to remove
the jobs from the print queue when we've given up on them.  Also, ensure
that we don't notify observers after cancellation has been requested.

TEST=Start job to unreachable printer, let it time out.  Start job to valid printer, click cancel.  Run 'lpstat -t', verify that no print jobs are pending.
BUG= 684853 

Review-Url: https://codereview.chromium.org/2748253005
Cr-Commit-Position: refs/heads/master@{#457627}

[modify] https://crrev.com/def538a8cd9e4d982d150d4308e44a2aa4c504ff/chrome/browser/chromeos/printing/cups_print_job.h
[modify] https://crrev.com/def538a8cd9e4d982d150d4308e44a2aa4c504ff/chrome/browser/chromeos/printing/cups_print_job_manager.h
[modify] https://crrev.com/def538a8cd9e4d982d150d4308e44a2aa4c504ff/chrome/browser/chromeos/printing/cups_print_job_manager_impl.cc
[modify] https://crrev.com/def538a8cd9e4d982d150d4308e44a2aa4c504ff/chrome/browser/chromeos/printing/cups_print_job_manager_impl.h
[modify] https://crrev.com/def538a8cd9e4d982d150d4308e44a2aa4c504ff/chrome/browser/chromeos/printing/cups_print_job_notification.cc
[modify] https://crrev.com/def538a8cd9e4d982d150d4308e44a2aa4c504ff/chrome/browser/chromeos/printing/fake_cups_print_job_manager.cc
[modify] https://crrev.com/def538a8cd9e4d982d150d4308e44a2aa4c504ff/chrome/browser/chromeos/printing/fake_cups_print_job_manager.h