Based on the stacktrace:

Metric "findit/try-job-errors" was given field "type" with invalid value "8" (<type 'unicode'>).
Traceback (most recent call last):
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1535, in __call__
    rv = self.handle_exception(request, response, e)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1529, in __call__
    rv = self.router.dispatch(request, response)
  File "/base/data/home/apps/s~findit-for-me/waterfall-backend:826d19430.399701639621221937/first_party/gae_ts_mon/config.py", line 254, in dispatch
    time_fn=time_fn)
  File "/base/data/home/apps/s~findit-for-me/waterfall-backend:826d19430.399701639621221937/first_party/gae_ts_mon/config.py", line 216, in _instrumented_dispatcher
    ret = dispatcher(request, response)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
    return route.handler_adapter(request, response)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1102, in __call__
    return handler.dispatch()
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 572, in dispatch
    return self.handle_exception(e, self.app.debug)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 570, in dispatch
    return method(*args, **kwargs)
  File "/base/data/home/apps/s~findit-for-me/waterfall-backend:826d19430.399701639621221937/first_party/pipeline_utils/appengine_third_party_pipeline_python_src_pipeline/pipeline.py", line 2786, in get
    self.run_callback()
  File "/base/data/home/apps/s~findit-for-me/waterfall-backend:826d19430.399701639621221937/first_party/pipeline_utils/appengine_third_party_pipeline_python_src_pipeline/pipeline.py", line 2863, in run_callback
    callback_result = perform_callback()
  File "/base/data/home/apps/s~findit-for-me/waterfall-backend:826d19430.399701639621221937/first_party/pipeline_utils/appengine_third_party_pipeline_python_src_pipeline/pipeline.py", line 2853, in perform_callback
    return stage._callback_internal(kwargs)
  File "/base/data/home/apps/s~findit-for-me/waterfall-backend:826d19430.399701639621221937/first_party/pipeline_utils/appengine_third_party_pipeline_python_src_pipeline/pipeline.py", line 1099, in _callback_internal
    return self.callback(**kwargs)
  File "/base/data/home/apps/s~findit-for-me/waterfall-backend:826d19430.399701639621221937/waterfall/monitor_try_job_pipeline.py", line 387, in callback
    build, error, False)
  File "/base/data/home/apps/s~findit-for-me/waterfall-backend:826d19430.399701639621221937/waterfall/monitor_try_job_pipeline.py", line 154, in _UpdateTryJobMetadata
    try_job_data.builder_name)
  File "/base/data/home/apps/s~findit-for-me/waterfall-backend:826d19430.399701639621221937/waterfall/monitor_try_job_pipeline.py", line 120, in _OnTryJobError
    'builder_name': builder_name
  File "/base/data/home/apps/s~findit-for-me/waterfall-backend:826d19430.399701639621221937/first_party/gae_ts_mon/common/metrics.py", line 404, in increment
    self._incr(fields, target_fields, 1)
  File "/base/data/home/apps/s~findit-for-me/waterfall-backend:826d19430.399701639621221937/first_party/gae_ts_mon/common/metrics.py", line 354, in _incr
    self.name, self._validate_fields(fields), target_fields,
  File "/base/data/home/apps/s~findit-for-me/waterfall-backend:826d19430.399701639621221937/first_party/gae_ts_mon/common/metrics.py", line 281, in _validate_fields
    spec.validate_value(self.name, fields[spec.name])
  File "/base/data/home/apps/s~findit-for-me/waterfall-backend:826d19430.399701639621221937/first_party/gae_ts_mon/common/metrics.py", line 41, in validate_value
    metric_name, self.name, value)
MonitoringInvalidFieldTypeError: Metric "findit/try-job-errors" was given field "type" with invalid value "8" (<type 'unicode'>).

try_job_type is an int, which here seems to have been converted into a string at some point when serializing.

Need to coerce int parameters back into int.

This may also be the root cause for the Nonetype exceptions when trying to read try job data, for example when checking

if try_job_type == failure_type.FLAKY_TEST, "18" is being compared to 18 which fails and the wrong try job data entity is read.

Over to robertocn@, as he's more familiar with the callback code.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/3b3682b765e85bd667d4d52c77acbb4c0b4283bd

commit 3b3682b765e85bd667d4d52c77acbb4c0b4283bd
Author: Roberto Carrillo <robertocn@google.com>
Date: Sat Mar 11 02:06:48 2017

[findit] Cast pipeline callback parameters

When the callback is stored as a url, the parameters are converted to
strings. This change will make sure that they are cast back to their
original types.

R=lijeffrey
BUG= 700498 

Change-Id: Iedb397d1bf28dabcef6cb07710705367f53312e7
Reviewed-on: https://chromium-review.googlesource.com/453271
Reviewed-by: Jeffrey Li <lijeffrey@chromium.org>
Commit-Queue: Jeffrey Li <lijeffrey@chromium.org>

[modify] https://crrev.com/3b3682b765e85bd667d4d52c77acbb4c0b4283bd/appengine/findit/waterfall/monitor_try_job_pipeline.py

https://findit-for-me.appspot.com/waterfall/check-flake?key=ag9zfmZpbmRpdC1mb3ItbWVy0gELEhdNYXN0ZXJGbGFrZUFuYWx5c2lzUm9vdCKbAWNocm9taXVtLmxpbnV4L0FuZHJvaWQgVGVzdHMvMzkxNjEvY2NfdW5pdHRlc3RzIG9uIEFuZHJvaWQvVkdWNGRIVnlaVXhoZVdWeVEyaGhibWRsU1c1MmFYTnBZbXhsVFdGcGJHSnZlRlJsYzNRdVVuVnVUWFZzZEdsVWFISmxZV1JmUkdWc1pXZGhkR2x1WjFKbGJtUmxjbVZ5DAsSE01hc3RlckZsYWtlQW5hbHlzaXMYAQw

Try jobs are back! I think this is fixed

This is cool, but the link above reveals a bug -- Swarming runs into errors without running the tests, and Findit determines that the test doesn't exist.

@lijeffrey, mind investigating more and filing a separate bug?

 issue 700980  logged to track Findit's failure to detect the error in the swarming task and misinterpreting it as a nonexistent test

Closing since the underlying issue now has a separate bug.