Example errors: 

======================================================================
ERROR: status.test.cq_status_test.CQStatusTestCase.test_create_tasks_to_update_stale_issues (/usr/local/google/home/sergeyberezin/code/infra/appengine/chromium_try_flakes/status/test/cq_status_test.expected/CQStatusTestCase.test_create_tasks_to_update_stale_issues.json)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/sergeyberezin/code/infra/ENV/local/lib/python2.7/site-packages/expect_tests/pipeline.py", line 397, in process_test
    subresult = subtest.run()
  File "/usr/local/google/home/sergeyberezin/code/infra/ENV/local/lib/python2.7/site-packages/expect_tests/type_definitions.py", line 240, in run
    return self.func_call(context=context)
  File "/usr/local/google/home/sergeyberezin/code/infra/ENV/local/lib/python2.7/site-packages/expect_tests/type_definitions.py", line 155, in __call__
    return f.func(*f.args, **f.kwargs)
  File "/usr/local/google/home/sergeyberezin/code/infra/ENV/local/lib/python2.7/site-packages/expect_tests/unittest_helper.py", line 30, in _RunTestCaseSingle
    return Result(getattr(test_instance, test_name)())
  File "/usr/local/google/home/sergeyberezin/code/infra/appengine/chromium_try_flakes/status/test/cq_status_test.py", line 354, in test_create_tasks_to_update_stale_issues
    self.assertEqual(len(tasks), 3)
  File "/usr/lib/python2.7/unittest/case.py", line 515, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib/python2.7/unittest/case.py", line 508, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: 0 != 3

==== captured logging output ====
LOG:
(empty)

STDOUT:
(empty)

STDERR:
(empty)


======================================================================
ERROR: status.test.cq_status_test.CQStatusTestCase.test_update_histograms (/usr/local/google/home/sergeyberezin/code/infra/appengine/chromium_try_flakes/status/test/cq_status_test.expected/CQStatusTestCase.test_update_histograms.json)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/sergeyberezin/code/infra/ENV/local/lib/python2.7/site-packages/expect_tests/pipeline.py", line 397, in process_test
    subresult = subtest.run()
  File "/usr/local/google/home/sergeyberezin/code/infra/ENV/local/lib/python2.7/site-packages/expect_tests/type_definitions.py", line 240, in run
    return self.func_call(context=context)
  File "/usr/local/google/home/sergeyberezin/code/infra/ENV/local/lib/python2.7/site-packages/expect_tests/type_definitions.py", line 155, in __call__
    return f.func(*f.args, **f.kwargs)
  File "/usr/local/google/home/sergeyberezin/code/infra/ENV/local/lib/python2.7/site-packages/expect_tests/unittest_helper.py", line 30, in _RunTestCaseSingle
    return Result(getattr(test_instance, test_name)())
  File "/usr/local/google/home/sergeyberezin/code/infra/appengine/chromium_try_flakes/status/test/cq_status_test.py", line 579, in test_update_histograms
    self.assertEqual({3: 1, 9: 1, 10: 1}, dist.buckets)
  File "/usr/lib/python2.7/unittest/case.py", line 515, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib/python2.7/unittest/case.py", line 508, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: {9: 1, 10: 1, 3: 1} != defaultdict(<type 'int'>, {})

==== captured logging output ====
LOG:
(empty)

STDOUT:
(empty)

STDERR:
(empty)


======================================================================
ERROR: handlers.test.flake_issues_test.CreateFlakyRunTestCase.test_get_flaky_run_reason (/usr/local/google/home/sergeyberezin/code/infra/appengine/chromium_try_flakes/handlers/test/flake_issues_test.expected/CreateFlakyRunTestCase.test_get_flaky_run_reason.json)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/sergeyberezin/code/infra/ENV/local/lib/python2.7/site-packages/expect_tests/pipeline.py", line 397, in process_test
    subresult = subtest.run()
  File "/usr/local/google/home/sergeyberezin/code/infra/ENV/local/lib/python2.7/site-packages/expect_tests/type_definitions.py", line 240, in run
    return self.func_call(context=context)
  File "/usr/local/google/home/sergeyberezin/code/infra/ENV/local/lib/python2.7/site-packages/expect_tests/type_definitions.py", line 155, in __call__
    return f.func(*f.args, **f.kwargs)
  File "/usr/local/google/home/sergeyberezin/code/infra/ENV/local/lib/python2.7/site-packages/expect_tests/unittest_helper.py", line 30, in _RunTestCaseSingle
    return Result(getattr(test_instance, test_name)())
  File "/usr/local/google/home/sergeyberezin/code/infra/appengine/chromium_try_flakes/handlers/test/flake_issues_test.py", line 702, in test_get_flaky_run_reason
    self.assertEqual(len(flaky_runs), 1)
  File "/usr/lib/python2.7/unittest/case.py", line 515, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib/python2.7/unittest/case.py", line 508, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: 0 != 1

==== captured logging output ====
LOG:
INFO: get_flaky_run_reason http://build.chromium.org/p/test.master/json/builders/test-builder/builds/100
ERROR: Failed to retrieve or parse JSON from http://test-results.appspot.com/testfile?builder=test-builder&name=full_results.json&master=test.master&testtype=foo8%20%28with%20patch%29&buildnumber=100
Traceback (most recent call last):
  File "/usr/local/google/home/sergeyberezin/code/infra/appengine/chromium_try_flakes/handlers/flake_issues.py", line 460, in get_flakes
    result = urlfetch.fetch(url)
  File "/usr/local/google/home/sergeyberezin/code/infra/ENV/local/lib/python2.7/site-packages/mock.py", line 974, in __call__
    return _mock_self._mock_call(*args, **kwargs)
  File "/usr/local/google/home/sergeyberezin/code/infra/ENV/local/lib/python2.7/site-packages/mock.py", line 1034, in _mock_call
    raise result
Exception

STDOUT:
(empty)

STDERR:
(empty)

These FP flakes show up on https://codereview.chromium.org/1797103003/#ps140001

E.g. https://build.chromium.org/p/tryserver.infra/builders/Infra%20Linux%20Precise%2032%20Tester/builds/1395
https://build.chromium.org/p/tryserver.infra/builders/Infra%20Linux%20Trusty%2064%20Tester/builds/1353

Thanks for reporting this, I'll have  a look.

I wasn't able to reproduce any flakes on TotT after 50+ runs of all chromium-try-flakes tests, but I looked at errors from comment #1:

status.test.cq_status_test.CQStatusTestCase.test_create_tasks_to_update_stale_issues
  Not sure how can this happen, the logic is trivial. Perhaps dev_appserver.py is flaky on datastore operations?

status.test.cq_status_test.CQStatusTestCase.test_update_histograms 
  Again, looks correct to me unless there is a flaky behavior in ts_mon.Distribution. Dave, can you PTAL?

handlers.test.flake_issues_test.CreateFlakyRunTestCase.test_get_flaky_run_reason
  I've checked the logic tested here and don't see any obvious flakiness sources.
  The log contains an error, but this is expected and shouldn't fail the test.

Without a reliable way to reproduce this (fail at least every 50 times), I'm not sure what I can do about it. If it flakes frequently on someone's machine, please let me know and perhaps I can debug it remotely. Also marking this as low priority.

A sure way to reproduce it is to patch in https://codereview.chromium.org/1797103003/#ps140001 . It's possible that I introduced something flaky in that CL, and I will also look into it once I get a chance. All the other apps behave well though. So my hunch is that there might be some source of flakiness in these tests that the patch uncovered.

That actually looks like a reliable failure to me rather than a flake. It only fails on Linux it seems. I don't really understand most of the changes in that code, but chromium-try-flakes uses ts_mon, so quite possibly some changes affect the app. I wonder if some changes make ts_mon throw an error, which is caught somewhere in the flakiness app and swallowed. That would prevent some statements from being executed while no error would be thrown.

Thanks for looking! Let me grab the bug for now, since it sounds like it could be a bug in my code. I'll return it back to you if I find otherwise.

And the winner is... AppEngine datastore testing stubs! Turns out, it's not thread-safe, and can flakily lose entities if the datastore is accessed from another thread, which is what ts_mon does. The change was in resetting more parameters in ts_mon.reset_for_unittest() method, leading to executing metric flush more often, which happens to access the datastore. Previously, it was also executed, but only by the very first test, reducing the probability of hitting a sensitive test.

http://crrev.com/1826133003 has a workaround.

Looks like this is fixed now, isn't it? Please reopen if I am wrong.