[Findit] Flaky test - CITestFailureTest.testUpdateFirstFailureOnTestLevelThenUpdateStepLevel |
|||
Issue description
======================================================================
ERROR: services.test_failure.test.ci_test_failure_test.CITestFailureTest.testUpdateFirstFailureOnTestLevelThenUpdateStepLevel (/usr/local/google/home/lijeffrey/infra/appengine/findit/services/test_failure/test/ci_test_failure_test.expected/CITestFailureTest.testUpdateFirstFailureOnTestLevelThenUpdateStepLevel.json)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/google/home/lijeffrey/infra/ENV/local/lib/python2.7/site-packages/expect_tests/pipeline.py", line 398, in process_test
subresult = subtest.run()
File "/usr/local/google/home/lijeffrey/infra/ENV/local/lib/python2.7/site-packages/expect_tests/type_definitions.py", line 253, in run
return self.func_call(context=context)
File "/usr/local/google/home/lijeffrey/infra/ENV/local/lib/python2.7/site-packages/expect_tests/type_definitions.py", line 168, in __call__
return f.func(*f.args, **f.kwargs)
File "/usr/local/google/home/lijeffrey/infra/ENV/local/lib/python2.7/site-packages/expect_tests/unittest_helper.py", line 40, in _RunTestCaseSingle
return Result(test_method())
File "/usr/local/google/home/lijeffrey/infra/appengine/findit/services/test_failure/test/ci_test_failure_test.py", line 344, in testUpdateFirstFailureOnTestLevelThenUpdateStepLevel
self.assertEqual(expected_failed_step, failed_step.ToSerializable())
File "/usr/lib/python2.7/unittest/case.py", line 515, in assertEqual
assertion_func(first, second, msg=msg)
File "/usr/lib/python2.7/unittest/case.py", line 837, in assertDictEqual
self.fail(self._formatMessage(msg, standardMsg))
File "/usr/lib/python2.7/unittest/case.py", line 412, in fail
raise self.failureException(msg)
AssertionError: {'last_pass': 220, 'tests': {'Unittest2.Subtest1': {'last_pass': 221, 'current_f [truncated]... != {'last_pass': 221, 'tests': {'Unittest2.Subtest1': {'last_pass': 221, 'current_f [truncated]...
{'current_failure': 224,
- 'first_failure': 221,
? ^
+ 'first_failure': 222,
? ^
- 'last_pass': 220,
? ^
+ 'last_pass': 221,
? ^
'list_isolated_data': None,
'tests': {'Unittest2.Subtest1': {'base_test_name': 'Unittest2.Subtest1',
'current_failure': 224,
'first_failure': 222,
'last_pass': 221},
'Unittest3.Subtest2': {'base_test_name': 'Unittest3.Subtest2',
'current_failure': 224,
- 'first_failure': 221,
? ^
+ 'first_failure': 222,
? ^
- 'last_pass': None}}}
? ^^^^
+ 'last_pass': 221}}}
? ^^^
==== captured logging output ====
LOG:
WARNING: Assigned None to FailedTest.last_pass as it is missing in {'current_failure': 224, 'first_failure': 223, 'base_test_name': 'Unittest3.Subtest2'}
WARNING: Assigned None to TestFailedStep.list_isolated_data as it is missing in {'last_pass': 220, 'tests': {'Unittest2.Subtest1': {'last_pass': 223, 'current_failure': 224, 'first_failure': 223, 'base_test_name': 'Unittest2.Subtest1'}, 'Unittest3.Subtest2': {'current_failure': 224, 'first_failure': 223, 'base_test_name': 'Unittest3.Subtest2'}}, 'current_failure': 224, 'first_failure': 221}
ERROR: log_data log of step m/b/223/abc_test is not json loadable.
INFO: got response status 200 for url https://chromium-swarm.appspot.com/_ah/api/swarming/v1/tasks/list?tags=master%3Am&tags=buildername%3Ab&tags=buildnumber%3A222&tags=stepname%3Aabc_test
INFO: got response status 200 for url https://chromium-swarm.appspot.com/_ah/api/swarming/v1/tasks/list?tags=master%3Am&tags=buildername%3Ab&tags=buildnumber%3A222&tags=stepname%3Aabc_test&cursor=thisisacursor
INFO: got response status 200 for url https://isolateserver.appspot.com/_ah/api/isolateservice/v1/retrieve
INFO: got response status 200 for url https://isolateserver.appspot.com/_ah/api/isolateservice/v1/retrieve
INFO: got response status 200 for url https://isolateserver.storage.googleapis.com/default-gzip/m_b_222_abc_test
It looks like this test may be sending real requests to real URLs which should be mocked
,
Jan 22 2018
,
Jan 22 2018
,
Jan 23 2018
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/b55caf3e8e70213e11d318d08604e92ef7b33305 commit b55caf3e8e70213e11d318d08604e92ef7b33305 Author: Chan <chanli@chromium.org> Date: Tue Jan 23 00:49:59 2018 [Findit] speculative fix flaky test CITestFailureTest.testUpdateFirstFailureOnTestLevelThenUpdateStepLevel Use another way to mock functions to make sure no real http requests are sent when running this test Bug: 801345 Change-Id: Ie31adb836962f18a165fb3d953c7b1e1f542684b Reviewed-on: https://chromium-review.googlesource.com/879130 Reviewed-by: Roberto Carrillo <robertocn@chromium.org> Reviewed-by: Jeffrey Li <lijeffrey@chromium.org> Commit-Queue: Chan Li <chanli@chromium.org> [modify] https://crrev.com/b55caf3e8e70213e11d318d08604e92ef7b33305/appengine/findit/waterfall/swarming_util.py [modify] https://crrev.com/b55caf3e8e70213e11d318d08604e92ef7b33305/appengine/findit/services/test_failure/test/ci_test_failure_test.py [delete] https://crrev.com/5f95add9a16b090c414c4e2e98d133b14bcebf4b/appengine/findit/services/test/data/m_b_221_abc_test.json
,
Jan 24 2018
|
|||
►
Sign in to add a comment |
|||
Comment 1 by chanli@chromium.org
, Jan 22 2018