New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 801345 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Closed: Jan 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

[Findit] Flaky test - CITestFailureTest.testUpdateFirstFailureOnTestLevelThenUpdateStepLevel

Project Member Reported by lijeffrey@chromium.org, Jan 11 2018

Issue description

======================================================================
ERROR: services.test_failure.test.ci_test_failure_test.CITestFailureTest.testUpdateFirstFailureOnTestLevelThenUpdateStepLevel (/usr/local/google/home/lijeffrey/infra/appengine/findit/services/test_failure/test/ci_test_failure_test.expected/CITestFailureTest.testUpdateFirstFailureOnTestLevelThenUpdateStepLevel.json)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/lijeffrey/infra/ENV/local/lib/python2.7/site-packages/expect_tests/pipeline.py", line 398, in process_test
    subresult = subtest.run()
  File "/usr/local/google/home/lijeffrey/infra/ENV/local/lib/python2.7/site-packages/expect_tests/type_definitions.py", line 253, in run
    return self.func_call(context=context)
  File "/usr/local/google/home/lijeffrey/infra/ENV/local/lib/python2.7/site-packages/expect_tests/type_definitions.py", line 168, in __call__
    return f.func(*f.args, **f.kwargs)
  File "/usr/local/google/home/lijeffrey/infra/ENV/local/lib/python2.7/site-packages/expect_tests/unittest_helper.py", line 40, in _RunTestCaseSingle
    return Result(test_method())
  File "/usr/local/google/home/lijeffrey/infra/appengine/findit/services/test_failure/test/ci_test_failure_test.py", line 344, in testUpdateFirstFailureOnTestLevelThenUpdateStepLevel
    self.assertEqual(expected_failed_step, failed_step.ToSerializable())
  File "/usr/lib/python2.7/unittest/case.py", line 515, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib/python2.7/unittest/case.py", line 837, in assertDictEqual
    self.fail(self._formatMessage(msg, standardMsg))
  File "/usr/lib/python2.7/unittest/case.py", line 412, in fail
    raise self.failureException(msg)
AssertionError: {'last_pass': 220, 'tests': {'Unittest2.Subtest1': {'last_pass': 221, 'current_f [truncated]... != {'last_pass': 221, 'tests': {'Unittest2.Subtest1': {'last_pass': 221, 'current_f [truncated]...
  {'current_failure': 224,
-  'first_failure': 221,
?                     ^

+  'first_failure': 222,
?                     ^

-  'last_pass': 220,
?                 ^

+  'last_pass': 221,
?                 ^

   'list_isolated_data': None,
   'tests': {'Unittest2.Subtest1': {'base_test_name': 'Unittest2.Subtest1',
                                    'current_failure': 224,
                                    'first_failure': 222,
                                    'last_pass': 221},
             'Unittest3.Subtest2': {'base_test_name': 'Unittest3.Subtest2',
                                    'current_failure': 224,
-                                   'first_failure': 221,
?                                                      ^

+                                   'first_failure': 222,
?                                                      ^

-                                   'last_pass': None}}}
?                                                ^^^^

+                                   'last_pass': 221}}}
?                                                ^^^


==== captured logging output ====
LOG:
WARNING: Assigned None to FailedTest.last_pass as it is missing in {'current_failure': 224, 'first_failure': 223, 'base_test_name': 'Unittest3.Subtest2'}
WARNING: Assigned None to TestFailedStep.list_isolated_data as it is missing in {'last_pass': 220, 'tests': {'Unittest2.Subtest1': {'last_pass': 223, 'current_failure': 224, 'first_failure': 223, 'base_test_name': 'Unittest2.Subtest1'}, 'Unittest3.Subtest2': {'current_failure': 224, 'first_failure': 223, 'base_test_name': 'Unittest3.Subtest2'}}, 'current_failure': 224, 'first_failure': 221}
ERROR: log_data log of step m/b/223/abc_test is not json loadable.
INFO: got response status 200 for url https://chromium-swarm.appspot.com/_ah/api/swarming/v1/tasks/list?tags=master%3Am&tags=buildername%3Ab&tags=buildnumber%3A222&tags=stepname%3Aabc_test
INFO: got response status 200 for url https://chromium-swarm.appspot.com/_ah/api/swarming/v1/tasks/list?tags=master%3Am&tags=buildername%3Ab&tags=buildnumber%3A222&tags=stepname%3Aabc_test&cursor=thisisacursor
INFO: got response status 200 for url https://isolateserver.appspot.com/_ah/api/isolateservice/v1/retrieve
INFO: got response status 200 for url https://isolateserver.appspot.com/_ah/api/isolateservice/v1/retrieve
INFO: got response status 200 for url https://isolateserver.storage.googleapis.com/default-gzip/m_b_222_abc_test



It looks like this test may be sending real requests to real URLs which should be mocked
 

Comment 1 by chanli@chromium.org, Jan 22 2018

Status: Started (was: Available)

Comment 2 by chanli@chromium.org, Jan 22 2018

Owner: chanli@chromium.org

Comment 3 by chanli@chromium.org, Jan 22 2018

Cc: charliea@chromium.org chanli@chromium.org
 Issue 804434  has been merged into this issue.
Project Member

Comment 4 by bugdroid1@chromium.org, Jan 23 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/b55caf3e8e70213e11d318d08604e92ef7b33305

commit b55caf3e8e70213e11d318d08604e92ef7b33305
Author: Chan <chanli@chromium.org>
Date: Tue Jan 23 00:49:59 2018

[Findit] speculative fix flaky test CITestFailureTest.testUpdateFirstFailureOnTestLevelThenUpdateStepLevel

Use another way to mock functions to make sure no real http requests are sent when running this test

Bug:  801345 
Change-Id: Ie31adb836962f18a165fb3d953c7b1e1f542684b
Reviewed-on: https://chromium-review.googlesource.com/879130
Reviewed-by: Roberto Carrillo <robertocn@chromium.org>
Reviewed-by: Jeffrey Li <lijeffrey@chromium.org>
Commit-Queue: Chan Li <chanli@chromium.org>

[modify] https://crrev.com/b55caf3e8e70213e11d318d08604e92ef7b33305/appengine/findit/waterfall/swarming_util.py
[modify] https://crrev.com/b55caf3e8e70213e11d318d08604e92ef7b33305/appengine/findit/services/test_failure/test/ci_test_failure_test.py
[delete] https://crrev.com/5f95add9a16b090c414c4e2e98d133b14bcebf4b/appengine/findit/services/test/data/m_b_221_abc_test.json

Comment 5 by chanli@chromium.org, Jan 24 2018

Status: Fixed (was: Started)

Sign in to add a comment