[Findit] Flake Analyzer - Failed swarming tasks are interpreted as nonexistent tests |
|||||||
Issue descriptionhttps://findit-for-me.appspot.com/waterfall/check-flake?key=ag9zfmZpbmRpdC1mb3ItbWVy0gELEhdNYXN0ZXJGbGFrZUFuYWx5c2lzUm9vdCKbAWNocm9taXVtLmxpbnV4L0FuZHJvaWQgVGVzdHMvMzkxNjEvY2NfdW5pdHRlc3RzIG9uIEFuZHJvaWQvVkdWNGRIVnlaVXhoZVdWeVEyaGhibWRsU1c1MmFYTnBZbXhsVFdGcGJHSnZlRlJsYzNRdVVuVnVUWFZzZEdsVWFISmxZV1JmUkdWc1pXZGhkR2x1WjFKbGJtUmxjbVZ5DAsSE01hc3RlckZsYWtlQW5hbHlzaXMYAQw The first data point suggests the test does not exist, however upon opening the swarming task link https://chromium-swarm.appspot.com/task?id=34d9e77870ffe210&refresh=10&show_raw=1 the task actually ended in error. Findit instead should have either bailed out and reported the error, or used a nearby neighbor to generate that data point.
,
Apr 17 2017
,
Apr 25 2017
Should we move this to the next release?
,
Apr 26 2017
Moved to the next release as discussed in the meeting yesterday
,
Apr 27 2017
It seems the change for this may be a bit more involved to get right, so moving out of the 5/2 release again to work on other more impactful bugs in the flake analyzer this week and resuming this one once I'm back in MTV.
,
May 19 2017
Another example the swarming task ran into an exception, causing Findit to think the test does not exist and thus suggested a false positive culprit https://findit-for-me.appspot.com/waterfall/flake?key=ag9zfmZpbmRpdC1mb3ItbWVy0QELEhdNYXN0ZXJGbGFrZUFuYWx5c2lzUm9vdCKaAWNocm9taXVtLmxpbnV4L0FuZHJvaWQgVGVzdHMgKGRiZykvNDI1ODcvbmV0X3VuaXR0ZXN0cyBvbiBBbmRyb2lkL1ZWSk1VbVZ4ZFdWemRFTnZiblJsZUhSQ2RXbHNaR1Z5VmpoVVpYTjBMbFk0U1c1UWNtOWpaWE56VTJoMWRHUnZkMjVYYVhSb1NIVnVaMUpsY1hWbGMzUT0MCxITTWFzdGVyRmxha2VBbmFseXNpcxgBDA
,
Aug 21 2017
,
Aug 21 2017
It may be possible to retrieve the test location to validate that the test indeed does not exist
,
Aug 22 2017
,
Aug 22 2017
I think this bug is meant to block bug 757517 instead.
,
Sep 5 2017
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/838e68e4d74a1794a2d36cbac341818b36afa662 commit 838e68e4d74a1794a2d36cbac341818b36afa662 Author: Jeffrey Li <lijeffrey@chromium.org> Date: Tue Sep 05 20:50:21 2017 [Findit] Flake Analyzer - Do not use exit codes when determining swarming task success When swarming returns exit_code, it is the underlying process' exit code which is arbitrarily chosen and should not be relied on when determining whether a test is stable, flaky, or some error occurred. In order to determine a test's outcome more reliably, the output_json must be parsed. 1. The task's status must be 'COMPLETED' 2. 'outputs_ref' Must be available in the swarming task result 3. The task's failure log must be available 4. 'per_iteration_data' Must be available and non-empty in the task's failure log. When all criteria are met can a task's outcome be determined. If any criteria is not met, assume there was an error in the task. Bug: 700980 Change-Id: I947cef9444a8a7605bdc0f9854ab02835c13c98f Reviewed-on: https://chromium-review.googlesource.com/644428 Reviewed-by: Chan Li <chanli@chromium.org> Reviewed-by: Brandon Wylie <wylieb@chromium.org> Commit-Queue: Jeffrey Li <lijeffrey@chromium.org> [modify] https://crrev.com/838e68e4d74a1794a2d36cbac341818b36afa662/appengine/findit/waterfall/swarming_util.py [modify] https://crrev.com/838e68e4d74a1794a2d36cbac341818b36afa662/appengine/findit/waterfall/flake/update_flake_analysis_data_points_pipeline.py [modify] https://crrev.com/838e68e4d74a1794a2d36cbac341818b36afa662/appengine/findit/waterfall/test/process_base_swarming_task_result_pipeline_test.py [modify] https://crrev.com/838e68e4d74a1794a2d36cbac341818b36afa662/appengine/findit/waterfall/process_base_swarming_task_result_pipeline.py [modify] https://crrev.com/838e68e4d74a1794a2d36cbac341818b36afa662/appengine/findit/waterfall/flake/test/update_flake_analysis_data_points_pipeline_test.py
,
Sep 5 2017
,
Sep 7 2017
|
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by lijeffrey@chromium.org
, Mar 13 2017