New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 836976 link

Starred by 2 users
Status: Verified
Owner:
lijeffrey@chromium.org
Closed: Aug 31
Cc:
█ robert...@chromium.org
chanli@chromium.org
lijeffrey@chromium.org
liaoyuke@chromium.org
st...@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

[Findit] Flake Analyzer - NextCommitPositionPipeline is suggesting to run a commit position that has already been analyzed

Project Member Reported by lijeffrey@chromium.org, Apr 25 2018 
https://findit-for-me.appspot.com/waterfall/flake?key=ag9zfmZpbmRpdC1mb3ItbWVypgELEhdNYXN0ZXJGbGFrZUFuYWx5c2lzUm9vdCJwY2hyb21pdW0ubWVtb3J5L0xpbnV4IE1TYW4gVGVzdHMvOTM5Mi9icm93c2VyX3Rlc3RzL1UyVnpjMmx2YmxKbGMzUnZjbVZVWlhOMExsSmxjM1J2Y21WUWFXNXVaV1JUWld4bFkzUmxaRlJoWWc9PQwLEhNNYXN0ZXJGbGFrZUFuYWx5c2lzGAEM

Generator pipelines.flake_failure.determine_approximate_pass_rate_pipeline.DetermineApproximatePassRatePipeline(*(u'ENCODED:boHH6HlAZEbA0jDg7NHrcEFIN',), **{'analysis_urlsafe_key': u'ag9zfmZpbmRpdC1mb3ItbWVypgELEhdNYXN0ZXJGbGFrZUFuYWx5c2lzUm9vdCJwY2hyb21pdW0ubWVtb3J5L0xpbnV4IE1TYW4gVGVzdHMvOTM5Mi9icm93c2VyX3Rlc3RzL1UyVnpjMmx2YmxKbGMzUnZjbVZVWlhOMExsSmxjM... (460 bytes))#f759bfa3881e4654852eab5a0ea5f9ae raised exception. AssertionError:  (/base/data/home/apps/s~findit-for-me/waterfall-backend:15111-6a36582.409252435710767791/infra_api_clients/../third_party/pipeline/pipeline.py:2434)
Traceback (most recent call last):
  File "/base/data/home/apps/s~findit-for-me/waterfall-backend:15111-6a36582.409252435710767791/infra_api_clients/../third_party/pipeline/pipeline.py", line 2207, in evaluate
    yielded = pipeline_iter.send(next_value)
  File "/base/data/home/apps/s~findit-for-me/waterfall-backend:15111-6a36582.409252435710767791/gae_libs/pipelines.py", line 295, in run
    sub_pipeline = pipeline_iter.send(next_future)
  File "/base/data/home/apps/s~findit-for-me/waterfall-backend:15111-6a36582.409252435710767791/pipelines/flake_failure/determine_approximate_pass_rate_pipeline.py", line 61, in RunImpl
    assert previous_swarming_task_output
AssertionError

Comment 1 by lijeffrey@chromium.org, Apr 27 2018

Summary: [Findit] Flake Analyzer - Lookback algorithm is suggesting to run a commit position that has already been analyzed (was: [Findit] Flake Analyzer - previous_swarming_task is None in DetermineApproximatePassRatePipeline)
This assertion appears to be handling things correctly. The root cause in this particular case (and possibly in others) is likely in the lookback algorithm. Looking at the pipeline history, it appears commit position 552585 was already analyzed and completed, yet a later pipeline attempted to analyze that commit position again. Luckily for this assert it's possible the analysis could have run into an infinite loop.

Updating the title summary accordingly

Comment 2 by lijeffrey@chromium.org, Apr 27 2018

Summary: [Findit] Flake Analyzer - NextCommitPositionPipeline is suggesting to run a commit position that has already been analyzed (was: [Findit] Flake Analyzer - Lookback algorithm is suggesting to run a commit position that has already been analyzed)
Project Member

Comment 3 by bugdroid1@chromium.org, May 4 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/cbdbc07ede5760d839736786baaec9895100f3c8

commit cbdbc07ede5760d839736786baaec9895100f3c8
Author: Jeffrey Li <lijeffrey@chromium.org>
Date: Fri May 04 20:45:40 2018

[Findit] Flake Analyzer - Fixing running existing data points

The root cause is a corner case when exponential search lands exactly
on a data point that has already been ran, when the distance is exactly
a square number. There is an off-by-1 error checking for when to restart
exponential search from 1, causing the existing data point to be suggested
to run again, which causes problems later on in the analysis.

1. Fix the off-by-1 error
2. Add assertion checks to ensure existing data points are never suggested
   again
3. Add assertion checks for when a suspected build is identified to ensure
   it is indeed a suspect (stable -> flaky).

Bug:  836976 
Change-Id: I68e0f8014a9efe8e01009bd380a9d255a9806941
Reviewed-on: https://chromium-review.googlesource.com/1043606
Reviewed-by: Chan Li <chanli@chromium.org>
Commit-Queue: Jeffrey Li <lijeffrey@chromium.org>

[modify] https://crrev.com/cbdbc07ede5760d839736786baaec9895100f3c8/appengine/findit/pipelines/flake_failure/next_commit_position_pipeline.py
[modify] https://crrev.com/cbdbc07ede5760d839736786baaec9895100f3c8/appengine/findit/model/flake/master_flake_analysis.py
[modify] https://crrev.com/cbdbc07ede5760d839736786baaec9895100f3c8/appengine/findit/services/flake_failure/test/lookback_algorithm_test.py
[modify] https://crrev.com/cbdbc07ede5760d839736786baaec9895100f3c8/appengine/findit/model/flake/test/master_flake_analysis_test.py
[modify] https://crrev.com/cbdbc07ede5760d839736786baaec9895100f3c8/appengine/findit/services/flake_failure/lookback_algorithm.py

Comment 4 by lijeffrey@chromium.org, Aug 31

Status: Verified (was: Assigned)
►

Sign in to add a comment