New issue
Advanced search Search tips

Issue 675472 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Closed: Jan 2017
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug-Regression



Sign in to add a comment

"infra go tests" are flaky in https://build.chromium.org/p/chromium.infra

Project Member Reported by kjellander@chromium.org, Dec 18 2016

Issue description

I wanted to land some CLs this weekend and found the tree was closed. I retried the failing build on https://build.chromium.org/p/chromium.infra/builders/infra-continuous-win-64/builds/5045 as https://build.chromium.org/p/chromium.infra/builders/infra-continuous-win-64/builds/5046 and it then succeeded.

Unfortunately https://uberchromegw.corp.google.com/i/chromium.infra/builders/infra-continuous-xenial-64/builds/2585 failed on the first retry (same revision), but then I forced a new build and it went green.

Same thing with https://uberchromegw.corp.google.com/i/chromium.infra/builders/infra-continuous-precise-64/builds/5116 which succeeded in the following build.

I'm not sure what's causing the flakiness but I could see Vadim had been reopening several times already at https://infra-status.appspot.com/

All these failures seems to be like: 
.......x......
Failures:

  * E:/b/rr/tmptcfj_l/w/infra/go/src/infra/monitoring/analyzer/step/test_step_test.go 
  Line 355:
  Expected: '[]messages.ReasonRaw{(*step.testFailure){TestNames:[]string{"...... too many results, data snipped....", "test_a", "test_b"}, StepName:"something_tests", Tests:[]step.testWithResult{step.testWithResult{TestName:"test_a", IsFlaky:false, SuspectedCLs:[]messages.SuspectCL{messages.SuspectCL{RepoName:"repo", Revision:"deadbeef", CommitPosition:1234, Confidence:0, AnalysisApproach:""}}}, step.testWithResult{TestName:"test_b", IsFlaky:false, SuspectedCLs:[]messages.SuspectCL{messages.SuspectCL{RepoName:"repo", Revision:"deadbeef", CommitPosition:1234, Confidence:0, AnalysisApproach:""}}}, step.testWithResult{TestName:"...... too many results, data snipped....", IsFlaky:false, SuspectedCLs:[]messages.SuspectCL(nil)}}}}'
  Actual:   '[]messages.ReasonRaw{(*step.testFailure){TestNames:[]string{"...... too many results, data snipped....", "test_b", "test_c"}, StepName:"something_tests", Tests:[]step.testWithResult{step.testWithResult{TestName:"test_b", IsFlaky:false, SuspectedCLs:[]messages.SuspectCL{messages.SuspectCL{RepoName:"repo", Revision:"deadbeef", CommitPosition:1234, Confidence:0, AnalysisApproach:""}}}, step.testWithResult{TestName:"test_c", IsFlaky:false, SuspectedCLs:[]messages.SuspectCL(nil)}, step.testWithResult{TestName:"...... too many results, data snipped....", IsFlaky:false, SuspectedCLs:[]messages.SuspectCL(nil)}}}}'
  (Should resemble)!


 
Project Member

Comment 1 by bugdroid1@chromium.org, Dec 18 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra.git/+/81ff0a4cbfede0c84a1d13d3d106757bf5186022

commit 81ff0a4cbfede0c84a1d13d3d106757bf5186022
Author: Henrik Kjellander <kjellander@chromium.org>
Date: Sun Dec 18 19:52:31 2016

Revert "AD: Analyze tests even when test results fails"

This reverts commit 6bd183c35d9b955fe999e0fbf915fb0dfeb777d2.

Reason for revert: Most likely made "infra go tests" flaky.
BUG= 675472 

Original change's description:
> AD: Analyze tests even when test results fails
> 
> Changes the analyzer logic to continue to try to analyze data
> even when test results has no data. This is important to
> take into account the test suite detection logic.
> 
> Also adds some special casing for chromium.perf's benchmark names.
> 
> BUG= 674708 
> 
> Change-Id: I6253bf9f6a57a97ad7ff295f982d3d84eb5f2202
> Reviewed-on: https://chromium-review.googlesource.com/421048
> Reviewed-by: Sean McCullough <seanmccullough@chromium.org>
> Commit-Queue: Stephen Martinis <martiniss@chromium.org>
> 

TBR=seanmccullough@chromium.org,martiniss@chromium.org,zhangtiff@chromium.org
BUG= 674708 
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true

Change-Id: Ic83ad5a150eb479f14e9a874460ecd6fe2415f5c
Reviewed-on: https://chromium-review.googlesource.com/420791
Reviewed-by: Henrik Kjellander <kjellander@chromium.org>
Commit-Queue: Henrik Kjellander <kjellander@chromium.org>

[modify] https://crrev.com/81ff0a4cbfede0c84a1d13d3d106757bf5186022/go/src/infra/monitoring/analyzer/analyzer.go
[modify] https://crrev.com/81ff0a4cbfede0c84a1d13d3d106757bf5186022/go/src/infra/monitoring/analyzer/step/test_step.go
[modify] https://crrev.com/81ff0a4cbfede0c84a1d13d3d106757bf5186022/go/src/infra/monitoring/analyzer/step/test_step_test.go

Yeah, this could be related to that CL. I know that test was flaky and it landed. That CL might have e actually fixed it. Not sure, but I'll take a look. 
Went green at least for all bots with the revert. Could have been just luck too I guess. 
Components: -Infra Infra>Monitoring
Status: Fixed (was: Assigned)
I think this is fixed now.

Sign in to add a comment