New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 819307 link

Starred by 5 users
Status: Available
Owner:
jclinton@chromium.org
Cc:
la...@chromium.org
jclinton@chromium.org
█ dbehr@chromium.org
chromeos-continuous-integration-fire-hose@google.com
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

Pre-CQ aborts incorrectly.

Project Member Reported by jkop@chromium.org, Mar 6 2018 
On several CLs simultaneously, I got abort messages out of step with what happened. (Affected: crrev.com/c/942094, crrev.com/c/942093, crrev.com/c/942095).

On each, they received a CR+2 some time on March 5th. Prior to this they were V+1, CR=0, CQ=0, TR=0. This sent them to the Pre-CQ (ex: https://luci-milo.appspot.com/buildbot/chromiumos.tryserver/pre_cq/86907), but no messages were displayed or comments added to the CLs.

Around 10:30 AM March 6th, I added CQ+1 to each. Within five minutes, each got several comments in rapid succession:
The Pre-Commit Queue has picked up your change...
The betty-pre-cq trybot for your change has timed out after 240 minutes...
Commit-Queue label set to 0 programmatically by chromite...
The moblab-generic-vm-pre-cq trybot for your change timed out after 240 minutes...

Comment 1 by pprabhu@chromium.org, Mar 6 2018

Components: Infra>Client>ChromeOS
Labels: -Pri-3 Pri-1
Owner: nxia@chromium.org
Status: Assigned (was: Untriaged)
Another example.

https://chrome-internal-review.googlesource.com/c/chromeos/vendor/mali-ddk/+/576977

This is very confusing to the users. nxia@ can you take a look?

Comment 2 by pprabhu@chromium.org, Mar 6 2018

Cc: dbehr@chromium.org

Comment 3 by nxia@chromium.org, May 18 2018

Components: -Infra>Client>ChromeOS Infra>Client>ChromeOS>CI
Owner: ----
Status: Available (was: Assigned)

Comment 4 by jclinton@chromium.org, Aug 10

Cc: jclinton@chromium.org la...@chromium.org
 Issue 866717  has been merged into this issue.

Comment 5 by jclinton@chromium.org, Aug 10

Owner: saklein@chromium.org
Alex, can you add the CL's that you saw this on?

Then, we'll take a look at the CIDB records for those CL's see if we have a pattern.

Then, we may consider moving the Builder PreCQ status update out of the Builder and back in to the PreCQ launcher.

Also, it's possible that the email experience is happening because the status of having launched jobs in the PreCQ doesn't happen for all jobs, therefore the launch email is not being sent.

Comment 6 by jclinton@chromium.org, Aug 10

Labels: -Pri-1 Pri-2

Comment 7 by saklein@chromium.org, Aug 10 

My CLs were TR+1 rather than CQ+1, but displayed the same behavior. It also turns out all 27 from my CL group had the problem at least once, so there's a good collection of examples here.

The instances of it happening were generally around 4:35 PM on August 8th, 9:10 AM on August 9th, and 8:30 or 9:10 AM on August 10th. Each CL might have a different combination of some of those times.


Examples:

If the pattern for the others holds, this one will do it again the next time it's marked as TR+1, so it might be worth looking at first.
https://chromium-review.googlesource.com/c/chromiumos/overlays/board-overlays/+/1166186

https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/1165329

https://chromium-review.googlesource.com/c/chromiumos/overlays/eclass-overlay/+/1165943

https://chromium-review.googlesource.com/c/chromiumos/overlays/board-overlays/+/1165961

https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-glkrvp-private/+/660343

https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-cnlrvp-private/+/660361

https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-sklrvp-private/+/660353

https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-pyro-private/+/660347

https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-nyan-private/+/660346

https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-aplrvp-private/+/660338

https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-rowan-private/+/660360

https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-iclrvp-private/+/660359

https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-reef-private/+/660350

https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-amenia-private/+/660339

https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-smaug-private/+/660355

https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-smaug-kasan-private/+/660354

https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-kblrvp-private/+/660344

https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-fizz-private/+/660342

https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-snappy-private/+/660356

https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-fizz-moblab-private/+/660349

https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-plaso-private/+/660358

https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-cyan-private/+/660341

https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-sand-private/+/660352

https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-samus-private/+/660351

https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-kefka-private/+/660345

https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-variant-parrot-ivb-private/+/660357

https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-fizz-accelerator-private/+/660348

Comment 8 by jclinton@chromium.org, Sep 20

Owner: jclinton@chromium.org
Going to find time to take a look at this. Thanks for the information, Alex.
►

Sign in to add a comment