New issue
Advanced search Search tips

Issue 911810 link

Starred by 1 user

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 3
Type: Bug



Sign in to add a comment

Pre-CQ failure not reported to gerrit

Project Member Reported by cmtm@google.com, Dec 4

Issue description

This CL:
https://chromium-review.googlesource.com/c/chromiumos/platform2/+/1352846

Last message from Chrome OS commit bot is: "The Pre-Commit Queue has picked up your change.".
The betty build failed hours ago, but this hasn't been posted to Gerrit.

cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8928098783849538720
 
Labels: OS-Chrome
Owner: jclinton@chromium.org
Status: Assigned (was: Untriaged)
The UI says that (based on CIDB data) two of the precq jobs haven't finished: https://chromeos-cl-viewer-ui.googleplex.com/cl_status/chromium-review.googlesource.com/1352846/2

The UI has a bug where it's trying to use the old Buildbot ID's and isn't using Buildbucket ID's. We're shutting down that UI at the end of Q1 but we can find the builds another way:
* https://chrome-swarming.appspot.com/task?id=419303e67229b610&refresh=10
* https://chrome-swarming.appspot.com/task?id=419217070171c010&refresh=10

We can see that betty and betty-arcnext timed out and so the CIDB record of the execution was never written (because the bot was forcibly killed before it could write).

I would remove Trybot+1 and re-add it to get this back in to the PreCQ; I believe that will work-around the failure.

Is it normal that the CQ still hasn't picked up my change again 20 hours after the Commit Queue was -1ed and then +1ed?
No, something is wrong with the pickup logic. Looking how to override it.
Well this is strange; it keeps running the two failed builds over and over again:

mysql> SELECT patch_number, reason, timestamp, buildbucket_id FROM clActionTable WHERE change_number = 1352846 AND action = 'trybot_launching';
+--------------+--------------------------------+---------------------+---------------------+
| patch_number | reason                         | timestamp           | buildbucket_id      |
+--------------+--------------------------------+---------------------+---------------------+
|            1 | daisy_spring-no-vmtest-pre-cq  | 2018-11-28 08:52:02 | 8928626645548123184 |
|            1 | betty-pre-cq                   | 2018-11-28 08:52:02 | 8928626644441263360 |
|            1 | grunt-no-vmtest-pre-cq         | 2018-11-28 08:52:02 | 8928626643938393520 |
|            1 | kevin-arcnext-no-vmtest-pre-cq | 2018-11-28 08:52:02 | 8928626642778657232 |
|            1 | betty-arcnext-pre-cq           | 2018-11-28 08:52:02 | 8928626642250232912 |
|            1 | nyan_blaze-no-vmtest-pre-cq    | 2018-11-28 08:52:02 | 8928626641706093968 |
|            1 | eve-no-vmtest-pre-cq           | 2018-11-28 08:52:02 | 8928626641109720272 |
|            1 | reef-no-vmtest-pre-cq          | 2018-11-28 08:52:02 | 8928626639236497968 |
|            1 | whirlwind-no-vmtest-pre-cq     | 2018-11-28 08:52:02 | 8928626637173838464 |
|            1 | lakitu-no-vmtest-pre-cq        | 2018-11-28 08:52:02 | 8928626636060906704 |
|            1 | cyan-no-vmtest-pre-cq          | 2018-11-28 08:52:02 | 8928626635361780592 |
|            1 | samus-no-vmtest-pre-cq         | 2018-11-28 08:52:02 | 8928626634492581648 |
|            1 | fizz-no-vmtest-pre-cq          | 2018-11-28 08:52:02 | 8928626633744803952 |
|            1 | guado_moblab-no-vmtest-pre-cq  | 2018-11-28 08:52:02 | 8928626633139273984 |
|            1 | zako-no-vmtest-pre-cq          | 2018-11-28 08:52:02 | 8928626632476644880 |
|            1 | betty-arcnext-pre-cq           | 2018-11-28 12:54:02 | 8928611407610564192 |
|            1 | betty-pre-cq                   | 2018-11-28 12:56:54 | 8928611227533207376 |
|            1 | betty-pre-cq                   | 2018-11-28 17:11:09 | 8928595232603777248 |
|            1 | betty-arcnext-pre-cq           | 2018-11-28 17:11:09 | 8928595232062510720 |
|            2 | daisy_spring-no-vmtest-pre-cq  | 2018-11-28 20:07:02 | 8928584176489180336 |
|            2 | betty-pre-cq                   | 2018-11-28 20:07:02 | 8928584175814405808 |
|            2 | grunt-no-vmtest-pre-cq         | 2018-11-28 20:07:02 | 8928584175202368416 |
|            2 | kevin-arcnext-no-vmtest-pre-cq | 2018-11-28 20:07:02 | 8928584174208599200 |
|            2 | betty-arcnext-pre-cq           | 2018-11-28 20:07:03 | 8928584173589673872 |
|            2 | nyan_blaze-no-vmtest-pre-cq    | 2018-11-28 20:07:03 | 8928584173069136816 |
|            2 | eve-no-vmtest-pre-cq           | 2018-11-28 20:07:03 | 8928584172332286080 |
|            2 | reef-no-vmtest-pre-cq          | 2018-11-28 20:07:03 | 8928584171329946304 |
|            2 | whirlwind-no-vmtest-pre-cq     | 2018-11-28 20:07:03 | 8928584170616822128 |
|            2 | lakitu-no-vmtest-pre-cq        | 2018-11-28 20:07:03 | 8928584169946899680 |
|            2 | cyan-no-vmtest-pre-cq          | 2018-11-28 20:07:04 | 8928584169315862816 |
|            2 | samus-no-vmtest-pre-cq         | 2018-11-28 20:07:04 | 8928584168496137600 |
|            2 | fizz-no-vmtest-pre-cq          | 2018-11-28 20:07:04 | 8928584167922100864 |
|            2 | guado_moblab-no-vmtest-pre-cq  | 2018-11-28 20:07:04 | 8928584166544690128 |
|            2 | zako-no-vmtest-pre-cq          | 2018-11-28 20:07:04 | 8928584165843835216 |
|            2 | betty-pre-cq                   | 2018-11-29 01:00:23 | 8928565710647193296 |
|            2 | betty-arcnext-pre-cq           | 2018-11-29 01:00:23 | 8928565710002999040 |
|            2 | betty-pre-cq                   | 2018-11-29 05:07:02 | 8928550193344415600 |
|            2 | betty-arcnext-pre-cq           | 2018-11-29 05:07:02 | 8928550192530316016 |
|            2 | betty-pre-cq                   | 2018-11-30 19:20:19 | 8928405910993536464 |
|            2 | betty-arcnext-pre-cq           | 2018-11-30 19:20:19 | 8928405910327383040 |
|            2 | betty-pre-cq                   | 2018-12-01 02:00:15 | 8928380749513766096 |
|            2 | betty-arcnext-pre-cq           | 2018-12-01 02:00:15 | 8928380748714770624 |
|            2 | betty-pre-cq                   | 2018-12-01 06:45:10 | 8928362824687813312 |
|            2 | betty-arcnext-pre-cq           | 2018-12-01 06:48:40 | 8928362603792580016 |
|            2 | betty-pre-cq                   | 2018-12-01 10:48:11 | 8928347534911274736 |
|            2 | betty-arcnext-pre-cq           | 2018-12-01 10:50:37 | 8928347380842700320 |
|            2 | betty-pre-cq                   | 2018-12-01 14:50:39 | 8928332279961116768 |
|            2 | betty-arcnext-pre-cq           | 2018-12-01 14:52:58 | 8928332133477072896 |
|            2 | betty-pre-cq                   | 2018-12-01 18:53:08 | 8928317023844607040 |
|            2 | betty-arcnext-pre-cq           | 2018-12-01 18:55:23 | 8928316882813988016 |
|            2 | betty-pre-cq                   | 2018-12-01 22:56:15 | 8928301729694837152 |
|            2 | betty-arcnext-pre-cq           | 2018-12-01 22:56:15 | 8928301728509913264 |
|            2 | betty-pre-cq                   | 2018-12-02 02:59:34 | 8928286421330622928 |
|            2 | betty-arcnext-pre-cq           | 2018-12-02 02:59:34 | 8928286420422649392 |
|            2 | betty-pre-cq                   | 2018-12-02 07:01:39 | 8928271190547753600 |
|            2 | betty-arcnext-pre-cq           | 2018-12-02 07:01:39 | 8928271189884538816 |
|            2 | betty-pre-cq                   | 2018-12-02 11:09:42 | 8928255584434317424 |
|            2 | betty-arcnext-pre-cq           | 2018-12-02 11:09:42 | 8928255583639019376 |
|            2 | betty-pre-cq                   | 2018-12-04 04:41:58 | 8928098784330217648 |
|            2 | betty-arcnext-pre-cq           | 2018-12-04 04:41:58 | 8928098783849538720 |
|            2 | betty-pre-cq                   | 2018-12-04 08:45:04 | 8928083489941057616 |
|            2 | betty-arcnext-pre-cq           | 2018-12-04 08:47:52 | 8928083313474610112 |
|            2 | betty-pre-cq                   | 2018-12-04 12:47:09 | 8928068258717086960 |
|            2 | betty-arcnext-pre-cq           | 2018-12-04 12:50:57 | 8928068019637155248 |
|            2 | betty-pre-cq                   | 2018-12-04 17:09:42 | 8928051741950218272 |
|            2 | betty-arcnext-pre-cq           | 2018-12-04 17:09:42 | 8928051740905939216 |
|            2 | betty-pre-cq                   | 2018-12-04 21:32:29 | 8928035208680106496 |
|            2 | betty-arcnext-pre-cq           | 2018-12-04 21:32:29 | 8928035208011426544 |
|            2 | betty-pre-cq                   | 2018-12-05 02:11:22 | 8928017662296161680 |
|            2 | betty-arcnext-pre-cq           | 2018-12-05 02:11:22 | 8928017661505348240 |
|            2 | betty-pre-cq                   | 2018-12-05 06:41:21 | 8928000676363967488 |
|            2 | betty-arcnext-pre-cq           | 2018-12-05 06:41:21 | 8928000675490969504 |
|            2 | betty-pre-cq                   | 2018-12-05 11:11:24 | 8927983687391695152 |
|            2 | betty-arcnext-pre-cq           | 2018-12-05 11:11:24 | 8927983686316060800 |
|            2 | betty-pre-cq                   | 2018-12-05 15:41:42 | 8927966681178735440 |
|            2 | betty-arcnext-pre-cq           | 2018-12-05 15:41:42 | 8927966680243311328 |
|            2 | betty-arcnext-pre-cq           | 2018-12-05 19:44:44 | 8927951389330557280 |
|            2 | betty-pre-cq                   | 2018-12-05 20:05:36 | 8927950076492744432 |
+--------------+--------------------------------+---------------------+---------------------+
78 rows in set (0.03 sec)

They are timing out each time with Android failing to start. Are you sure that your change isn't causing these timeouts?

Owner: cmtm@chromium.org
Well, it would be more accurate to say that the first pass through VMTest fails because of several failing Android tests and then the second VMTest attempt times out before it could run.

I'm not sure that my change isn't causing Android start failures. I'll try it out locally on betty.
On the side, I think I'll increase the betty timeout a bit and also maybe add a COMMIT-QUEUE.ini file to include this directory.

Sign in to add a comment