New issue
Advanced search Search tips

Issue 624736 link

Starred by 1 user

Issue metadata

Status: Verified
Owner:
Closed: Jul 2016
Cc:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

Fix bvt-cq swarming timeout on cyan-cheets-chrome-pfq

Project Member Reported by cn...@chromium.org, Jun 30 2016

Issue description

cyan-cheets-chrome-pfq seems to timeout on bvt-cq step very often. 
There are 8 machines available but 1 is down (so 7 healthy).

 

Comment 1 by cn...@chromium.org, Jun 30 2016

Cc: jhorwich@chromium.org cn...@chromium.org
Project Member

Comment 2 by bugdroid1@chromium.org, Jul 1 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/25e3fdeebec183efc9ce9988725ff092346b9496

commit 25e3fdeebec183efc9ce9988725ff092346b9496
Author: cnwan <cnwan@google.com>
Date: Thu Jun 30 09:49:13 2016

Decrease number of available DUTs assigned to bvt-cq to 3 since we have 1 machine down in the lab.

BUG= chromium:624736 
TEST=FEATURES=test emerge-veyron_minnie-cheets chromite

Change-Id: Ief3ad29b6ba87b33759f345a51868ec9bacc7979
Reviewed-on: https://chromium-review.googlesource.com/357770
Reviewed-by: Chi-Ngai Wan <cnwan@google.com>
Commit-Queue: Chi-Ngai Wan <cnwan@google.com>
Tested-by: Chi-Ngai Wan <cnwan@google.com>
Trybot-Ready: Chi-Ngai Wan <cnwan@google.com>
Reviewed-by: Chung-yih Wang <cywang@google.com>

[modify] https://crrev.com/25e3fdeebec183efc9ce9988725ff092346b9496/cbuildbot/config_dump.json
[modify] https://crrev.com/25e3fdeebec183efc9ce9988725ff092346b9496/cbuildbot/chromeos_config.py

Project Member

Comment 3 by bugdroid1@chromium.org, Jul 1 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/25e3fdeebec183efc9ce9988725ff092346b9496

commit 25e3fdeebec183efc9ce9988725ff092346b9496
Author: cnwan <cnwan@google.com>
Date: Thu Jun 30 09:49:13 2016

Decrease number of available DUTs assigned to bvt-cq to 3 since we have 1 machine down in the lab.

BUG= chromium:624736 
TEST=FEATURES=test emerge-veyron_minnie-cheets chromite

Change-Id: Ief3ad29b6ba87b33759f345a51868ec9bacc7979
Reviewed-on: https://chromium-review.googlesource.com/357770
Reviewed-by: Chi-Ngai Wan <cnwan@google.com>
Commit-Queue: Chi-Ngai Wan <cnwan@google.com>
Tested-by: Chi-Ngai Wan <cnwan@google.com>
Trybot-Ready: Chi-Ngai Wan <cnwan@google.com>
Reviewed-by: Chung-yih Wang <cywang@google.com>

[modify] https://crrev.com/25e3fdeebec183efc9ce9988725ff092346b9496/cbuildbot/config_dump.json
[modify] https://crrev.com/25e3fdeebec183efc9ce9988725ff092346b9496/cbuildbot/chromeos_config.py

Comment 4 by cn...@chromium.org, Jul 1 2016

Status: Fixed (was: Assigned)
The root cause is actually on cautotest that 

1. it took almost 20 mins in _run_cleanup task which aborted some jobs.
2.  

06/29 23:01:57.897 DEBUG|        monitor_db:0332| Calling _drone_manager.sync_refresh().
06/29 23:01:57.959 DEBUG|        monitor_db:0332| Calling _run_cleanup().
06/29 23:01:57.960 DEBUG|        monitor_db:0332| Calling _find_aborting().
06/29 23:02:01.247 DEBUG|        monitor_db:0332| Calling _find_aborted_special_tasks().
06/29 23:02:13.637 DEBUG|        monitor_db:0332| Calling _handle_agents().
06/29 23:04:23.009 DEBUG|        monitor_db:0332| Calling _host_scheduler.tick().
06/29 23:04:23.009 DEBUG|        monitor_db:0332| Calling _drone_manager.execute_actions().
06/29 23:04:26.344 DEBUG|        monitor_db:0332| Calling _drone_manager.trigger_refresh().
06/29 23:04:26.421 DEBUG|        monitor_db:0332| Calling _process_recurring_runs().
06/29 23:04:26.612 DEBUG|        monitor_db:0332| Calling _schedule_delay_tasks().
06/29 23:04:26.645 DEBUG|        monitor_db:0332| Calling _schedule_running_host_queue_entries().
06/29 23:04:59.074 DEBUG|        monitor_db:0332| Calling _schedule_special_tasks().
06/29 23:05:12.047 DEBUG|        monitor_db:0332| Calling _schedule_new_jobs().
06/29 23:06:59.463 DEBUG|        monitor_db:0332| Calling _drone_manager.sync_refresh().
06/29 23:06:59.526 DEBUG|        monitor_db:0332| Calling _run_cleanup().
06/29 23:20:12.357 DEBUG|        monitor_db:0332| Calling _find_aborting().
06/29 23:20:14.656 DEBUG|        monitor_db:0332| Calling _find_aborted_special_tasks().
06/29 23:20:22.796 DEBUG|        monitor_db:0332| Calling _handle_agents().

Labels: VerifyIn-54
Status: Verified (was: Fixed)
bulk verified

Sign in to add a comment