New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 730830 link

Starred by 1 user

Issue metadata

Status: Verified
Owner:
Last visit > 30 days ago
Closed: Jun 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

make test_importer asynchronous on push-to-prod in server lib task.

Project Member Reported by pprabhu@chromium.org, Jun 7 2017

Issue description

I broke the lab today ( issue 730272 )

I spent ~10 + 10 minutes understanding what was wrong.
I wpent ~10 + 10 minutes fixing it.

I spent one day trying to push to prod (without any testing) and validate the changes.

We should speed up push-to-prod to be less than 15 minutes.

The folklore in the team is that the slow steps are:

- build_externals
  - Start moving things to puppet wherever possible.
  - For the rest, add in smartness about when something needs to be updated.
- test_importer (this parses test control files from the new sources and updates the local DB on all shards + master DB with the new tests)
  - Make this asynchronous. Have a cron job once a day to run test_importer. Most prod pushes don't care if the test list updates a little later.


 
Labels: Chase-Pending
Justification:
- This made what should have been at worst a 2 hour outage a day long affair.
- Scope is a couple weeks, since I'm only asking to focus on some low hanging fruits.
Cc: pho...@chromium.org
At one point, an additional slow step in push to prod was waiting for things like apache and scheduler to restart, which was slowed by  Issue 673639 . Do you know if that was improved?
Labels: -Pri-2 -Chase-Pending Chase Pri-1
Owner: shuqianz@chromium.org
Status: Assigned (was: Untriaged)
Summary: make test_importer asynchronous on push-to-prod (was: Speed up push-to-prod)
Summary: make test_importer asynchronous on push-to-prod in server lib task. (was: make test_importer asynchronous on push-to-prod)
I am working on adding devserver push to push-to-prod. One of the steps is to replace deployment script with the server lib task. Therefore, I think it may be better to make the change there.
Project Member

Comment 5 by bugdroid1@chromium.org, Jun 16 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/e0e2bdf0f0fad65ee00281c0b72e081bce7f0b79

commit e0e2bdf0f0fad65ee00281c0b72e081bce7f0b79
Author: Shuqian Zhao <shuqianz@chromium.org>
Date: Fri Jun 16 22:55:40 2017

Project Member

Comment 6 by bugdroid1@chromium.org, Jun 16 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/e0e2bdf0f0fad65ee00281c0b72e081bce7f0b79

commit e0e2bdf0f0fad65ee00281c0b72e081bce7f0b79
Author: Shuqian Zhao <shuqianz@chromium.org>
Date: Fri Jun 16 22:55:40 2017

Cc: ayatane@chromium.org
Due to issues during push I don't know how useful this is, but the push clocked 23 minutes

https://bugs.chromium.org/p/chromium/issues/detail?id=734836
Status: Started (was: Assigned)
23 minutes is definitely better than last week (> 1 hour)
Status: Verified (was: Started)
Project Member

Comment 11 by bugdroid1@chromium.org, Jun 28 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/c165128e27bb9f0a46d8528284876a00d010d72d

commit c165128e27bb9f0a46d8528284876a00d010d72d
Author: Shuqian Zhao <shuqianz@chromium.org>
Date: Wed Jun 28 21:45:13 2017

autotest: move test_importer out of deploy script

test_importer is a time-consuming task in push-to-prod, Move it out of
push to prod, and make it as a cronjob running in the backend once a
day.

BUG= chromium:730830 
TEST=unittest

Change-Id: I4aed9125ffee3cb93cdaf8abffe678439730b854
Reviewed-on: https://chromium-review.googlesource.com/537970
Commit-Ready: Shuqian Zhao <shuqianz@chromium.org>
Tested-by: Shuqian Zhao <shuqianz@chromium.org>
Reviewed-by: Allen Li <ayatane@chromium.org>

[modify] https://crrev.com/c165128e27bb9f0a46d8528284876a00d010d72d/site_utils/deploy_server_local.py

Sign in to add a comment