Build BigQuery tables to archive historical autotest results. Design doc: go/cros-bq-design
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/ci_results_archiver/+/5dcee54290894cfe30d634077b214ad89f118c8c commit 5dcee54290894cfe30d634077b214ad89f118c8c Author: Shuhei Takahashi <nya@chromium.org> Date: Mon Jul 10 18:22:38 2017 Add .gitignore. BUG= chromium:733103 TEST=git status Change-Id: I5eb93fb61eefb091ac93cc252cfeb35688c48fe8 Reviewed-on: https://chromium-review.googlesource.com/563221 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [add] https://crrev.com/5dcee54290894cfe30d634077b214ad89f118c8c/.gitignore
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra_virtualenv/+/5cda24fbb171f62bf59f3065de5119c40b185df0 commit 5cda24fbb171f62bf59f3065de5119c40b185df0 Author: Shuhei Takahashi <nya@chromium.org> Date: Wed Jul 19 08:23:38 2017 infra_virtualenv: Introduce python_venv. This change will introduce a new script `python_venv` which transparently executes a Python interpreter in a virtualenv, possibly creating a new virtualenv if it has not been created yet. This script is a better replacement to `find_virtualenv.sh` because: 1. `python_venv` does not require to be sourced from bash. This allows any program (e.g. Python) to run a Python interpreter in a virtualenv. Also it makes it easy to run an interactive Python shell in a virtualenv, which is convenient to manually inspect the virtualenv environment for debug. 2. `python_venv` can be called from any working directory, which makes the caller code simpler. BUG= chromium:733103 TEST=bin/run_tests TEST=bin/turtle Change-Id: I94e85a3f3e5ec31d29094e273f0ff9466a8405a2 Reviewed-on: https://chromium-review.googlesource.com/563279 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Shuhei Takahashi <nya@chromium.org> [delete] https://crrev.com/704dbb824410432fcd5902ed41509f40735e7fd8/bin/find_virtualenv.sh [modify] https://crrev.com/5cda24fbb171f62bf59f3065de5119c40b185df0/bin/turtle [add] https://crrev.com/5cda24fbb171f62bf59f3065de5119c40b185df0/bin/python_venv [modify] https://crrev.com/5cda24fbb171f62bf59f3065de5119c40b185df0/bin/run_tests [modify] https://crrev.com/5cda24fbb171f62bf59f3065de5119c40b185df0/README.md
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra_virtualenv/+/d0ff929b53a7db85860519b937ba16f3e21dd628 commit d0ff929b53a7db85860519b937ba16f3e21dd628 Author: Shuhei Takahashi <nya@chromium.org> Date: Fri Jul 21 13:32:04 2017 Update comment in python_venv. Comment is updated so that one can tell where this script was copied from. BUG= chromium:733103 TEST=None Change-Id: I733e55dbc9aefdc60f074eed86c5544218487649 Reviewed-on: https://chromium-review.googlesource.com/578702 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/d0ff929b53a7db85860519b937ba16f3e21dd628/bin/python_venv
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra_virtualenv/+/81e2d298887973d8522186121afaaafd365c6515 commit 81e2d298887973d8522186121afaaafd365c6515 Author: Shuhei Takahashi <nya@chromium.org> Date: Mon Jul 24 05:32:47 2017 Add deps for ci_results_archiver. See CL:564960 for requirements.txt. Command to generate wheels: $ pip wheel -w pip_packages \ google-cloud-bigquery \ google-cloud-storage \ pytz BUG= chromium:733103 TEST=bin/run_tests Change-Id: Id0b4e91a9069b292ff2d209a45ffae292de658de Reviewed-on: https://chromium-review.googlesource.com/566144 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [add] https://crrev.com/81e2d298887973d8522186121afaaafd365c6515/pip_packages/google_auth-1.0.1-py2.py3-none-any.whl [add] https://crrev.com/81e2d298887973d8522186121afaaafd365c6515/pip_packages/setuptools-36.0.1-py2.py3-none-any.whl [add] https://crrev.com/81e2d298887973d8522186121afaaafd365c6515/pip_packages/chardet-3.0.4-py2.py3-none-any.whl [add] https://crrev.com/81e2d298887973d8522186121afaaafd365c6515/pip_packages/cachetools-2.0.0-py2.py3-none-any.whl [add] https://crrev.com/81e2d298887973d8522186121afaaafd365c6515/pip_packages/google_auth_httplib2-0.0.2-py2.py3-none-any.whl [add] https://crrev.com/81e2d298887973d8522186121afaaafd365c6515/pip_packages/pyasn1-0.2.3-py2.py3-none-any.whl [add] https://crrev.com/81e2d298887973d8522186121afaaafd365c6515/pip_packages/protobuf-3.3.0-py2.py3-none-any.whl [add] https://crrev.com/81e2d298887973d8522186121afaaafd365c6515/pip_packages/google_cloud_core-0.25.0-py2.py3-none-any.whl [add] https://crrev.com/81e2d298887973d8522186121afaaafd365c6515/pip_packages/urllib3-1.21.1-py2.py3-none-any.whl [add] https://crrev.com/81e2d298887973d8522186121afaaafd365c6515/pip_packages/certifi-2017.4.17-py2.py3-none-any.whl [add] https://crrev.com/81e2d298887973d8522186121afaaafd365c6515/pip_packages/google_resumable_media-0.1.1-py2.py3-none-any.whl [add] https://crrev.com/81e2d298887973d8522186121afaaafd365c6515/pip_packages/httplib2-0.10.3-py2-none-any.whl [add] https://crrev.com/81e2d298887973d8522186121afaaafd365c6515/pip_packages/google_cloud_storage-1.2.0-py2.py3-none-any.whl [add] https://crrev.com/81e2d298887973d8522186121afaaafd365c6515/pip_packages/pyasn1_modules-0.0.9-py2.py3-none-any.whl [add] https://crrev.com/81e2d298887973d8522186121afaaafd365c6515/pip_packages/googleapis_common_protos-1.5.2-py2-none-any.whl [add] https://crrev.com/81e2d298887973d8522186121afaaafd365c6515/pip_packages/requests-2.18.1-py2.py3-none-any.whl [add] https://crrev.com/81e2d298887973d8522186121afaaafd365c6515/pip_packages/pytz-2017.2-py2.py3-none-any.whl [add] https://crrev.com/81e2d298887973d8522186121afaaafd365c6515/pip_packages/idna-2.5-py2.py3-none-any.whl [add] https://crrev.com/81e2d298887973d8522186121afaaafd365c6515/pip_packages/google_cloud_bigquery-0.25.0-py2.py3-none-any.whl
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra_virtualenv/+/7d677a5d2c8cf00ec05ad20f39ba27baafe25e55 commit 7d677a5d2c8cf00ec05ad20f39ba27baafe25e55 Author: Shuhei Takahashi <nya@chromium.org> Date: Mon Jul 24 05:32:47 2017 Add yapf. yapf is Python code formatter like clang-format. https://github.com/google/yapf Command to generate wheels: $ pip wheel -w pip_packages yapf BUG= chromium:733103 TEST=bin/run_tests Change-Id: I0bb6a787b0ef709a3e097fc25e54d6182a98be6e Reviewed-on: https://chromium-review.googlesource.com/566345 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [add] https://crrev.com/7d677a5d2c8cf00ec05ad20f39ba27baafe25e55/pip_packages/yapf-0.16.2-py2.py3-none-any.whl
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/ci_results_archiver/+/b9b6a1129675a77ee54975bbe7a5a7101b52ada3 commit b9b6a1129675a77ee54975bbe7a5a7101b52ada3 Author: Shuhei Takahashi <nya@chromium.org> Date: Mon Jul 24 07:47:16 2017 Set up venv for ci_results_archiver. CQ-DEPEND=CL:563279 CQ-DEPEND=CL:566144 BUG= chromium:733103 TEST=bin/python_venv Change-Id: Id7d950f5e903bc2a0b4fd8e1a0fecb6104f2575d Reviewed-on: https://chromium-review.googlesource.com/564960 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Shuhei Takahashi <nya@chromium.org> [add] https://crrev.com/b9b6a1129675a77ee54975bbe7a5a7101b52ada3/ci_results_archiver/__init__.py [add] https://crrev.com/b9b6a1129675a77ee54975bbe7a5a7101b52ada3/venv/ci_results_archiver [add] https://crrev.com/b9b6a1129675a77ee54975bbe7a5a7101b52ada3/bin/python_venv [add] https://crrev.com/b9b6a1129675a77ee54975bbe7a5a7101b52ada3/venv/chromite [add] https://crrev.com/b9b6a1129675a77ee54975bbe7a5a7101b52ada3/venv/requirements.txt
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/ci_results_archiver/+/f131fd3e844e3e65c4509f0f1b559e3fe3aba1c7 commit f131fd3e844e3e65c4509f0f1b559e3fe3aba1c7 Author: Shuhei Takahashi <nya@chromium.org> Date: Mon Jul 24 07:47:16 2017 Set up lint and unit tests. BUG= chromium:733103 TEST=bin/run_lint TEST=bin/run_tests TEST=bin/run_tests -v Change-Id: I01e5c02e3ca23fe42d1c577688e34264d4182c76 Reviewed-on: https://chromium-review.googlesource.com/564961 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Shuhei Takahashi <nya@chromium.org> [add] https://crrev.com/f131fd3e844e3e65c4509f0f1b559e3fe3aba1c7/pylintrc [add] https://crrev.com/f131fd3e844e3e65c4509f0f1b559e3fe3aba1c7/bin/run_tests [add] https://crrev.com/f131fd3e844e3e65c4509f0f1b559e3fe3aba1c7/PRESUBMIT.cfg [add] https://crrev.com/f131fd3e844e3e65c4509f0f1b559e3fe3aba1c7/bin/run_lint
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/ci_results_archiver/+/b80feb0446874b7216f31c5376909cce81200881 commit b80feb0446874b7216f31c5376909cce81200881 Author: Shuhei Takahashi <nya@chromium.org> Date: Mon Jul 24 07:47:16 2017 Run yapf in presubmit script. yapf is a Python code formatter like clang-format. It will eliminate unproductive coding-style discussions. CQ-DEPEND=CL:566345 BUG= chromium:733103 TEST=bin/run_yapf ci_results_archiver TEST=bin/run_yapf -i ci_results_archiver Change-Id: I3eb97d3d6617dd68f538c6df06efd832e6bc6b31 Reviewed-on: https://chromium-review.googlesource.com/566347 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Shuhei Takahashi <nya@chromium.org> [add] https://crrev.com/b80feb0446874b7216f31c5376909cce81200881/.style.yapf [add] https://crrev.com/b80feb0446874b7216f31c5376909cce81200881/bin/run_yapf [modify] https://crrev.com/b80feb0446874b7216f31c5376909cce81200881/venv/requirements.txt [modify] https://crrev.com/b80feb0446874b7216f31c5376909cce81200881/PRESUBMIT.cfg
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/ci_results_archiver/+/378c12e30cc8eb070f909ab58ccc0dbf46b23295 commit 378c12e30cc8eb070f909ab58ccc0dbf46b23295 Author: Shuhei Takahashi <nya@chromium.org> Date: Tue Sep 12 13:38:06 2017 BigQuery/MySQL utilities. BUG= chromium:733103 TEST=bin/run_lint && bin/run_tests Change-Id: Ia55fe9d24257335d922614dbf3c2684455c41b28 Reviewed-on: https://chromium-review.googlesource.com/564962 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [add] https://crrev.com/378c12e30cc8eb070f909ab58ccc0dbf46b23295/ci_results_archiver/utils/mysql_wrapper_unittest.py [add] https://crrev.com/378c12e30cc8eb070f909ab58ccc0dbf46b23295/ci_results_archiver/utils/__init__.py [add] https://crrev.com/378c12e30cc8eb070f909ab58ccc0dbf46b23295/ci_results_archiver/utils/bigquery_wrapper_unittest.py [add] https://crrev.com/378c12e30cc8eb070f909ab58ccc0dbf46b23295/ci_results_archiver/utils/mysql_wrapper.py [add] https://crrev.com/378c12e30cc8eb070f909ab58ccc0dbf46b23295/ci_results_archiver/utils/bigquery_wrapper.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/ci_results_archiver/+/8dfe051a1d77cabc4ad5d081409128ec6b0d425d commit 8dfe051a1d77cabc4ad5d081409128ec6b0d425d Author: Shuhei Takahashi <nya@chromium.org> Date: Tue Sep 12 13:38:06 2017 Allow run_yapf to run without arguments. When run_yapf is run without arguments, it checks all Python files managed by git, which is consistent with behavior of run_lint. BUG= chromium:733103 TEST=bin/run_yapf Change-Id: I31d8f73b3d50f7e3146124a580c84eb70d4f2cd2 Reviewed-on: https://chromium-review.googlesource.com/649935 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/8dfe051a1d77cabc4ad5d081409128ec6b0d425d/bin/run_yapf
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/ci_results_archiver/+/57e7217484fcf666a483eb49ca5ad99927a37694 commit 57e7217484fcf666a483eb49ca5ad99927a37694 Author: Shuhei Takahashi <nya@chromium.org> Date: Tue Sep 12 13:38:06 2017 Fix function name restrictions in pylintrc. Google Python style guide does not mention about function name length. Also, the regex is fixed not to accept names such as "_fooBar". BUG= chromium:733103 TEST=bin/run_lint Change-Id: I2453f69e0129308ffe42130c3f8aafbb4b16f08b Reviewed-on: https://chromium-review.googlesource.com/649936 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/57e7217484fcf666a483eb49ca5ad99927a37694/pylintrc
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/ci_results_archiver/+/fa020b1f95ce72f71256c23996bc7319e846c8e4 commit fa020b1f95ce72f71256c23996bc7319e846c8e4 Author: Shuhei Takahashi <nya@chromium.org> Date: Wed Sep 13 08:51:00 2017 Return lists from MySQLWrapper. MySQLWrapper.RunQuery() unexpectedly returned tuples, not lists. BUG= chromium:733103 TEST=bin/run_lint && bin/run_tests && bin/run_yapf Change-Id: Ifeb6129042e066f9f18deeab9956e53d9baff4e7 Reviewed-on: https://chromium-review.googlesource.com/662897 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/fa020b1f95ce72f71256c23996bc7319e846c8e4/ci_results_archiver/utils/mysql_wrapper.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/ci_results_archiver/+/076a9baf9536eefe429a9b627a92d92443e3f353 commit 076a9baf9536eefe429a9b627a92d92443e3f353 Author: Shuhei Takahashi <nya@chromium.org> Date: Tue Sep 19 12:25:10 2017 Process TIMESTAMP values with datetime. With this change, all TIMESTAMP values are processed as timezone-aware datetime objects. BUG= chromium:733103 TEST=bin/run_lint && bin/run_tests && bin/run_yapf Change-Id: I2bb1f88c00a27cf99d442eadfacd436286cc7603 Reviewed-on: https://chromium-review.googlesource.com/668388 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Shuhei Takahashi <nya@chromium.org> [modify] https://crrev.com/076a9baf9536eefe429a9b627a92d92443e3f353/ci_results_archiver/utils/mysql_wrapper_unittest.py [modify] https://crrev.com/076a9baf9536eefe429a9b627a92d92443e3f353/ci_results_archiver/utils/bigquery_wrapper.py [modify] https://crrev.com/076a9baf9536eefe429a9b627a92d92443e3f353/ci_results_archiver/utils/bigquery_wrapper_unittest.py [modify] https://crrev.com/076a9baf9536eefe429a9b627a92d92443e3f353/ci_results_archiver/utils/mysql_wrapper.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/76bcba73c261b684e66e42f539014dda081b5494 commit 76bcba73c261b684e66e42f539014dda081b5494 Author: Shuhei Takahashi <nya@chromium.org> Date: Thu Sep 21 08:07:56 2017 cidb: Enhance CIDBConnection.GetBuild(s?)History(). - Allows |build_configs| parameter to be None. - Adds |reverse| parameter to allow retrieving builds in chronological order. Usually this parameter is used together with |starting_build_id|. - Adds more columns to returned dictionaries. This change also affects other functions returning build dictionaries. BUG= chromium:733103 TEST=Call GetBuildsHistory(build_config=None, num_results=1000, starting_build_id=..., reverse=True) Change-Id: Icd83036131fc07dea64fccaee92db6fd20351038 Reviewed-on: https://chromium-review.googlesource.com/559027 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/76bcba73c261b684e66e42f539014dda081b5494/lib/cidb.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/76713a05094da5770bdcab8809e333f3e20ec099 commit 76713a05094da5770bdcab8809e333f3e20ec099 Author: Shuhei Takahashi <nya@chromium.org> Date: Thu Sep 21 08:07:56 2017 cidb: Update fake_cidb. - GetBuild(s?)History() supports more arguments. - InsertBuild() takes start_time. BUG= chromium:733103 TEST=CL:662898 Change-Id: I34feea4142bec36bed107970d721c717e6e2fd20 Reviewed-on: https://chromium-review.googlesource.com/664518 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/76713a05094da5770bdcab8809e333f3e20ec099/lib/fake_cidb.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/ci_results_archiver/+/9cfba4845031fb58208697b347319f29514af618 commit 9cfba4845031fb58208697b347319f29514af618 Author: Shuhei Takahashi <nya@chromium.org> Date: Thu Sep 21 08:08:05 2017 Use pytest for unit tests and coverage. BUG= chromium:733103 TEST=bin/run_lint && bin/run_tests && bin/run_yapf Change-Id: Iced8435c8fea03b1918cf20f69e370dd0948122b Reviewed-on: https://chromium-review.googlesource.com/674515 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [add] https://crrev.com/9cfba4845031fb58208697b347319f29514af618/pytest.ini [modify] https://crrev.com/9cfba4845031fb58208697b347319f29514af618/bin/run_tests [modify] https://crrev.com/9cfba4845031fb58208697b347319f29514af618/.gitignore [modify] https://crrev.com/9cfba4845031fb58208697b347319f29514af618/venv/requirements.txt
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/ci_results_archiver/+/e4efe2795451242698e2a331276b6fa7b80b4b24 commit e4efe2795451242698e2a331276b6fa7b80b4b24 Author: Shuhei Takahashi <nya@chromium.org> Date: Thu Sep 21 12:41:57 2017 Improve dictionary literal formatting. This change tunes yapf parameters since current formatting of dictionary literals is too sparse. The new formatting is still Google Python coding style compliant. CQ-DEPEND=CL:674849 BUG= chromium:733103 TEST=bin/run_lint && bin/run_tests && bin/run_yapf Change-Id: Icbcd5d9975cc634c7371187bfdf547f4da69184f Reviewed-on: https://chromium-review.googlesource.com/674426 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/e4efe2795451242698e2a331276b6fa7b80b4b24/.style.yapf [modify] https://crrev.com/e4efe2795451242698e2a331276b6fa7b80b4b24/ci_results_archiver/utils/bigquery_wrapper_unittest.py [modify] https://crrev.com/e4efe2795451242698e2a331276b6fa7b80b4b24/ci_results_archiver/utils/mysql_wrapper_unittest.py [modify] https://crrev.com/e4efe2795451242698e2a331276b6fa7b80b4b24/venv/requirements.txt
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra_virtualenv/+/f58eb63f6d5353567b5cb8b51b1afcab06a74781 commit f58eb63f6d5353567b5cb8b51b1afcab06a74781 Author: Shuhei Takahashi <nya@chromium.org> Date: Thu Sep 21 12:41:57 2017 Add yapf v0.18.0. Command to generate wheels: $ pip wheel -w pip_packages yapf==0.18.0 BUG= chromium:733103 TEST=bin/run_tests Change-Id: Iedca599c977583662966700e280fda2fa04d9fdb Reviewed-on: https://chromium-review.googlesource.com/674849 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [add] https://crrev.com/f58eb63f6d5353567b5cb8b51b1afcab06a74781/pip_packages/yapf-0.18.0-py2.py3-none-any.whl
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/ci_results_archiver/+/f0e9b9eecc3583072725c6c8861a496cefd6c661 commit f0e9b9eecc3583072725c6c8861a496cefd6c661 Author: Shuhei Takahashi <nya@chromium.org> Date: Thu Sep 28 06:52:09 2017 Implement importers. Note that this change depends on a chromite cidb change in CL:559027. BUG= chromium:733103 TEST=bin/run_lint && bin/run_tests && bin/run_yapf CQ-DEPEND=CL:559027 Change-Id: I605e64a3de9b63e2023e22582342b99027a3e18b Reviewed-on: https://chromium-review.googlesource.com/662898 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [add] https://crrev.com/f0e9b9eecc3583072725c6c8861a496cefd6c661/ci_results_archiver/utils/test/fake_afe_connection.py [add] https://crrev.com/f0e9b9eecc3583072725c6c8861a496cefd6c661/ci_results_archiver/importers/grace_period_unittest.py [add] https://crrev.com/f0e9b9eecc3583072725c6c8861a496cefd6c661/ci_results_archiver/utils/test/fake_tko_connection.py [add] https://crrev.com/f0e9b9eecc3583072725c6c8861a496cefd6c661/ci_results_archiver/importers/afe_job_importer_unittest.py [add] https://crrev.com/f0e9b9eecc3583072725c6c8861a496cefd6c661/ci_results_archiver/importers/grace_period.py [add] https://crrev.com/f0e9b9eecc3583072725c6c8861a496cefd6c661/ci_results_archiver/importers/tko_job_importer.py [add] https://crrev.com/f0e9b9eecc3583072725c6c8861a496cefd6c661/ci_results_archiver/importers/cidb_build_importer.py [modify] https://crrev.com/f0e9b9eecc3583072725c6c8861a496cefd6c661/pylintrc [add] https://crrev.com/f0e9b9eecc3583072725c6c8861a496cefd6c661/ci_results_archiver/importers/tko_job_importer_unittest.py [add] https://crrev.com/f0e9b9eecc3583072725c6c8861a496cefd6c661/ci_results_archiver/importers/__init__.py [add] https://crrev.com/f0e9b9eecc3583072725c6c8861a496cefd6c661/ci_results_archiver/utils/tko_connection.py [add] https://crrev.com/f0e9b9eecc3583072725c6c8861a496cefd6c661/ci_results_archiver/importers/cidb_build_importer_unittest.py [add] https://crrev.com/f0e9b9eecc3583072725c6c8861a496cefd6c661/ci_results_archiver/utils/test/__init__.py [add] https://crrev.com/f0e9b9eecc3583072725c6c8861a496cefd6c661/ci_results_archiver/importers/abstract_importer.py [add] https://crrev.com/f0e9b9eecc3583072725c6c8861a496cefd6c661/ci_results_archiver/utils/afe_connection.py [add] https://crrev.com/f0e9b9eecc3583072725c6c8861a496cefd6c661/ci_results_archiver/importers/afe_job_importer.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/ci_results_archiver/+/338a16c4c1fda67b2e28576bdcad350a91b376c5 commit 338a16c4c1fda67b2e28576bdcad350a91b376c5 Author: Shuhei Takahashi <nya@chromium.org> Date: Thu Sep 28 06:52:09 2017 Introduce TableType and TableSpec. BUG= chromium:733103 TEST=bin/run_lint && bin/run_tests && bin/run_yapf Change-Id: Ibf64a3be17a920f0a1fd2af329a7923515cdb160 Reviewed-on: https://chromium-review.googlesource.com/674486 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Dan Shi <dshi@google.com> [add] https://crrev.com/338a16c4c1fda67b2e28576bdcad350a91b376c5/ci_results_archiver/table_types.py [add] https://crrev.com/338a16c4c1fda67b2e28576bdcad350a91b376c5/ci_results_archiver/table_specs.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/ci_results_archiver/+/f4ee95437426fee83b92c5e178fe2c820c8f74a7 commit f4ee95437426fee83b92c5e178fe2c820c8f74a7 Author: Shuhei Takahashi <nya@chromium.org> Date: Thu Sep 28 06:52:09 2017 Implement modifiers. BUG= chromium:733103 TEST=bin/run_lint && bin/run_tests && bin/run_yapf Change-Id: Ib0aebd689838ac2fda5be3a46835d5e07eb038c1 Reviewed-on: https://chromium-review.googlesource.com/674487 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Shuhei Takahashi <nya@chromium.org> [add] https://crrev.com/f4ee95437426fee83b92c5e178fe2c820c8f74a7/ci_results_archiver/modifiers/tko_job_test_invalidator_unittest.py [add] https://crrev.com/f4ee95437426fee83b92c5e178fe2c820c8f74a7/ci_results_archiver/utils/bigquery_tables_unittest.py [add] https://crrev.com/f4ee95437426fee83b92c5e178fe2c820c8f74a7/ci_results_archiver/utils/test/fake_bigquery_tables.py [add] https://crrev.com/f4ee95437426fee83b92c5e178fe2c820c8f74a7/ci_results_archiver/modifiers/tko_job_test_invalidator.py [add] https://crrev.com/f4ee95437426fee83b92c5e178fe2c820c8f74a7/ci_results_archiver/utils/bigquery_tko_tables.py [add] https://crrev.com/f4ee95437426fee83b92c5e178fe2c820c8f74a7/ci_results_archiver/utils/bigquery_tables.py [modify] https://crrev.com/f4ee95437426fee83b92c5e178fe2c820c8f74a7/ci_results_archiver/utils/bigquery_wrapper.py [add] https://crrev.com/f4ee95437426fee83b92c5e178fe2c820c8f74a7/ci_results_archiver/modifiers/__init__.py [add] https://crrev.com/f4ee95437426fee83b92c5e178fe2c820c8f74a7/ci_results_archiver/modifiers/abstract_modifier.py [add] https://crrev.com/f4ee95437426fee83b92c5e178fe2c820c8f74a7/ci_results_archiver/utils/test/fake_bigquery_tko_tables.py [modify] https://crrev.com/f4ee95437426fee83b92c5e178fe2c820c8f74a7/ci_results_archiver/utils/bigquery_wrapper_unittest.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra_virtualenv/+/28108475244df2512c2834323be49c4e48ac0821 commit 28108475244df2512c2834323be49c4e48ac0821 Author: Shuhei Takahashi <nya@chromium.org> Date: Thu Sep 28 10:59:27 2017 Add apscheduler. Command to generate wheels: $ pip wheel -w pip_packages apscheduler BUG= chromium:733103 TEST=bin/run_tests Change-Id: I3beb9bb615365e16834b4a7474992616d523df63 Reviewed-on: https://chromium-review.googlesource.com/683976 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [add] https://crrev.com/28108475244df2512c2834323be49c4e48ac0821/pip_packages/APScheduler-3.3.1-py2.py3-none-any.whl [add] https://crrev.com/28108475244df2512c2834323be49c4e48ac0821/pip_packages/tzlocal-1.4-py2-none-any.whl
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/ci_results_archiver/+/f8f7314368194d959594b796eff3dc7234d7644b commit f8f7314368194d959594b796eff3dc7234d7644b Author: Shuhei Takahashi <nya@chromium.org> Date: Thu Sep 28 10:59:24 2017 Implement the exporter. BUG= chromium:733103 TEST=bin/run_lint && bin/run_tests && bin/run_yapf Change-Id: Ifd89ef86212099df091b8df5c28d6de9d5d999e6 Reviewed-on: https://chromium-review.googlesource.com/674488 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Shuhei Takahashi <nya@chromium.org> [add] https://crrev.com/f8f7314368194d959594b796eff3dc7234d7644b/ci_results_archiver/bigquery_exporter_unittest.py [add] https://crrev.com/f8f7314368194d959594b796eff3dc7234d7644b/ci_results_archiver/bigquery_exporter.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/ci_results_archiver/+/2e7d4e3f7f53a4f2b3556dcb49de01fb49a3cec3 commit 2e7d4e3f7f53a4f2b3556dcb49de01fb49a3cec3 Author: Shuhei Takahashi <nya@chromium.org> Date: Thu Sep 28 10:59:23 2017 Use outer joins. Inner joins may drop rows when the joined column has null values. Though we generally do not expect such cases, we want to avoid dropping rows. BUG= chromium:733103 TEST=bin/run_lint && bin/run_tests && bin/run_yapf Change-Id: I32b553db20e6c4927fb2247ebc3a49f4a9904157 Reviewed-on: https://chromium-review.googlesource.com/683782 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/2e7d4e3f7f53a4f2b3556dcb49de01fb49a3cec3/ci_results_archiver/utils/tko_connection.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/ci_results_archiver/+/731cb229e565b6308839aa0d8e4c74727fc3449e commit 731cb229e565b6308839aa0d8e4c74727fc3449e Author: Shuhei Takahashi <nya@chromium.org> Date: Mon Oct 02 10:03:46 2017 Implement checkpoint. BUG= chromium:733103 TEST=bin/run_lint && bin/run_tests && bin/run_yapf Change-Id: I262811d6cede022da55fab7a71a81bb5edf85d51 Reviewed-on: https://chromium-review.googlesource.com/683996 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [add] https://crrev.com/731cb229e565b6308839aa0d8e4c74727fc3449e/ci_results_archiver/checkpoint.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/ci_results_archiver/+/ea7af78d088bb584d4d40c75d0c476b60f51ffdd commit ea7af78d088bb584d4d40c75d0c476b60f51ffdd Author: Shuhei Takahashi <nya@chromium.org> Date: Mon Oct 02 18:23:18 2017 Implement config loader. BUG= chromium:733103 TEST=bin/run_lint && bin/run_tests && bin/run_yapf Change-Id: Id31268aa2a9fb508659637c131df5e96a6186a17 Reviewed-on: https://chromium-review.googlesource.com/683997 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Shuhei Takahashi <nya@chromium.org> [add] https://crrev.com/ea7af78d088bb584d4d40c75d0c476b60f51ffdd/ci_results_archiver/config_loader.py [add] https://crrev.com/ea7af78d088bb584d4d40c75d0c476b60f51ffdd/ci_results_archiver/config_loader_unittest.py [modify] https://crrev.com/ea7af78d088bb584d4d40c75d0c476b60f51ffdd/pylintrc
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/ci_results_archiver/+/263b11cb29f56b39f3179a1d866ad4592e3e5a8f commit 263b11cb29f56b39f3179a1d866ad4592e3e5a8f Author: Shuhei Takahashi <nya@chromium.org> Date: Mon Oct 02 18:23:18 2017 Reconnect when MySQL connection is lost. BUG= chromium:733103 TEST=bin/run_tests && bin/run_yapf -i && bin/run_lint Change-Id: I2c95b6405c6658a21f865c0ec7f3741f890e7ab0 Reviewed-on: https://chromium-review.googlesource.com/689394 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/263b11cb29f56b39f3179a1d866ad4592e3e5a8f/ci_results_archiver/utils/mysql_wrapper.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/ci_results_archiver/+/719f0c6a82b8ba29de96e63ab5fc658387c6af79 commit 719f0c6a82b8ba29de96e63ab5fc658387c6af79 Author: Shuhei Takahashi <nya@chromium.org> Date: Thu Oct 05 12:16:51 2017 The entry point of ci_results_archiver. BUG= chromium:733103 TEST=bin/run_lint && bin/run_tests && bin/run_yapf TEST=bin/ci_results_archiver --config=config.yaml --run-soon CQ-DEPEND=CL:683976 Change-Id: I535500bb4c1d6eda6e6d41d19afb382dec1282f0 Reviewed-on: https://chromium-review.googlesource.com/649937 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/719f0c6a82b8ba29de96e63ab5fc658387c6af79/ci_results_archiver/test/config_example.yaml [add] https://crrev.com/719f0c6a82b8ba29de96e63ab5fc658387c6af79/ci_results_archiver/main.py [modify] https://crrev.com/719f0c6a82b8ba29de96e63ab5fc658387c6af79/venv/requirements.txt [add] https://crrev.com/719f0c6a82b8ba29de96e63ab5fc658387c6af79/bin/ci_results_archiver
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/ci_results_archiver/+/15df28880d37cd041b1c05da8f71c9b1dc2ea954 commit 15df28880d37cd041b1c05da8f71c9b1dc2ea954 Author: Shuhei Takahashi <nya@chromium.org> Date: Thu Oct 05 12:16:50 2017 Implement ArchiveBuilder. BUG= chromium:733103 TEST=bin/run_lint && bin/run_tests && bin/run_yapf Change-Id: I22c96b3a4086203547a250925cadb96097f14d8b Reviewed-on: https://chromium-review.googlesource.com/683998 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [add] https://crrev.com/15df28880d37cd041b1c05da8f71c9b1dc2ea954/ci_results_archiver/test/config_example.yaml [add] https://crrev.com/15df28880d37cd041b1c05da8f71c9b1dc2ea954/ci_results_archiver/utils/text_util.py [add] https://crrev.com/15df28880d37cd041b1c05da8f71c9b1dc2ea954/ci_results_archiver/archive_builder.py [add] https://crrev.com/15df28880d37cd041b1c05da8f71c9b1dc2ea954/ci_results_archiver/archive_builder_factory.py [add] https://crrev.com/15df28880d37cd041b1c05da8f71c9b1dc2ea954/ci_results_archiver/archive_builder_factory_unittest.py [add] https://crrev.com/15df28880d37cd041b1c05da8f71c9b1dc2ea954/ci_results_archiver/utils/text_util_unittest.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/ci_results_archiver/+/285173383dbd7c95824c5d7ec77cdcea885dbc76 commit 285173383dbd7c95824c5d7ec77cdcea885dbc76 Author: Shuhei Takahashi <nya@chromium.org> Date: Thu Oct 05 12:16:50 2017 Make importers return new_next_id. Though it is not currently the case, it is valid as per contract that an importer returns modify_ids with empty entries. In this case, archive builder can not save next_id appropriately to avoid duplicated modifications. This patch updates AbstractImporter and its implementations to return new_next_id to allow such cases. BUG= chromium:733103 TEST=bin/run_tests && bin/run_yapf -i && bin/run_lint Change-Id: I8cda782b5a7c4b7389daefbe84a118bd91735d9b Reviewed-on: https://chromium-review.googlesource.com/694921 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/285173383dbd7c95824c5d7ec77cdcea885dbc76/ci_results_archiver/importers/cidb_build_importer_unittest.py [modify] https://crrev.com/285173383dbd7c95824c5d7ec77cdcea885dbc76/ci_results_archiver/importers/afe_job_importer_unittest.py [modify] https://crrev.com/285173383dbd7c95824c5d7ec77cdcea885dbc76/ci_results_archiver/importers/tko_job_importer.py [modify] https://crrev.com/285173383dbd7c95824c5d7ec77cdcea885dbc76/ci_results_archiver/importers/cidb_build_importer.py [modify] https://crrev.com/285173383dbd7c95824c5d7ec77cdcea885dbc76/ci_results_archiver/importers/tko_job_importer_unittest.py [modify] https://crrev.com/285173383dbd7c95824c5d7ec77cdcea885dbc76/ci_results_archiver/bigquery_exporter.py [modify] https://crrev.com/285173383dbd7c95824c5d7ec77cdcea885dbc76/ci_results_archiver/archive_builder.py [modify] https://crrev.com/285173383dbd7c95824c5d7ec77cdcea885dbc76/ci_results_archiver/importers/abstract_importer.py [modify] https://crrev.com/285173383dbd7c95824c5d7ec77cdcea885dbc76/ci_results_archiver/importers/afe_job_importer.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/ci_results_archiver/+/a18f4cc444c3f95a955af4c221ba029e48768d85 commit a18f4cc444c3f95a955af4c221ba029e48768d85 Author: Shuhei Takahashi <nya@chromium.org> Date: Fri Oct 13 21:44:32 2017 Record various metrics. chromeos/ci_results_archiver/builder/{afe_jobs,tko_jobs,cidb_builds}/ tick ... counter of build runs (with success info) total_duration ... total duration of build breakdown/ ... duration breakdowns import modify export load_checkpoint save_checkpoint next_id ... gauge of next_id modify_count ... counter of modify export_count ... counter of export BUG= chromium:733103 TEST=bin/run_lint && bin/run_tests && bin/run_yapf TEST=manually ran the pipeline without errors Change-Id: Ia0160f8bded66e5013a03b72012210e13669a312 Reviewed-on: https://chromium-review.googlesource.com/683783 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/a18f4cc444c3f95a955af4c221ba029e48768d85/ci_results_archiver/main.py [add] https://crrev.com/a18f4cc444c3f95a955af4c221ba029e48768d85/ci_results_archiver/constants.py [modify] https://crrev.com/a18f4cc444c3f95a955af4c221ba029e48768d85/ci_results_archiver/archive_builder.py [modify] https://crrev.com/a18f4cc444c3f95a955af4c221ba029e48768d85/ci_results_archiver/archive_builder_factory.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/ci_results_archiver/+/960b4af421260208746e30d8e9fdba3e1cb42ff1 commit 960b4af421260208746e30d8e9fdba3e1cb42ff1 Author: Shuhei Takahashi <nya@chromium.org> Date: Fri Oct 13 21:44:32 2017 Record the last build time in the checkpoint file. This information is useful to tell the snapshot time of BigQuery tables. That is, we can expect all tests finished before this time should be included in the BigQuery tables. In order to do record multiple fields in checkpoint files, they are replaced to JSON files instead of simple text files. BUG= chromium:733103 TEST=bin/run_tests && bin/run_yapf -i && bin/run_lint TEST=bin/ci_results_archiver --config config.yaml --run-soon Change-Id: I110c614b99b5deececd4d954be4ca5e0a6c0e2af Reviewed-on: https://chromium-review.googlesource.com/701896 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Shuhei Takahashi <nya@chromium.org> [modify] https://crrev.com/960b4af421260208746e30d8e9fdba3e1cb42ff1/ci_results_archiver/archive_builder_factory_unittest.py [add] https://crrev.com/960b4af421260208746e30d8e9fdba3e1cb42ff1/ci_results_archiver/checkpointer.py [delete] https://crrev.com/a18f4cc444c3f95a955af4c221ba029e48768d85/ci_results_archiver/checkpoint.py [modify] https://crrev.com/960b4af421260208746e30d8e9fdba3e1cb42ff1/ci_results_archiver/archive_builder.py [modify] https://crrev.com/960b4af421260208746e30d8e9fdba3e1cb42ff1/ci_results_archiver/archive_builder_factory.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/ci_results_archiver/+/780035c315c0c37092ff8e1513bfa58e4e8a0068 commit 780035c315c0c37092ff8e1513bfa58e4e8a0068 Author: Shuhei Takahashi <nya@chromium.org> Date: Fri Oct 13 21:44:32 2017 Use different GCS buckets by purpose. We use GCS for two purpose: 1. For persistent storage of checkpoints. 2. For temporary storage to import entries into BigQuery. With this change, we use different buckets for these two purposes. This allows setting TTL to the temporary bucket to automatically delete unnecessary intermediate files. BUG= chromium:733103 TEST=bin/run_tests && bin/run_yapf -i && bin/run_lint TEST=ran the pipeline manually without errors. Change-Id: Iccb4b6b221f2768fbb97bed9df5d27add809b1ba Reviewed-on: https://chromium-review.googlesource.com/701897 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/780035c315c0c37092ff8e1513bfa58e4e8a0068/ci_results_archiver/test/config_example.yaml [modify] https://crrev.com/780035c315c0c37092ff8e1513bfa58e4e8a0068/ci_results_archiver/archive_builder_factory_unittest.py [modify] https://crrev.com/780035c315c0c37092ff8e1513bfa58e4e8a0068/ci_results_archiver/archive_builder_factory.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/ci_results_archiver/+/e8733f758ec15a4dac29d8c060cf4cacc3ce286f commit e8733f758ec15a4dac29d8c060cf4cacc3ce286f Author: Shuhei Takahashi <nya@chromium.org> Date: Fri Oct 13 21:44:32 2017 Dump BigQuery tables into Google Cloud Storage. This patch adds a new component, BigQueryDumper, to the last of ArchiveBuilder pipeline. The dumper will export BigQuery tables updated by modifier/exporter to Google Cloud Storage as gzip'ed newline-delimited JSON files. They are useful for batch access. BUG= chromium:733103 TEST=bin/run_tests && bin/run_yapf -i && bin/run_lint TEST=ran the pipeline manually without errors. Change-Id: I7bb8c6a580d8bb0d9b603f6aebc4e72d7cc9cb7d Reviewed-on: https://chromium-review.googlesource.com/701898 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Shuhei Takahashi <nya@chromium.org> [modify] https://crrev.com/e8733f758ec15a4dac29d8c060cf4cacc3ce286f/ci_results_archiver/modifiers/tko_job_test_invalidator_unittest.py [modify] https://crrev.com/e8733f758ec15a4dac29d8c060cf4cacc3ce286f/ci_results_archiver/utils/test/fake_bigquery_tables.py [modify] https://crrev.com/e8733f758ec15a4dac29d8c060cf4cacc3ce286f/ci_results_archiver/modifiers/tko_job_test_invalidator.py [add] https://crrev.com/e8733f758ec15a4dac29d8c060cf4cacc3ce286f/ci_results_archiver/bigquery_dumper_unittest.py [modify] https://crrev.com/e8733f758ec15a4dac29d8c060cf4cacc3ce286f/ci_results_archiver/modifiers/abstract_modifier.py [modify] https://crrev.com/e8733f758ec15a4dac29d8c060cf4cacc3ce286f/ci_results_archiver/utils/bigquery_tables.py [modify] https://crrev.com/e8733f758ec15a4dac29d8c060cf4cacc3ce286f/ci_results_archiver/utils/bigquery_wrapper.py [add] https://crrev.com/e8733f758ec15a4dac29d8c060cf4cacc3ce286f/ci_results_archiver/bigquery_dumper.py [modify] https://crrev.com/e8733f758ec15a4dac29d8c060cf4cacc3ce286f/ci_results_archiver/bigquery_exporter.py [modify] https://crrev.com/e8733f758ec15a4dac29d8c060cf4cacc3ce286f/ci_results_archiver/archive_builder.py [modify] https://crrev.com/e8733f758ec15a4dac29d8c060cf4cacc3ce286f/ci_results_archiver/archive_builder_factory.py [modify] https://crrev.com/e8733f758ec15a4dac29d8c060cf4cacc3ce286f/ci_results_archiver/archive_builder_factory_unittest.py [modify] https://crrev.com/e8733f758ec15a4dac29d8c060cf4cacc3ce286f/ci_results_archiver/bigquery_exporter_unittest.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/ci_results_archiver/+/e142f1d18cdc893b504ca0229b725bfacd5927b3 commit e142f1d18cdc893b504ca0229b725bfacd5927b3 Author: Shuhei Takahashi <nya@chromium.org> Date: Fri Oct 13 21:44:33 2017 Invalidate TKO tests only when necessary. Before running expensive UPDATE queries, check the invalidation statuses of TKO tests and issue a query only when needed. BUG= chromium:733103 TEST=bin/run_tests && bin/run_yapf -i && bin/run_lint Change-Id: I621a0874ae1d8b020723015f7cd4e336aec35e3e Reviewed-on: https://chromium-review.googlesource.com/708098 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/e142f1d18cdc893b504ca0229b725bfacd5927b3/ci_results_archiver/modifiers/tko_job_test_invalidator.py [modify] https://crrev.com/e142f1d18cdc893b504ca0229b725bfacd5927b3/ci_results_archiver/modifiers/tko_job_test_invalidator_unittest.py [modify] https://crrev.com/e142f1d18cdc893b504ca0229b725bfacd5927b3/ci_results_archiver/utils/test/fake_bigquery_tko_tables.py [modify] https://crrev.com/e142f1d18cdc893b504ca0229b725bfacd5927b3/ci_results_archiver/utils/bigquery_tko_tables.py
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/ff057f32b6279c9d62a04625be7cb77da902930b commit ff057f32b6279c9d62a04625be7cb77da902930b Author: Shuhei Takahashi <nya@google.com> Date: Fri Oct 20 08:51:19 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/ci_results_archiver/+/ed00935597f44fdaec2d59e7f67ab4d70f494fd3 commit ed00935597f44fdaec2d59e7f67ab4d70f494fd3 Author: Shuhei Takahashi <nya@chromium.org> Date: Fri Oct 20 08:51:18 2017 Do not generate *.pyc files. Leftover *.pyc files are often problematic; for example, pytest raises errors on running unit tests if a absolute path recorded in *.pyc is different from its current apparent path, which is often the case with our chroot build environment. This change tells the Python interpreter not to generate *.pyc files by passing -B option. Also, run_tests is updated not to delete existing *.pyc files so that the command can be run inside portage sandbox. BUG= chromium:733103 TEST=CL:709174 Change-Id: Ibd5e2f2a707ef3bc9fa48c4acc2ed7ef7a8a228c Reviewed-on: https://chromium-review.googlesource.com/711779 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Shuhei Takahashi <nya@chromium.org> [modify] https://crrev.com/ed00935597f44fdaec2d59e7f67ab4d70f494fd3/bin/python_venv [modify] https://crrev.com/ed00935597f44fdaec2d59e7f67ab4d70f494fd3/bin/run_tests
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/manifest-internal/+/2b75a22ca33e01cb26f7896b77e82c16d1752000 commit 2b75a22ca33e01cb26f7896b77e82c16d1752000 Author: Shuhei Takahashi <nya@google.com> Date: Tue Oct 24 05:39:10 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/manifest/+/61a62adedadc8307207aa58a54d65a47f8cacd1f commit 61a62adedadc8307207aa58a54d65a47f8cacd1f Author: Shuhei Takahashi <nya@chromium.org> Date: Tue Oct 24 05:39:04 2017 manifest: Checkout ci_results_archiver in chromeos-admin. BUG= chromium:733103 TEST=None Change-Id: I086508b6bcec4aebb7e201de9eb8ac334d38fc97 Reviewed-on: https://chromium-review.googlesource.com/731743 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/61a62adedadc8307207aa58a54d65a47f8cacd1f/full.xml
ci_results_archiver started running on production. Yay!
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/09fae4e3eef31b7568cebe6887cf02e72398b476 commit 09fae4e3eef31b7568cebe6887cf02e72398b476 Author: Shuhei Takahashi <nya@google.com> Date: Wed Oct 25 05:47:58 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/726b3045ea4ce586f7ded65ab171f06b47bd9846 commit 726b3045ea4ce586f7ded65ab171f06b47bd9846 Author: Shuhei Takahashi <nya@chromium.org> Date: Mon Nov 27 17:05:16 2017 ci-results-archiver: Run unit tests. CQ-DEPEND=CL:776208 BUG= chromium:733103 TEST=./build_packages --board=veyron_minnie TEST=../../chromite/bin/cros_run_unit_tests --board=veyron_minnie \ --packages=chromeos-base/ci-results-archiver Change-Id: I7aa18fac53b6c6a17d33e7e3ce864ae6e6dd51d6 Reviewed-on: https://chromium-review.googlesource.com/709155 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [add] https://crrev.com/726b3045ea4ce586f7ded65ab171f06b47bd9846/chromeos-base/ci-results-archiver/ci-results-archiver-9999.ebuild [rename] https://crrev.com/726b3045ea4ce586f7ded65ab171f06b47bd9846/chromeos-base/autotest-all/autotest-all-0.0.1-r40.ebuild [modify] https://crrev.com/726b3045ea4ce586f7ded65ab171f06b47bd9846/chromeos-base/autotest-all/autotest-all-0.0.1.ebuild
Comment 1 by nya@chromium.org
, Jun 28 2017