Deploy an appengine service to return the probability that a CL is a bad CL.
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/bad_cl_detector/+/9ad799a148ab563df2a4790a54dfa5ae27289289 commit 9ad799a148ab563df2a4790a54dfa5ae27289289 Author: Paul Hobbs <phobbs@google.com> Date: Tue Nov 28 23:23:34 2017 Add .gitignore TEST=None BUG= chromium:789287 Change-Id: I24e92e01335547ac5821ca6a01a5d515841368c9 [add] https://crrev.com/9ad799a148ab563df2a4790a54dfa5ae27289289/.gitignore
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/bad_cl_detector/+/0f6bfacaca34dc0275b988da77703ea59cf163cb commit 0f6bfacaca34dc0275b988da77703ea59cf163cb Author: Paul Hobbs <phobbs@google.com> Date: Tue Nov 28 23:24:00 2017 Added symlinks and __init__.py TEST=None BUG= chromium:789287 Change-Id: I44cdd704e6315983fef586ca616e623c3f1a18f2 [add] https://crrev.com/0f6bfacaca34dc0275b988da77703ea59cf163cb/model/__init__.py [add] https://crrev.com/0f6bfacaca34dc0275b988da77703ea59cf163cb/lib/model [add] https://crrev.com/0f6bfacaca34dc0275b988da77703ea59cf163cb/lib/chromite
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/bad_cl_detector/+/9f451cb7dab8e881e46645eb4550f4b069b978f6 commit 9f451cb7dab8e881e46645eb4550f4b069b978f6 Author: Paul Hobbs <phobbs@google.com> Date: Tue Nov 28 23:24:05 2017 Fix stage_feature_extractor_integration_test Also, added a pytest.ini TEST=ran the test BUG= chromium:789287 Change-Id: Ib98e504fb834032059bdf87c468529578a6e41ee [add] https://crrev.com/9f451cb7dab8e881e46645eb4550f4b069b978f6/pytest.ini [modify] https://crrev.com/9f451cb7dab8e881e46645eb4550f4b069b978f6/lib/stage_feature_extractor_integration_test.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/bad_cl_detector/+/6af58811d71d6336f63a30631fc71b9ebae500b9 commit 6af58811d71d6336f63a30631fc71b9ebae500b9 Author: Paul Hobbs <phobbs@google.com> Date: Tue Nov 28 23:24:06 2017 fetch_data_engine: Use verbose flag for gerrit Use verbose=True for fetching change details from gerrit. This requests all patch details instead of just the most recent. TEST=Fetching data fails without this, succeeds with. BUG= chromium:789287 Change-Id: Iba640897ae8116c65b94319453deac06cfa982d6 [modify] https://crrev.com/6af58811d71d6336f63a30631fc71b9ebae500b9/lib/fetch_data_engine.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/bad_cl_detector/+/fc29867a8bea9df606aceb9bbfeff0e0c2f97cf1 commit fc29867a8bea9df606aceb9bbfeff0e0c2f97cf1 Author: Paul Hobbs <phobbs@google.com> Date: Tue Nov 28 23:24:08 2017 fetch_data_engine: Don't assume unresolved_comment_count unresolved_comment_count is not guaranteed to be present in the change info's keys. Also, fix up some logging levels. BUG= chromium:789287 TEST=None Change-Id: I1de3a900aa4a30ff2b06c6d777d9674718da1032 [modify] https://crrev.com/fc29867a8bea9df606aceb9bbfeff0e0c2f97cf1/lib/fetch_data_engine.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/bad_cl_detector/+/0487082e292de49af1fae8d1cac08f858d43f500 commit 0487082e292de49af1fae8d1cac08f858d43f500 Author: Paul Hobbs <phobbs@google.com> Date: Tue Nov 28 23:24:10 2017 Handle non-existant CLs Sometimes CLs don't exist. Handle None ChangeDetails. BUG= chromium:789287 TEST=Ran it Change-Id: I7fca49f506468d8b9fbd5ac496ae92766aede613 [modify] https://crrev.com/0487082e292de49af1fae8d1cac08f858d43f500/lib/fetch_data_engine.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/bad_cl_detector/+/acfad1cc0adefb254ce4b204759ab4dc44e4ba5e commit acfad1cc0adefb254ce4b204759ab4dc44e4ba5e Author: Paul Hobbs <phobbs@google.com> Date: Tue Nov 28 23:24:11 2017 Mute cidb INFO messages. TEST=None BUG= chromium:789287 Change-Id: I539491208953ce92ca903bb102edcf3be249928e [modify] https://crrev.com/acfad1cc0adefb254ce4b204759ab4dc44e4ba5e/lib/bad_cl_detector_driver.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/bad_cl_detector/+/a447d24ee80dcbcf0bd0bafeb1ec2dbf45c110a9 commit a447d24ee80dcbcf0bd0bafeb1ec2dbf45c110a9 Author: Paul Hobbs <phobbs@google.com> Date: Tue Nov 28 23:24:13 2017 Remove useless IOError catch BUG= chromium:789287 TEST=None Change-Id: I1a56b56351a9124f239062487cfba771f904266a [modify] https://crrev.com/a447d24ee80dcbcf0bd0bafeb1ec2dbf45c110a9/lib/cl_feature_extractor.py [modify] https://crrev.com/a447d24ee80dcbcf0bd0bafeb1ec2dbf45c110a9/lib/feature_preprocessor.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/bad_cl_detector/+/e7cfe061780721725f10134489626a7885cdd52c commit e7cfe061780721725f10134489626a7885cdd52c Author: Paul Hobbs <phobbs@google.com> Date: Tue Nov 28 23:24:15 2017 Added Pipfiles. Pipfile is the equivalent of requirements.txt for "pipenv," the New Way to do Virtualenvs Right (officially recommended by python.org). See: https://github.com/kennethreitz/pipenv BUG= chromium:789287 TEST=Made a pipenv, ran bad_cl_detector_driver.py Change-Id: I098d0b732e3a818e498d5af15130649f622d6aab [add] https://crrev.com/e7cfe061780721725f10134489626a7885cdd52c/Pipfile.lock [add] https://crrev.com/e7cfe061780721725f10134489626a7885cdd52c/Pipfile
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/bad_cl_detector/+/b50e8f278c9db338075a589b3a17df4b6d5d1707 commit b50e8f278c9db338075a589b3a17df4b6d5d1707 Author: Paul Hobbs <phobbs@google.com> Date: Tue Nov 28 23:24:16 2017 Simplify "GetFilenames" function TEST=None BUG= chromium:789287 Change-Id: Idee909a8cfe52a8b5f1c8fa87b2c3a5df950c836 [modify] https://crrev.com/b50e8f278c9db338075a589b3a17df4b6d5d1707/lib/feature_extractor_utils.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/bad_cl_detector/+/b2736ce497c8831955e0e13a58de0cf0c028aae1 commit b2736ce497c8831955e0e13a58de0cf0c028aae1 Author: Paul Hobbs <phobbs@google.com> Date: Tue Nov 28 23:24:17 2017 Renamed *unittest -> *test Allows using pytest with default file regex. BUG= chromium:789287 TEST=pytest Change-Id: I6acd945ce95b7b6a7808875589630983be2e917d [rename] https://crrev.com/b2736ce497c8831955e0e13a58de0cf0c028aae1/lib/cl_feature_extractor_test.py [rename] https://crrev.com/b2736ce497c8831955e0e13a58de0cf0c028aae1/lib/fetch_data_engine_test.py [rename] https://crrev.com/b2736ce497c8831955e0e13a58de0cf0c028aae1/lib/feature_extractor_utils_test.py [rename] https://crrev.com/b2736ce497c8831955e0e13a58de0cf0c028aae1/lib/stage_feature_extractor_test.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/bad_cl_detector/+/b457e0639c489e6c58ca36f045af4c771fbddcf5 commit b457e0639c489e6c58ca36f045af4c771fbddcf5 Author: Paul Hobbs <phobbs@google.com> Date: Tue Nov 28 23:24:18 2017 Remove runner.py; use pytest. BUG= chromium:789287 TEST=Tests pass Change-Id: Ibaa34e5485b6f9fe7c34de7c252167c92f342492 [delete] https://crrev.com/b2736ce497c8831955e0e13a58de0cf0c028aae1/runner.py [modify] https://crrev.com/b457e0639c489e6c58ca36f045af4c771fbddcf5/lib/feature_extractor_utils_test.py [modify] https://crrev.com/b457e0639c489e6c58ca36f045af4c771fbddcf5/bin/run_tests
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/bad_cl_detector/+/52d340e83097b895a71a17885428945ed0ad1225 commit 52d340e83097b895a71a17885428945ed0ad1225 Author: Paul Hobbs <phobbs@google.com> Date: Tue Nov 28 23:24:19 2017 Rewrite README, add protobuf to Pipfile TEST=None BUG= chromium:789287 Change-Id: I2bb9d9d70d30cc310cbf0f8c3bde231d73542288 [modify] https://crrev.com/52d340e83097b895a71a17885428945ed0ad1225/Pipfile.lock [modify] https://crrev.com/52d340e83097b895a71a17885428945ed0ad1225/Pipfile [modify] https://crrev.com/52d340e83097b895a71a17885428945ed0ad1225/README.md
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/bad_cl_detector/+/697ff81482adccd19c74297f67a752d0eac9c10b commit 697ff81482adccd19c74297f67a752d0eac9c10b Author: Paul Hobbs <phobbs@google.com> Date: Tue Nov 28 23:24:21 2017 Lint cleanups BUG= chromium:789287 TEST=None Change-Id: I3ca2722802b5063ad44e26eac14c2ab62552c5f7 [modify] https://crrev.com/697ff81482adccd19c74297f67a752d0eac9c10b/lib/stage_feature_extractor.py [modify] https://crrev.com/697ff81482adccd19c74297f67a752d0eac9c10b/lib/feature_feeder.py [modify] https://crrev.com/697ff81482adccd19c74297f67a752d0eac9c10b/lib/bad_cl_detector_driver.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/bad_cl_detector/+/6bb8a1151b4f86c69dba2c3efd9855ccad8dfe20 commit 6bb8a1151b4f86c69dba2c3efd9855ccad8dfe20 Author: Paul Hobbs <phobbs@google.com> Date: Tue Nov 28 23:35:05 2017 Add dev dependencies for IPython notebooks BUG= chromium:789287 TEST=None Change-Id: I91f68b4fc9ddf6ad4ab9108277e1608eb0fb1b91 [modify] https://crrev.com/6bb8a1151b4f86c69dba2c3efd9855ccad8dfe20/Pipfile.lock [modify] https://crrev.com/6bb8a1151b4f86c69dba2c3efd9855ccad8dfe20/Pipfile [add] https://crrev.com/6bb8a1151b4f86c69dba2c3efd9855ccad8dfe20/lib/__init__.py [modify] https://crrev.com/6bb8a1151b4f86c69dba2c3efd9855ccad8dfe20/.gitignore
This is in production now.
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/824cb904f306ae5bd66b9e4e7d3371f791cb1117 commit 824cb904f306ae5bd66b9e4e7d3371f791cb1117 Author: Paul Hobbs <phobbs@google.com> Date: Fri Jan 12 03:37:44 2018 report_stages: Add CL-Scanner report BUG= chromium:789287 TEST=report_stages unittest Change-Id: I65f2f460e739aa5537ab4d675cce976c0e1a378c Reviewed-on: https://chromium-review.googlesource.com/837702 Commit-Ready: Paul Hobbs <phobbs@google.com> Tested-by: Paul Hobbs <phobbs@google.com> Reviewed-by: Paul Hobbs <phobbs@google.com> [add] https://crrev.com/824cb904f306ae5bd66b9e4e7d3371f791cb1117/lib/risk_report_unittest.py [add] https://crrev.com/824cb904f306ae5bd66b9e4e7d3371f791cb1117/lib/risk_report.py [add] https://crrev.com/824cb904f306ae5bd66b9e4e7d3371f791cb1117/lib/risk_report_unittest [modify] https://crrev.com/824cb904f306ae5bd66b9e4e7d3371f791cb1117/cbuildbot/stages/report_stages_unittest.py [modify] https://crrev.com/824cb904f306ae5bd66b9e4e7d3371f791cb1117/cbuildbot/stages/report_stages.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/bad_cl_detector/+/10ce327568ca25b8c5be4da97e1de25cf69f116a commit 10ce327568ca25b8c5be4da97e1de25cf69f116a Author: Paul Hobbs <phobbs@google.com> Date: Sat Jan 13 06:37:43 2018 Added GBM model evaluation functions TEST=Used it BUG= chromium:789287 Change-Id: I80616362fdcbcabe07ec5e5e66ee5bab81630ccc Reviewed-on: https://chromium-review.googlesource.com/794027 Commit-Ready: Paul Hobbs <phobbs@google.com> Tested-by: Paul Hobbs <phobbs@google.com> Reviewed-by: Xixuan Wu <xixuan@chromium.org> [add] https://crrev.com/10ce327568ca25b8c5be4da97e1de25cf69f116a/lib/trees.py [modify] https://crrev.com/10ce327568ca25b8c5be4da97e1de25cf69f116a/model/tf_models.py [add] https://crrev.com/10ce327568ca25b8c5be4da97e1de25cf69f116a/creds/README.md
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/bad_cl_detector/+/58a69f050d525a20e7dd5f2d94385179ccc307c8 commit 58a69f050d525a20e7dd5f2d94385179ccc307c8 Author: Paul Hobbs <phobbs@google.com> Date: Sat Jan 13 06:37:44 2018 Add route for classifying CLs in a build TEST=ran local server with gunicorn BUG= chromium:789287 Change-Id: If0c770696de30e25f00a5280bc02c1dd70a895ac Reviewed-on: https://chromium-review.googlesource.com/794028 Commit-Ready: Paul Hobbs <phobbs@google.com> Tested-by: Paul Hobbs <phobbs@google.com> Reviewed-by: Xixuan Wu <xixuan@chromium.org> [modify] https://crrev.com/58a69f050d525a20e7dd5f2d94385179ccc307c8/Pipfile.lock [modify] https://crrev.com/58a69f050d525a20e7dd5f2d94385179ccc307c8/lib/trees.py [modify] https://crrev.com/58a69f050d525a20e7dd5f2d94385179ccc307c8/lib/fetch_data_engine.py [add] https://crrev.com/58a69f050d525a20e7dd5f2d94385179ccc307c8/deploy_app.py [modify] https://crrev.com/58a69f050d525a20e7dd5f2d94385179ccc307c8/.gitignore [add] https://crrev.com/58a69f050d525a20e7dd5f2d94385179ccc307c8/app.yaml [modify] https://crrev.com/58a69f050d525a20e7dd5f2d94385179ccc307c8/Pipfile [add] https://crrev.com/58a69f050d525a20e7dd5f2d94385179ccc307c8/chromite [add] https://crrev.com/58a69f050d525a20e7dd5f2d94385179ccc307c8/gunicorn.conf.py [add] https://crrev.com/58a69f050d525a20e7dd5f2d94385179ccc307c8/main.py [delete] https://crrev.com/10ce327568ca25b8c5be4da97e1de25cf69f116a/lib/chromite
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/bad_cl_detector/+/0288996a7e37a28130d1ec7dfc3551b5a9233616 commit 0288996a7e37a28130d1ec7dfc3551b5a9233616 Author: Paul Hobbs <phobbs@google.com> Date: Sat Jan 13 06:37:43 2018 notebooks: Got 0.89 AUC with GBM TEST=experimenting with jupyter notebooks BUG= chromium:789287 Change-Id: I1192f8f4b338e14e06e249779f6ef0df6e2676a1 Reviewed-on: https://chromium-review.googlesource.com/794297 Commit-Ready: Paul Hobbs <phobbs@google.com> Tested-by: Paul Hobbs <phobbs@google.com> Reviewed-by: Xixuan Wu <xixuan@chromium.org> [add] https://crrev.com/0288996a7e37a28130d1ec7dfc3551b5a9233616/notebooks/Linear-on-word2vec.ipynb [add] https://crrev.com/0288996a7e37a28130d1ec7dfc3551b5a9233616/notebooks/Experimenting-LightGBM.ipynb [add] https://crrev.com/0288996a7e37a28130d1ec7dfc3551b5a9233616/notebooks/Experimenting-GBF.ipynb [add] https://crrev.com/0288996a7e37a28130d1ec7dfc3551b5a9233616/notebooks/Linear-on-individual-words.ipynb [add] https://crrev.com/0288996a7e37a28130d1ec7dfc3551b5a9233616/notebooks/Experimenting-GBF-RandomProjections.ipynb
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/bad_cl_detector/+/07f0089adebff56a993c89ee9900d46f179adda8 commit 07f0089adebff56a993c89ee9900d46f179adda8 Author: Paul Hobbs <phobbs@google.com> Date: Sat Jan 13 06:37:44 2018 Use absolute imports TEST=Tests pass BUG= chromium:789287 Change-Id: If0dcd6d788e83aa4a3bdf2331012c5a242523064 Reviewed-on: https://chromium-review.googlesource.com/794901 Commit-Ready: Paul Hobbs <phobbs@google.com> Tested-by: Paul Hobbs <phobbs@google.com> Reviewed-by: Xixuan Wu <xixuan@chromium.org> [modify] https://crrev.com/07f0089adebff56a993c89ee9900d46f179adda8/lib/fetch_data_engine.py [modify] https://crrev.com/07f0089adebff56a993c89ee9900d46f179adda8/lib/feature_extractor_utils_test.py [modify] https://crrev.com/07f0089adebff56a993c89ee9900d46f179adda8/lib/stage_feature_extractor.py [modify] https://crrev.com/07f0089adebff56a993c89ee9900d46f179adda8/lib/bad_cl_detector_driver.py [modify] https://crrev.com/07f0089adebff56a993c89ee9900d46f179adda8/lib/stage_feature_extractor_test.py [modify] https://crrev.com/07f0089adebff56a993c89ee9900d46f179adda8/lib/feature_extractor_utils.py [modify] https://crrev.com/07f0089adebff56a993c89ee9900d46f179adda8/lib/fetch_data_engine_test.py [modify] https://crrev.com/07f0089adebff56a993c89ee9900d46f179adda8/lib/feature_preprocessor.py [modify] https://crrev.com/07f0089adebff56a993c89ee9900d46f179adda8/lib/feature_feeder.py [modify] https://crrev.com/07f0089adebff56a993c89ee9900d46f179adda8/lib/cl_feature_extractor.py [modify] https://crrev.com/07f0089adebff56a993c89ee9900d46f179adda8/lib/cl_feature_extractor_test.py [modify] https://crrev.com/07f0089adebff56a993c89ee9900d46f179adda8/lib/stage_feature_extractor_integration_test.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/bad_cl_detector/+/899f13f9936df9c3cd83e8452bea1be4989c62ff commit 899f13f9936df9c3cd83e8452bea1be4989c62ff Author: Paul Hobbs <phobbs@google.com> Date: Sat Jan 13 06:37:44 2018 Add more imports required for tests Run pytest within the virtualenv with "pipenv run pytest". Also, ignore glint errors if the binary doesn't exist. External developers or Mac OS X developers don't have glint installed. TEST=Unit tests pass with the proper pytest binary now. BUG= chromium:789287 Change-Id: I06506811a60c3446f1d320ac83b7b77bdce1085e Reviewed-on: https://chromium-review.googlesource.com/794902 Commit-Ready: Paul Hobbs <phobbs@google.com> Tested-by: Paul Hobbs <phobbs@google.com> Reviewed-by: Xixuan Wu <xixuan@chromium.org> [modify] https://crrev.com/899f13f9936df9c3cd83e8452bea1be4989c62ff/Pipfile.lock [modify] https://crrev.com/899f13f9936df9c3cd83e8452bea1be4989c62ff/Pipfile [modify] https://crrev.com/899f13f9936df9c3cd83e8452bea1be4989c62ff/bin/run_tests [modify] https://crrev.com/899f13f9936df9c3cd83e8452bea1be4989c62ff/bin/run_lint
Comment 1 by pho...@chromium.org
, Nov 28 2017