Design doc is https://docs.google.com/a/google.com/document/d/1fDhJ2suyTsCLXukJwNDqtTAx_hK0hjHrOuUFFR_jDtE/edit?usp=sharing (sorry, internal only). The integration will be through Pub/Sub with well-defined messages. More detail is in the design doc.
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra.git/+/b72a1b8cd1f0d3429ae8e781126596c4242ff23c commit b72a1b8cd1f0d3429ae8e781126596c4242ff23c Author: stgao <stgao@chromium.org> Date: Thu Apr 07 06:31:10 2016 [Findit] A huge refactoring and some bug fixing. The refactoring is a preparation to merge Findit for ClusterFuzz into the findit-for-me app and for the integration with Fracas. Refacoring includes: 1. Separate the waterfall-frontend from the default module. 2. Use dispatch to route requests to the appropriate modules. 3. Refactor wf_analysis_result_status and wf_analysis_status and removed dead code there. 4. Use constants and remove scattered hard-coded strings for queue names, module names, and urls. 5. Setup new task queues with clear names. 6. Use a new app id for testing findit-for-me-dev. Functional change: 1. Update util_scripts/run.sh to: * Enforce USER name as version for testing deployment * Make it possible to deploy to prod for staging testing * Provide commands to update task queue, cron job, and dispatch 2. Add links to step log back to build_failure.html page. Bug fixing: 1. Fix versioned_model when no entity is available yet. 2. Fix a bug in handlers/handlers_util.py BUG= 600460 Review URL: https://codereview.chromium.org/1866883002 [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/app.yaml [add] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/common/constants.py [add] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/common/findit_testcase.py [add] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/dispatch.yaml [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/findit_api.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/handlers/build_failure.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/handlers/check_duplicate_failures.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/handlers/handlers_util.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/handlers/list_analyses.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/handlers/monitor_alerts.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/handlers/result_status.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/handlers/test/build_failure_test.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/handlers/test/check_duplicate_failures_test.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/handlers/test/handlers_util_test.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/handlers/test/list_analyses_test.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/handlers/test/monitor_alerts_test.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/handlers/test/triage_analysis_test.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/handlers/triage_analysis.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/handlers/trigger_analyses.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/main.py [add] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/model/analysis_status.py [rename] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/model/result_status.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/model/test/versioned_model_test.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/model/test/wf_analysis_test.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/model/test/wf_try_job_test.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/model/versioned_model.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/model/wf_analysis.py [delete] https://crrev.com/36fbc27d6caa3951e74848cc7ce9f08f9ea6ad95/appengine/findit/model/wf_analysis_status.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/model/wf_swarming_task.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/model/wf_try_job.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/queue.yaml [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/templates/build_failure.html [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/templates/config.html [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/templates/home.html [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/templates/list_analyses.html [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/test/findit_api_test.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/util_scripts/remote_queries/swarming_task_data_query.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/util_scripts/run.sh [rename] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/waterfall-backend.yaml [add] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/waterfall-frontend.yaml [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/waterfall/analyze_build_failure_pipeline.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/waterfall/build_failure_analysis_pipelines.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/waterfall/identify_culprit_pipeline.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/waterfall/identify_try_job_culprit_pipeline.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/waterfall/monitor_try_job_pipeline.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/waterfall/process_swarming_task_result_pipeline.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/waterfall/run_try_job_for_reliable_failure_pipeline.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/waterfall/test/analyze_build_failure_pipeline_test.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/waterfall/test/build_failure_analysis_pipelines_test.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/waterfall/test/detect_first_failure_pipeline_test.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/waterfall/test/identify_culprit_pipeline_test.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/waterfall/test/identify_try_job_culprit_pipeline_test.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/waterfall/test/monitor_try_job_pipeline_test.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/waterfall/test/process_swarming_task_result_pipeline_test.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/waterfall/test/run_try_job_for_reliable_failure_pipeline_test.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/waterfall/test/trigger_swarming_task_pipeline_test.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/waterfall/test/try_job_pipeline_test.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/waterfall/test/try_job_util_test.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/waterfall/test/wf_testcase.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/waterfall/trigger_swarming_task_pipeline.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/waterfall/try_job_pipeline.py [modify] https://crrev.com/b72a1b8cd1f0d3429ae8e781126596c4242ff23c/appengine/findit/waterfall/try_job_util.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra.git/+/0c1485443c65aeab3e432bb86ecbe583cb7bb989 commit 0c1485443c65aeab3e432bb86ecbe583cb7bb989 Author: stgao <stgao@chromium.org> Date: Thu Apr 14 23:25:15 2016 [Findit] Integrate with Fracas through Pub/Sub. Main changes are: 1. Add a new frontend to handle crashes, and update dispatching rule accordingly. 2. Add a new backend and a new task queue to run analysis of crashes from Fracas. 3. Add crash configration(NDB module, handler, and UI template). 4. Add a handler to accept Pub/Sub push of Fracas messages. 5. Add the NDB model to save Fracas crashes and the analysis result. 6. Add the pipeline to run the analysis and publish results to a configured topic. 7. Update util_scripts/run.sh to set the app id through ENV for testing deployment. A demo client app to mimic Fracas is in https://chromereviews.googleplex.com/392327013/. The implementation is according to design doc https://docs.google.com/a/google.com/document/d/1fDhJ2suyTsCLXukJwNDqtTAx_hK0hjHrOuUFFR_jDtE/edit?usp=sharing BUG= 600460 Review URL: https://codereview.chromium.org/1852383002 [modify] https://crrev.com/0c1485443c65aeab3e432bb86ecbe583cb7bb989/appengine/findit/common/constants.py [add] https://crrev.com/0c1485443c65aeab3e432bb86ecbe583cb7bb989/appengine/findit/common/pubsub_util.py [add] https://crrev.com/0c1485443c65aeab3e432bb86ecbe583cb7bb989/appengine/findit/crash-backend-fracas.yaml [add] https://crrev.com/0c1485443c65aeab3e432bb86ecbe583cb7bb989/appengine/findit/crash-frontend.yaml [add] https://crrev.com/0c1485443c65aeab3e432bb86ecbe583cb7bb989/appengine/findit/crash/fracas.py [add] https://crrev.com/0c1485443c65aeab3e432bb86ecbe583cb7bb989/appengine/findit/crash/fracas_crash_pipeline.py [add] https://crrev.com/0c1485443c65aeab3e432bb86ecbe583cb7bb989/appengine/findit/crash/test/crash_testcase.py [add] https://crrev.com/0c1485443c65aeab3e432bb86ecbe583cb7bb989/appengine/findit/crash/test/fracas_crash_pipeline_test.py [modify] https://crrev.com/0c1485443c65aeab3e432bb86ecbe583cb7bb989/appengine/findit/dispatch.yaml [add] https://crrev.com/0c1485443c65aeab3e432bb86ecbe583cb7bb989/appengine/findit/handlers/crash/__init__.py [add] https://crrev.com/0c1485443c65aeab3e432bb86ecbe583cb7bb989/appengine/findit/handlers/crash/crash_config.py [add] https://crrev.com/0c1485443c65aeab3e432bb86ecbe583cb7bb989/appengine/findit/handlers/crash/fracas_crash.py [add] https://crrev.com/0c1485443c65aeab3e432bb86ecbe583cb7bb989/appengine/findit/handlers/crash/test/__init__.py [add] https://crrev.com/0c1485443c65aeab3e432bb86ecbe583cb7bb989/appengine/findit/handlers/crash/test/fracas_crash_test.py [modify] https://crrev.com/0c1485443c65aeab3e432bb86ecbe583cb7bb989/appengine/findit/main.py [add] https://crrev.com/0c1485443c65aeab3e432bb86ecbe583cb7bb989/appengine/findit/model/crash/__init__.py [add] https://crrev.com/0c1485443c65aeab3e432bb86ecbe583cb7bb989/appengine/findit/model/crash/crash_analysis.py [add] https://crrev.com/0c1485443c65aeab3e432bb86ecbe583cb7bb989/appengine/findit/model/crash/crash_config.py [add] https://crrev.com/0c1485443c65aeab3e432bb86ecbe583cb7bb989/appengine/findit/model/crash/fracas_crash_analysis.py [add] https://crrev.com/0c1485443c65aeab3e432bb86ecbe583cb7bb989/appengine/findit/model/crash/test/__init__.py [add] https://crrev.com/0c1485443c65aeab3e432bb86ecbe583cb7bb989/appengine/findit/model/crash/test/crash_analysis_test.py [add] https://crrev.com/0c1485443c65aeab3e432bb86ecbe583cb7bb989/appengine/findit/model/crash/test/fracas_crash_analysis_test.py [modify] https://crrev.com/0c1485443c65aeab3e432bb86ecbe583cb7bb989/appengine/findit/queue.yaml [add] https://crrev.com/0c1485443c65aeab3e432bb86ecbe583cb7bb989/appengine/findit/templates/crash/crash_config.html [add] https://crrev.com/0c1485443c65aeab3e432bb86ecbe583cb7bb989/appengine/findit/third_party/googleapiclient [add] https://crrev.com/0c1485443c65aeab3e432bb86ecbe583cb7bb989/appengine/findit/third_party/httplib2 [add] https://crrev.com/0c1485443c65aeab3e432bb86ecbe583cb7bb989/appengine/findit/third_party/oauth2client [add] https://crrev.com/0c1485443c65aeab3e432bb86ecbe583cb7bb989/appengine/findit/third_party/six.py [add] https://crrev.com/0c1485443c65aeab3e432bb86ecbe583cb7bb989/appengine/findit/third_party/uritemplate [modify] https://crrev.com/0c1485443c65aeab3e432bb86ecbe583cb7bb989/appengine/findit/util_scripts/run.sh
As the main work for the integration was done, I'd close this bug. Sharu will work out the message data format with Fracas team when the analysis code is hooked up with appengine/findit/crash/fracas.py in the CL above.
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra.git/+/24b1aae2d3e97b2fc14deae85b73691535157aae commit 24b1aae2d3e97b2fc14deae85b73691535157aae Author: stgao <stgao@chromium.org> Date: Tue May 03 05:52:13 2016 [Findit] Use PubSub built-in authentication for push endpoint. After b/25788912, subscription in the same project as the push endpoint could bypass the authentication as shown in this CL. The authentication approach is described in https://cloud.google.com/pubsub/prereqs. This approach is verified through dummy apps on both appspot and googleplex. That means Fracas could also do push delivery instead of pull. BUG= 600460 Review-Url: https://codereview.chromium.org/1940843003 [modify] https://crrev.com/24b1aae2d3e97b2fc14deae85b73691535157aae/appengine/findit/crash-frontend.yaml [modify] https://crrev.com/24b1aae2d3e97b2fc14deae85b73691535157aae/appengine/findit/crash/test/crash_testcase.py [modify] https://crrev.com/24b1aae2d3e97b2fc14deae85b73691535157aae/appengine/findit/handlers/crash/fracas_crash.py [modify] https://crrev.com/24b1aae2d3e97b2fc14deae85b73691535157aae/appengine/findit/handlers/crash/test/fracas_crash_test.py [modify] https://crrev.com/24b1aae2d3e97b2fc14deae85b73691535157aae/appengine/findit/main.py [modify] https://crrev.com/24b1aae2d3e97b2fc14deae85b73691535157aae/appengine/findit/model/crash/crash_config.py
Comment 1 by bugdroid1@chromium.org
, Apr 7 2016