[Findit] Backfill existing analysis info to BQ |
||
Issue descriptionSend existing data to BQ via a remote_api. There's a total of 350G of data * 7500 Flake Analysis * 204000 WfAnalyses Use cursors to continue in manageable chunks.
,
Mar 2 2018
,
Mar 6 2018
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/1ef044f1c1ba21c7fffcb85b668e2ea9b6127b78 commit 1ef044f1c1ba21c7fffcb85b668e2ea9b6127b78 Author: Brandon Wylie <wylieb@chromium.org> Date: Tue Mar 06 22:28:37 2018 [Findit] Backfill existing analysis data to BigQuery Upload historical analysis events to BigQuery. A full design for BigQuery events and the backfill itself can be found at go/findit-bq-events. This script is run locally, but is probably useful to commit in case backfilling is required in the future. Bug:809315 Change-Id: I9bc7978b6d4e129fbf7aa87b48ca96ed703d332e Reviewed-on: https://chromium-review.googlesource.com/905703 Commit-Queue: Brandon Wylie <wylieb@chromium.org> Reviewed-by: Jeffrey Li <lijeffrey@chromium.org> Reviewed-by: Shuotao Gao <stgao@chromium.org> [add] https://crrev.com/1ef044f1c1ba21c7fffcb85b668e2ea9b6127b78/appengine/findit/util_scripts/remote_queries/backfill_bq_events.py [delete] https://crrev.com/1bf6b06fbd5a49d3979e37b1ef945f45b5bab8d4/appengine/findit/util_scripts/remote_queries/bq_event_example.py |
||
►
Sign in to add a comment |
||
Comment 1 by wylieb@chromium.org
, Feb 6 2018