New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 798448 link

Starred by 1 user
Status: Fixed
Owner:
█ katthomas@chromium.org
Last visit > 30 days ago
Closed: Jan 2018
Components:
EstimatedDays: 1
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

Update bigquery libraries to have a configurable batch size

Project Member Reported by katthomas@chromium.org, Jan 2 2018 
Maximum rows per request: 10,000 rows per request.

eventupload batches rows per request, ensuring that no more than 10,000 rows are sent per request. BigQuery recommends using 500 as a practical limit, and experimenting with your specific schema and data sizes to determine the batch size with the ideal balance of throughput and latency for your use case. There is a bug to modify infra/libs/bigquery to batch rows. We should modify the libraries to allow custom batch sizes, and to use a default batch size of 500.

Comment 1 by katthomas@chromium.org, Jan 2 2018

Status: Started (was: Assigned)
Project Member

Comment 2 by bugdroid1@chromium.org, Jan 5 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/c210229a52f28a6190f6bdb21e9049fc31b49dfe

commit c210229a52f28a6190f6bdb21e9049fc31b49dfe
Author: Katie Thomas <katthomas@google.com>
Date: Fri Jan 05 16:27:25 2018

[infra/libs/bigquery] batch rows

Bug:  798448 
Change-Id: I76be84ed08ad37f8aeebe0b556c3348959be50cf
Reviewed-on: https://chromium-review.googlesource.com/847691
Commit-Queue: Katie Thomas <katthomas@google.com>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/c210229a52f28a6190f6bdb21e9049fc31b49dfe/infra/libs/bigquery/helper.py
[modify] https://crrev.com/c210229a52f28a6190f6bdb21e9049fc31b49dfe/infra/libs/bigquery/test/helper_test.py
Project Member

Comment 3 by bugdroid1@chromium.org, Jan 6 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/ecd05509bd0407cf90194ca08a9b0531b313b7b8

commit ecd05509bd0407cf90194ca08a9b0531b313b7b8
Author: Katie Thomas <katthomas@google.com>
Date: Sat Jan 06 01:21:01 2018

[eventupload] allow batch size to be configured

Bug:798448
Change-Id: Ie4e54dea06e10a6230dec7d8d8ca6bb3d15da912
Reviewed-on: https://chromium-review.googlesource.com/847643
Commit-Queue: Katie Thomas <katthomas@google.com>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/ecd05509bd0407cf90194ca08a9b0531b313b7b8/go/src/infra/libs/eventupload/eventupload.go

Comment 4 by katthomas@chromium.org, Jan 6 2018

Status: Fixed (was: Started)
►

Sign in to add a comment