New issue
Advanced search Search tips

Issue 841407 link

Starred by 1 user
Status: Available
Owner: ----
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Feature



Sign in to add a comment

Batch datastore Put operations whenever putting multiple entities in parallel

Project Member Reported by qyears...@chromium.org, May 9 2018 
In general, when putting multiple entities in the datastore, the put operations can be batched together as: `datastore.Put(ctx, entity1, entity2)`.

Context: https://chromium-review.googlesource.com/c/infra/infra/+/1048766/2/go/src/infra/tricium/appengine/frontend/rpc_analyze.go#215

The advantage of batching is possible performance gain. I think that in some cases, keeping datastore Puts separate may make some the logic clearer, so when batching them together we should make sure that (1) the logic is still clear and (2) it actually improves performance.

Some places where potential improvements could be made include:

 - frontend/rpc_analyze.go (putting GerritChangeIDtoRunID and AnalyzeRequestResult)
 - gerrit/poll.go (putting entities related to changes and projects)
 - and maybe others.

Comment 1 by no...@chromium.org, May 9 2018 

I'd like to explicitly mention that in some cases batching instead of concurrency might also simplify code: if entity1 and entity2 computations/modifications are independent of each other, but very lightweight, there is little value parallelizing that:

entity1 := ... // compute entity1
entity2 := ... // compute entity2
// Strictly speaking, these two lines could be parallelized, but same applies to the majority of code overall, so it is not 
// enough justification to parallelize it
datastore.Put(c, entity1, entity2)

Comment 2 by qyears...@chromium.org, May 15 2018

Labels: Hotlist-CodeHealth
Summary: Batch datastore Put operations whenever putting multiple entities in parallel (was: Batch datastore Put operations whenever putting multiple entities in parallel.)

Comment 3 by qyears...@chromium.org, Jun 15 2018

Components: Infra>Platform>Tricium

Comment 4 by qyears...@chromium.org, Jun 15 2018

Components: -Infra>CodeAnalysis
Labels: -Tricium
►

Sign in to add a comment