New issue
Advanced search Search tips

Issue 915728 link

Starred by 2 users

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Feature



Sign in to add a comment

Buildbucket metric are close to cardinality limit

Project Member Reported by nodir@google.com, Dec 17

Issue description

Buildbucket currently puts a builder to a metric field. This won't scale to the number of Chrome OS builders (tens of thousands), thus prevents onboarding Chrome OS.

ddoman@ mentioned a possibility to create a custom monarch target schema. This needs to be implemented.

Assigning to triage.
 
Cc: mikenichols@chromium.org
Thanks for filing this ticket, nodir@.

Here are some useful info I found

(1) go/monarch-high-cardinality-metrics
>> We **recommend** a limit of 5,000 for metric cardinality for a single target, and 10,000 across all your targets.

(2) Metric fields at collection time
http://go/monarch-high-cardinality-metrics#metric-fields-at-collection-time

>> There are **hard limits** for the amount of data that Monarch will collect in a single scrape from a target: 200,000 cells with the legacy Streamz protocols, and 40MB with the new Streamz protocols.

I am not sure if this limit applies to ProdXMon API because I have seen a ProdXMon API proposal doc to change the data flush model between ProdX and Monarch from polling to pushing. I cannot find the doc by quick searches, but I highly believe that they have already implemented the proposed idea.

(3) How many targets is it safe to create for your custom target schema?
http://go/monarch-high-cardinality-metrics#how-many-targets-is-it-safe-to-create-for-your-custom-target-schema

>> As with metric cardinality, we don't have a hard limit, but the following two limits are suggested ...
>> In general, you should not plan on creating more than 100,000 targets per Monarch zone for your custom target schema.

(4) https://groups.google.com/a/google.com/d/msg/monarch-team/wCydVd3f2Ng/DaYDMx-rBQAJ

>> The 100k limit is **soft** (ie, there is no single breaking point), and there's a good chance that 400k targets would not experience problems, but I can't provide any guarantees, so I would strongly encourage thinking about alternate solutions first.

------------------------------------
With the above information, use of custom schema(s) and storing builder information in target fields will probably provide us enough space for covering Chrome OS builders within the existing ts-mon metrics.


what's the plan/timeline for this feature?
Hi nodir,

I'm planning to work on this, starting Feb. Once my current projects are done, I will write a doc for the plan and required work, and share with you. Do you have information about when ChromeOS will start migrating their builders to BuildBucket?
ChromeOS is already almost entirely on Buildbucket.
When will Chrome OS start generating thousands of builders in cr-buildbucket.cfg?
Cc: la...@chromium.org
Roughly within the next 30-45 days.

Sign in to add a comment