Buildbucket metric are close to cardinality limit |
||
Issue descriptionBuildbucket currently puts a builder to a metric field. This won't scale to the number of Chrome OS builders (tens of thousands), thus prevents onboarding Chrome OS. ddoman@ mentioned a possibility to create a custom monarch target schema. This needs to be implemented. Assigning to triage.
,
Dec 18
Thanks for filing this ticket, nodir@. Here are some useful info I found (1) go/monarch-high-cardinality-metrics >> We **recommend** a limit of 5,000 for metric cardinality for a single target, and 10,000 across all your targets. (2) Metric fields at collection time http://go/monarch-high-cardinality-metrics#metric-fields-at-collection-time >> There are **hard limits** for the amount of data that Monarch will collect in a single scrape from a target: 200,000 cells with the legacy Streamz protocols, and 40MB with the new Streamz protocols. I am not sure if this limit applies to ProdXMon API because I have seen a ProdXMon API proposal doc to change the data flush model between ProdX and Monarch from polling to pushing. I cannot find the doc by quick searches, but I highly believe that they have already implemented the proposed idea. (3) How many targets is it safe to create for your custom target schema? http://go/monarch-high-cardinality-metrics#how-many-targets-is-it-safe-to-create-for-your-custom-target-schema >> As with metric cardinality, we don't have a hard limit, but the following two limits are suggested ... >> In general, you should not plan on creating more than 100,000 targets per Monarch zone for your custom target schema. (4) https://groups.google.com/a/google.com/d/msg/monarch-team/wCydVd3f2Ng/DaYDMx-rBQAJ >> The 100k limit is **soft** (ie, there is no single breaking point), and there's a good chance that 400k targets would not experience problems, but I can't provide any guarantees, so I would strongly encourage thinking about alternate solutions first. ------------------------------------ With the above information, use of custom schema(s) and storing builder information in target fields will probably provide us enough space for covering Chrome OS builders within the existing ts-mon metrics.
,
Jan 11
what's the plan/timeline for this feature?
,
Jan 11
Hi nodir, I'm planning to work on this, starting Feb. Once my current projects are done, I will write a doc for the plan and required work, and share with you. Do you have information about when ChromeOS will start migrating their builders to BuildBucket?
,
Jan 12
ChromeOS is already almost entirely on Buildbucket.
,
Jan 13
When will Chrome OS start generating thousands of builders in cr-buildbucket.cfg?
,
Jan 14
Roughly within the next 30-45 days. |
||
►
Sign in to add a comment |
||
Comment 1 by jclinton@chromium.org
, Dec 17