First release processing unacceptably erroneous |
||||
Issue descriptionhttps://chromiumdash.appspot.com/commit/cf1c597e7a016c4d05e68059bd968403e7850516 This commit shows Canary 6262.0.3202.3 Dev 6262.0.3202.12 Stable 6262.0.3202.66 Why is Beta missing? A tooltip over a greyed-out Beta would be helpful.
,
Oct 21 2017
I see something similar with another commit: https://chromiumdash-staging.googleplex.com/commit/74a36cbe1036bdb685578ccd4be9bd60cae83272 only has Dev 62 62.0.3188.2 Stable 62 62.0.3202.62 For Linux.
,
Nov 1 2017
Thanks for re-raising this. I spent some time reviewing the accuracy of our algorithms for processing when a commit was first released and have found they're unacceptably bad for a few recent releases; updating issue summary to match and marking this as a beta blocker. prasadv@, can you PTAL and let me know what you think? Investigation: I put together a list of the most recent versions of Android released for all channels, canary -> stable. I then wrote up and executed a Python script to count the number of commits present in each release based on counting items in gitiles, results here (considered "truth" since it's straight from gitiles): stable 61.0.3163.81: 14734 commits 61.0.3163.98: 134 commits 62.0.3202.66: 11282 commits 62.0.3202.73: 27 commits beta 62.0.3202.19: 10733 commits 62.0.3202.29: 176 commits 62.0.3202.38: 124 commits 62.0.3202.45: 110 commits 62.0.3202.52: 80 commits 63.0.3239.20: 9713 commits dev 63.0.3223.7: 1246 commits 63.0.3233.3: 2639 commits 63.0.3236.6: 693 commits 63.0.3239.10: 1333 commits 63.0.3239.17: 130 commits canary 64.0.3245.0: 326 commits 64.0.3247.0: 421 commits 64.0.3248.0: 297 commits 64.0.3249.0: 330 commits 64.0.3250.0: 362 commits On the Chromium Dash staging console, I then ran Python scripts to count the number of commits which had android_deployment objects set to the version in question (how accurate our algorithms are, if accurate they should match above); the results of that are here: stable 61.0.3163.81: 12716 commits 61.0.3163.98: 134 commits 62.0.3202.66: 900 commits 62.0.3202.73: 27 commits beta 62.0.3202.19: 2250 commits 62.0.3202.29: 176 commits 62.0.3202.38: 124 commits 62.0.3202.45: 110 commits 62.0.3202.52: 80 commits 63.0.3239.20: 9713 commits dev 63.0.3223.7: 1246 commits 63.0.3233.3: 1800 commits 63.0.3236.6: 693 commits 63.0.3239.10: 1333 commits 63.0.3239.17: 130 commits canary 64.0.3245.0: 326 commits 64.0.3247.0: 421 commits 64.0.3248.0: 297 commits 64.0.3249.0: 330 commits 64.0.3250.0: 362 commits In canary we're at 100% accuracy. For dev, we have our first miss: for 63.0.3233.3 we should have 2639 commits, but we instead only have tagged 1800, which is a 30% error rate. Beta looks good aside from a single version, but the version that's bad is *bad*: for 62.0.3202.19 we should have 10733 commits, but instead we have 2250, which is a failure rate of ~80%. On stable, we are good on smaller point releases, but we have some bad misses there as well: 61.0.3163.81 has only 12716 of 14734 commits tagged (14% error rate) and 62.0.3202.66 has only 900 of 11282 commits tagged (93% failure rate). The scripts I ran to get this data are here: https://docs.google.com/a/google.com/document/d/1WLL3UO3_ipBz_RF5P-ukejIUvS3iM3q7Phg1L4dBprs/edit?usp=sharing This is clearly unacceptable so I'm marking this as beta blocking. prasadv@, can you take a first look? Holler if you're curious about the scripts I used or have any other questions...
,
Nov 2 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/infra/infra_internal/+/f8ee63c2dd46bba9fc7fba2b55fc79f77b070c10 commit f8ee63c2dd46bba9fc7fba2b55fc79f77b070c10 Author: prasadv <prasadv@google.com> Date: Thu Nov 02 21:23:47 2017
,
Nov 10 2017
I'm going to mark this as fixed given the CL in c#4 is landed and deployed. We made some tweaks on the staging instance, and re-ran all jobs for *Android only* and now are seeing 100% accuracy there. Data for all new versions (e.g. things we ship after today) should be good as well. dgrogan@, appreciate your reporting this - if you see issues on the staging instance for future versions, please let us know. We'll also be re-running the analysis above periodically in the future as well. Thanks for helping improve Chromium Dash!
,
Nov 10 2017
|
||||
►
Sign in to add a comment |
||||
Comment 1 by amineer@chromium.org
, Oct 20 2017