New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 777010 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit 27 days ago
Closed: Nov 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

First release processing unacceptably erroneous

Project Member Reported by dgro...@chromium.org, Oct 20 2017

Issue description

https://chromiumdash.appspot.com/commit/cf1c597e7a016c4d05e68059bd968403e7850516

This commit shows

Canary  6262.0.3202.3
Dev  6262.0.3202.12
Stable  6262.0.3202.66

Why is Beta missing? A tooltip over a greyed-out Beta would be helpful.
 
Screenshot from 2017-10-20 15:40:59.png
56.8 KB View Download
Status: Fixed (was: Unassigned)
Blah, sorry - the explanation is, there's a bug in our backend which processes data related to which commit has been released where - it seems to affect ~1% of commits, though let me know if you've seen it more frequently.  This change has clearly already been released to beta, its omission is erroneous.

We have a few KIs here and we've been hard at work on fixing them; we've actually completely rewritten the entire backend.  That said, because it's been so much work, we've been doing it behind the scenes in a different app, with the goal of flipping the new code to production in the next 1 month or so.  You can see in the new version of the backend we're working on now, things are working as expected for this commit: https://chromiumdash-staging.googleplex.com/commit/cf1c597e7a016c4d05e68059bd968403e7850516

I'm going to mark this as fixed, because we do see the proper data with the new code in the staging instance.  That said, if you see this in other places, or have any questions, please feel free to re-open this bug.  Thanks for sending feedback!
Status: Unconfirmed (was: Fixed)
I see something similar with another commit:
https://chromiumdash-staging.googleplex.com/commit/74a36cbe1036bdb685578ccd4be9bd60cae83272

only has

Dev 62 62.0.3188.2
Stable 62 62.0.3202.62

For Linux.
Labels: Beta-Blocking
Owner: pras...@chromium.org
Status: Assigned (was: Unconfirmed)
Summary: First release processing unacceptably erroneous (was: No explanation for why a channel is missing)
Thanks for re-raising this.  I spent some time reviewing the accuracy of our algorithms for processing when a commit was first released and have found they're unacceptably bad for a few recent releases; updating issue summary to match and marking this as a beta blocker.  prasadv@, can you PTAL and let me know what you think?

Investigation: I put together a list of the most recent versions of Android released for all channels, canary -> stable.  I then wrote up and executed a Python script to count the number of commits present in each release based on counting items in gitiles, results here (considered "truth" since it's straight from gitiles): 

stable
61.0.3163.81: 14734 commits
61.0.3163.98: 134 commits
62.0.3202.66: 11282 commits
62.0.3202.73: 27 commits
beta
62.0.3202.19: 10733 commits
62.0.3202.29: 176 commits
62.0.3202.38: 124 commits
62.0.3202.45: 110 commits
62.0.3202.52: 80 commits
63.0.3239.20: 9713 commits
dev
63.0.3223.7: 1246 commits
63.0.3233.3: 2639 commits
63.0.3236.6: 693 commits
63.0.3239.10: 1333 commits
63.0.3239.17: 130 commits
canary
64.0.3245.0: 326 commits
64.0.3247.0: 421 commits
64.0.3248.0: 297 commits
64.0.3249.0: 330 commits
64.0.3250.0: 362 commits

On the Chromium Dash staging console, I then ran Python scripts to count the number of commits which had android_deployment objects set to the version in question (how accurate our algorithms are, if accurate they should match above); the results of that are here:

stable
61.0.3163.81: 12716 commits
61.0.3163.98: 134 commits
62.0.3202.66: 900 commits
62.0.3202.73: 27 commits
beta
62.0.3202.19: 2250 commits
62.0.3202.29: 176 commits
62.0.3202.38: 124 commits
62.0.3202.45: 110 commits
62.0.3202.52: 80 commits
63.0.3239.20: 9713 commits
dev
63.0.3223.7: 1246 commits
63.0.3233.3: 1800 commits
63.0.3236.6: 693 commits
63.0.3239.10: 1333 commits
63.0.3239.17: 130 commits
canary
64.0.3245.0: 326 commits
64.0.3247.0: 421 commits
64.0.3248.0: 297 commits
64.0.3249.0: 330 commits
64.0.3250.0: 362 commits

In canary we're at 100% accuracy.

For dev, we have our first miss: for 63.0.3233.3 we should have 2639 commits, but we instead only have tagged 1800, which is a 30% error rate.

Beta looks good aside from a single version, but the version that's bad is *bad*: for 62.0.3202.19 we should have 10733 commits, but instead we have 2250, which is a failure rate of ~80%.

On stable, we are good on smaller point releases, but we have some bad misses there as well: 61.0.3163.81 has only 12716 of 14734 commits tagged (14% error rate) and 62.0.3202.66 has only 900 of 11282 commits tagged (93% failure rate).

The scripts I ran to get this data are here: https://docs.google.com/a/google.com/document/d/1WLL3UO3_ipBz_RF5P-ukejIUvS3iM3q7Phg1L4dBprs/edit?usp=sharing

This is clearly unacceptable so I'm marking this as beta blocking.  prasadv@, can you take a first look?  Holler if you're curious about the scripts I used or have any other questions...
Project Member

Comment 4 by bugdroid1@chromium.org, Nov 2 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/infra_internal/+/f8ee63c2dd46bba9fc7fba2b55fc79f77b070c10

commit f8ee63c2dd46bba9fc7fba2b55fc79f77b070c10
Author: prasadv <prasadv@google.com>
Date: Thu Nov 02 21:23:47 2017

I'm going to mark this as fixed given the CL in c#4 is landed and deployed.  We made some tweaks on the staging instance, and re-ran all jobs for *Android only* and now are seeing 100% accuracy there.  Data for all new versions (e.g. things we ship after today) should be good as well.

dgrogan@, appreciate your reporting this - if you see issues on the staging instance for future versions, please let us know.  We'll also be re-running the analysis above periodically in the future as well.

Thanks for helping improve Chromium Dash!
Status: Fixed (was: Assigned)

Sign in to add a comment