I see CLs being marked as submitted in cidb.

mysql> select * from clActionTable WHERE action='submitted' ORDER BY ID DESC LIMIT 20;
+----------+----------+---------------+--------------+---------------+-----------+------------------------------------------------+---------------------+----------------+
| id       | build_id | change_number | patch_number | change_source | action    | reason                                         | timestamp           | buildbucket_id |
+----------+----------+---------------+--------------+---------------+-----------+------------------------------------------------+---------------------+----------------+
| 17562061 |  1936599 |        674365 |            6 | external      | submitted | strategy:non-manifest-submit                   | 2017-10-12 07:30:52 | NULL           |
| 17561889 |  1936219 |        477178 |            3 | internal      | submitted | strategy:cq-submit-partial-pool-cq-history     | 2017-10-12 07:12:01 | NULL           |
| 17561886 |  1936219 |        687234 |            1 | external      | submitted | strategy:cq-submit-partial-pool-builds-passed  | 2017-10-12 07:11:55 | NULL           |
| 17561885 |  1936219 |        462092 |            1 | internal      | submitted | strategy:cq-submit-partial-pool-builds-passed  | 2017-10-12 07:11:51 | NULL           |
| 17561884 |  1936219 |        686080 |            2 | external      | submitted | strategy:cq-submit-partial-pool-cq-history     | 2017-10-12 07:11:48 | NULL           |
| 17561883 |  1936219 |        686081 |            2 | external      | submitted | strategy:cq-submit-partial-pool-cq-history     | 2017-10-12 07:11:47 | NULL           |
| 17561881 |  1936219 |        470333 |            2 | internal      | submitted | strategy:cq-submit-partial-pool-builds-passed  | 2017-10-12 07:11:24 | NULL           |
| 17561880 |  1936219 |        711195 |            6 | external      | submitted | strategy:cq-submit-partial-pool-ignored-stages | 2017-10-12 07:11:21 | NULL           |
| 17561878 |  1936219 |        711617 |            1 | external      | submitted | strategy:cq-submit-partial-pool-ignored-stages | 2017-10-12 07:11:11 | NULL           |
| 17561877 |  1936219 |        704297 |            7 | external      | submitted | strategy:cq-submit-partial-pool-ignored-stages | 2017-10-12 07:11:11 | NULL           |
| 17561876 |  1936219 |        711737 |            1 | external      | submitted | strategy:cq-submit-partial-pool-ignored-stages | 2017-10-12 07:11:10 | NULL           |
| 17561875 |  1936219 |        711538 |            1 | external      | submitted | strategy:cq-submit-partial-pool-ignored-stages | 2017-10-12 07:11:10 | NULL           |
| 17561874 |  1936219 |        704275 |            5 | external      | submitted | strategy:cq-submit-partial-pool-ignored-stages | 2017-10-12 07:11:10 | NULL           |
| 17561873 |  1936219 |        477413 |            1 | internal      | submitted | strategy:cq-submit-partial-pool-builds-passed  | 2017-10-12 07:10:59 | NULL           |
| 17561872 |  1936219 |        470335 |            3 | internal      | submitted | strategy:cq-submit-partial-pool-builds-passed  | 2017-10-12 07:10:55 | NULL           |
| 17561870 |  1936219 |        710859 |            2 | external      | submitted | strategy:cq-submit-partial-pool-builds-passed  | 2017-10-12 07:10:52 | NULL           |
| 17561867 |  1936219 |        714396 |            2 | external      | submitted | strategy:cq-submit-partial-pool-cq-history     | 2017-10-12 07:10:35 | NULL           |
| 17561866 |  1936219 |        714024 |            1 | external      | submitted | strategy:cq-submit-partial-pool-cq-history     | 2017-10-12 07:10:29 | NULL           |
| 17561865 |  1936219 |        699521 |            5 | external      | submitted | strategy:cq-submit-partial-pool-builds-passed  | 2017-10-12 07:10:12 | NULL           |
| 17561864 |  1936219 |        477832 |            1 | internal      | submitted | strategy:cq-submit-partial-pool-builds-passed  | 2017-10-12 07:10:03 | NULL           |
+----------+----------+---------------+--------------+---------------+-----------+------------------------------------------------+---------------------+----------------+
20 rows in set (0.24 sec)

Possibly related, this query http://shortn/_haPvfXg7wr reports that the metric is not retained by any chrome-infra retention policy, whereas it certainly used to be a few days ago.

This looks to me like either a monarch bug, or some kind of change in retention. The same effective query shows different results based on whether streams are filtered during fetch or after fetch:

http://shortn/_2hOcVnlGIY

+some monarch advisors

+estaab who owns monarch things these days?

katthomas might be able to help, but this might be too chromeos-specific.

Is chrome-infra-retentions running out of quota?

Repository=Force gets rid of this inconsistency, on pcon. However, query still misbehaving in viceroy.

The inconsistency happens when the value (e.g. "submitted") isn't one of the default options in the "Filter" textbox. "kicked_out" is one of the popped default options and the query is correct for "metric:action == kicked_out". 

Is this fixed? It no longer shows a constant value...

The symptom fixed itself (something related to monarch query cache I guess), but in pcon I still see the "Not retained by any retention policy of role chrome-infra" message which is alarming and might have contributed to the problem.

I don't know who else would take this. If anyone else *wants* to, feel free. :)

It is retained, but under a slightly different set of target fields: http://shortn/_cfv95SPL3I

Our retention policy is different for host_name ~= autogen:.* metrics vs. the non-autogen ones. Pcon may get confused trying to figure out the policy due to target fields being in the mix.

So, I'd say it's WAI.