test_push should report metrics to monarch |
||||||||
Issue descriptionmetrics should enable us to determine: - how often test_push is running - how often it is succeeding - how often it is failing in the various stages (deploy vs. test) - potentially: how often the failure is due to specific reasons This is mostly just a matter of hooking test_push up to monarch, and making sure that ts_mon credentials are being deployed to test_push. The only complication I see is that we want to make sure test_push is able to report to ts_mon, but we may not want the autotest instance itself to report to monarch (e.g. don't want metrics about chromeos-autotest.hot's schedulers or shards in our production metrics), so may want to place ts_mon credentials in a non standard location when deployed to chromeos-autotest.hot).
,
Mar 31 2017
Re: not reporting scheduler / host_scheduler etc metrics from push servers: That ship has sailed: http://shortn/_YXw4u0GL48 FYI: I recently moved where test_push runs from on the push_master. Instead of running as root from /root/chromeos-admin, it now runs from ~/chromiumos/chromeos-admin. Also, there's the full chromeos-admin repo checkout available. This means you now have access to chromite and ts_mon from test_push. So adding metrics should be easy at this point.
,
Apr 4 2017
,
Apr 6 2017
I suggest this is a high priority. Without long term metrics, we don't know how well we're doing in mopping up the remaining sources of push-to-prod flake.
,
Apr 6 2017
,
Jun 1 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/5696954d3310349da064afae350955c476716ba3 commit 5696954d3310349da064afae350955c476716ba3 Author: Shuqian Zhao <shuqianz@chromium.org> Date: Thu Jun 01 04:14:33 2017 autotest: add metric to track whether test_push passes BUG= chromium:707258 TEST=unittest Change-Id: Ied4fa1d2262b75b8cd66ff8dcba8efd6ebe1118b Reviewed-on: https://chromium-review.googlesource.com/518307 Commit-Ready: Paul Hobbs <phobbs@google.com> Tested-by: Shuqian Zhao <shuqianz@chromium.org> Tested-by: Paul Hobbs <phobbs@google.com> Reviewed-by: Paul Hobbs <phobbs@google.com> [modify] https://crrev.com/5696954d3310349da064afae350955c476716ba3/site_utils/test_push.py
,
Jun 1 2017
Lower the priority for the followup job
,
Jun 1 2017
Which followup job do you mean? I don't see the metric yet, either in panopticon or via monarch_tool. $ monarch_tool listmetrics /chrome/infra/chromeos/autotest/test_push
,
Jun 1 2017
I mean - how often it is failing in the various stages (deploy vs. test) - potentially: how often the failure is due to specific reasons Push to prod is needed for the new metric I think.
,
Jun 1 2017
The metric comes from the test_push server, so push to prod is not needed.
,
Jun 1 2017
Hmm, there is no error from the testing push log about the metric. Is there a way to debug why the metric is not there?
,
Jun 1 2017
I'd start by looking for logging about ts_mon. Perhaps it is missing credentials, in which case it should complain about that at ts_mon setup time.
,
Jun 1 2017
from both the scheduler and host_scheduler logs, I find one line about the ts_mon: 06/01 09:18:33.748 NOTIC| cros_logging:0037| ts_mon was set up.
,
Jun 1 2017
What matters is the logs from test_push, since that's where the new metric comes from.
,
Jun 1 2017
In the test_push log, there is only one line log related to ts_mon: [chromeos-autotest.hot.corp.google.com] out: NOTICE:root:ts_mon was set up.
,
Jun 1 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/034d85e1dfac9480d1d6b3787ffc908f004a92db commit 034d85e1dfac9480d1d6b3787ffc908f004a92db Author: Shuqian Zhao <shuqianz@chromium.org> Date: Thu Jun 01 22:27:10 2017 [autotest] fix test_push metric report BUG= chromium:707258 TEST=None Change-Id: I4349cab4824795610f993bd66bca6dc29fd7ee0f Reviewed-on: https://chromium-review.googlesource.com/521723 Tested-by: Shuqian Zhao <shuqianz@chromium.org> Reviewed-by: Aviv Keshet <akeshet@chromium.org> [modify] https://crrev.com/034d85e1dfac9480d1d6b3787ffc908f004a92db/site_utils/test_push.py
,
Sep 18 2017
,
Jan 22 2018
,
Jan 23 2018
|
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by shuqianz@chromium.org
, Mar 31 2017