New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 662131 link

Starred by 1 user
Status: Duplicate
Owner:
█ pho...@chromium.org
Last visit > 30 days ago
Closed: Nov 2016
Cc:
akes...@chromium.org
ayatane@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

sysmon is flushing too frequently.

Project Member Reported by pho...@chromium.org, Nov 3 2016 
From the acquisition-mon-api-shared logs (https://analog.corp.google.com/#task_log?focus_entry_filename=/remote/ivaw4:4903/task/logs.7.acquisition-mon-api-shared-prod.api-server.acquisition-mon-api-shared.32025019519/storage_svc.ivaw4.acquisition-mon-api-shared.log.java.20161103-111521.20051&focus_entry_offset=6804969&focus_entry_timestamp=1478197781461999&focus_target_specific_entry_index=988&focus_target_type=default&ignore_case=true&load_offset=9223372036854775806&load_pivot_alignment=earlier&load_timestamp=9223372036854775807&q=%252Fchrome%252Finfra%252Fchromeos&resolver_file_glob=storage_svc.*.log.java.*&target_name=acquisition-mon-api-shared-prod.api-server&target_type=JOB&user=acquisition-mon-api-shared)

Error for scope: acquisition-mon-api-chrome-infrastructure: Message received from remote endpoint: iecrm1.prod.google.com Proto ID: 1878375151, 0 Monarch Error Code: 9 Monarch only accepted 0 of 70 points. Example error: Point was sampled too soon compared to sampling period; ignoring. Timestamp: '2016/11/03-11:29:41.419 (1478197781419000)' Youngest existing timestamp: '2016/11/03-11:29:22.599 (1478197762599000)' Densest retention period: MIN_01 (monarch.acquisitions.Task{proxy_environment = 'shared-production' acquisition_name = 'acquisition-mon-api-chrome-infrastructure' service_name = 'sysmon' job_name = 'sysmon' data_center = 'hot' host_name = 'chromeos-server3' task_num = 0 proxy_zone = 'atl'} /chrome/infra/chromeos/sysmon/dev/cpu/count{})

Comment 1 by ayatane@chromium.org, Nov 3 2016 

It looks like autoserv and scheduler is affected too.

Error for scope: acquisition-mon-api-chrome-infrastructure: Message received from remote endpoint: iebbu14.prod.google.com Proto ID: 1882686060, 0 Monarch Error Code: 9 Monarch only accepted 0 of 2 points. Example error: Point was sampled too soon compared to sampling period; ignoring. Timestamp: '2016/11/03-12:04:27.786 (1478199867786000)' Youngest existing timestamp: '2016/11/03-12:04:27.000 (1478199867000000)' Densest retention period: MIN_01 (monarch.acquisitions.Task{proxy_environment = 'shared-production' acquisition_name = 'acquisition-mon-api-chrome-infrastructure' service_name = 'autoserv' job_name = 'autoserv' data_center = 'hot' host_name = 'autogen:chromeos-server10' task_num = 8970 proxy_zone = 'atl'} /chrome/infra/chromeos/autotest/provision/verify_url{'release', 'peach_pit', '100.115.219.130', '55'})

Error for scope: acquisition-mon-api-chrome-infrastructure: Message received from remote endpoint: iebyd7.prod.google.com Proto ID: 363876592, 0 Monarch Error Code: 9 Monarch only accepted 0 of 9 points. Example error: Point was sampled too soon compared to sampling period; ignoring. Timestamp: '2016/11/03-12:04:27.720 (1478199867720000)' Youngest existing timestamp: '2016/11/03-12:04:25.112 (1478199865112000)' Densest retention period: MIN_01 (monarch.acquisitions.Task{proxy_environment = 'shared-production' acquisition_name = 'acquisition-mon-api-chrome-infrastructure' service_name = 'autotest_host_scheduler' job_name = 'autotest_host_scheduler' data_center = 'hot' host_name = 'chromeos-server50' task_num = 0 proxy_zone = 'atl'} /chrome/infra/chromeos/autotest/host_scheduler/tick{})
Project Member

Comment 2 by bugdroid1@chromium.org, Nov 29 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/7ee1aaa2b70372eccb820f27ef1c974a06932264

commit 7ee1aaa2b70372eccb820f27ef1c974a06932264
Author: Paul Hobbs <phobbs@google.com>
Date: Thu Nov 03 19:02:31 2016

sysmon: Set auto_flush=False

The sysmon script calls metrics.Flush() manually, so it shouldn't also
auto-flush metrics.

BUG= chromium:662131 
TEST=None

Change-Id: I8e397b4f2d4c7b47e6fe25c84f4f8af197ca0052
Reviewed-on: https://chromium-review.googlesource.com/407336
Commit-Ready: Allen Li <ayatane@chromium.org>
Tested-by: Paul Hobbs <phobbs@google.com>
Reviewed-by: Allen Li <ayatane@chromium.org>

[modify] https://crrev.com/7ee1aaa2b70372eccb820f27ef1c974a06932264/scripts/sysmon/__init__.py

Comment 3 by ayatane@chromium.org, Nov 29 2016

Mergedinto: 666013
Status: Duplicate (was: Started)
►

Sign in to add a comment