New issue
Advanced search Search tips

Issue 806700 link

Starred by 2 users

Issue metadata

Status: Verified
Owner:
Closed: Jan 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

Sheriff-o-Matic is not refreshed (stale)

Project Member Reported by vitaliii@chromium.org, Jan 29 2018

Issue description

Now 1:53 PST and Sheriff-o-Matic last update was at "1/28/2018, 11:21 pm PST (3 hours ago)". 

When I refresh the page or press the refresh button, nothing happens.

The developed console shows:

=====
som-app.vulcanized.html:38241 ...... too many results, data snipped....,BrowserCloseManagerWithDownloadsBrowserTest/BrowserCloseManagerWithDownloadsBrowserTest.TestWithDownloads/0,BrowserEncodingTest.TestEncodingAutoDetect,BrowserWindowControllerTest.FullscreenResizeFlags,BrowsingDataRemoverBrowserTest.Download,ChromeResourceDispatcherHostDelegateBrowserTest.ThrottlesAddedExactlyOnceToADownloads,ChromeResourceDispatcherHostDelegateBrowserTest.ThrottlesAddedExactlyOnceToLargeSniffedDownloads,ChromeResourceDispatcherHostDelegateBrowserTest.ThrottlesAddedExactlyOnceToTinySniffedDownloads,ConstrainedWindowMacTest.BrowserWindowFullscreen,DownloadExtensionTest.DownloadExtensionTest_Download_AuthBasic,DownloadExtensionTest.DownloadExtensionTest_Download_AuthBasic_Fail,DownloadExtensionTest.DownloadExtensionTest_Download_Basic,DownloadExtensionTest.DownloadExtensionTest_Download_ConflictAction,DownloadExtensionTest.DownloadExtensionTest_Download_DataURL,DownloadExtensionTest.DownloadExtensionTest_Download_File,DownloadExtensionTest.DownloadExtensionTest_Download_Headers,DownloadExtensionTest.DownloadExtensionTest_Download_Headers_Fail,DownloadExtensionTest.DownloadExtensionTest_Download_InterruptAndResume,DownloadExtensionTest.DownloadExtensionTest_Download_Post,DownloadExtensionTest.DownloadExtensionTest_Download_Post_Get,DownloadExtensionTest.DownloadExtensionTest_Download_Redirect,DownloadExtensionTest.DownloadExtensionTest_Download_Subdirectory,DownloadExtensionTest.DownloadExtensionTest_Download_URLFragment,DownloadExtensionTest.DownloadExtensionTest_FileIcon_Active,DownloadExtensionTest.DownloadExtensionTest_FileIcon_History,DownloadExtensionTest.DownloadExtensionTest_OnDeterminingFilename_AbsPathInvalid,DownloadExtensionTest.DownloadExtensionTest_OnDeterminingFilename_CurDirInvalid,DownloadExtensionTest.DownloadExtensionTest_OnDeterminingFilename_EmptyBasenameInvalid,DownloadExtensionTest.DownloadExtensionTest_OnDeterminingFilename_IllegalFilenameExtension,DownloadExtensionTest.DownloadExtensionTest_OnDeterminingFilename_IncognitoSpanning,DownloadExtensionTest.DownloadExtensionTest_OnDeterminingFilename_IncognitoSplit,DownloadExtensionTest.DownloadExtensionTest_OnDeterminingFilename_NoChange,DownloadExtensionTest.DownloadExtensionTest_OnDeterminingFilename_Override,DownloadExtensionTest.DownloadExtensionTest_OnDeterminingFilename_ParentDirInvalid,DownloadExtensionTest.DownloadExtensionTest_OnDeterminingFilename_ReferencesParentInvalid,DownloadExtensionTest.DownloadExtensionTest_OnDeterminingFilename_ReservedFilename,DownloadExtensionTest.DownloadExtensionTest_OnDeterminingFilename_Twice,DownloadExtensionTest.DownloadExtensionTest_Open,DownloadExtensionTest.DownloadExtensionTest_PauseResumeCancelErase,DownloadExtensionTest.DownloadExtensionTest_SearchDanger,DownloadExtensionTest.DownloadExtensionTest_SearchEmptyQuery is not equal to BrowserCommandControllerInteractiveTest.KeyEventsShouldBeConsumedByWebPageInJsFullscreenExceptForEsc,BrowserCommandControllerInteractiveTest.KeyEventsShouldBeConsumedByWebPageInJsFullscreenExceptForF11,BrowserCommandControllerInteractiveTest.ShortcutsShouldTakeEffectInWindowMode,DevToolsManagerDelegateTest.ExitFullscreenWindow,DevToolsManagerDelegateTest.MaximizedToFullscreenWindow,DevToolsManagerDelegateTest.NormalToFullscreenWindow,ExtensionApiTest.FocusWindowDoesNotExitFullscreen,NotificationsTest.TestShouldDisplayFullscreen,NotificationsTest.TestShouldDisplayPopupNotification,SitePerProcessInteractiveBrowserTest.FullscreenElementInABAAndExitViaEscapeKey,SitePerProcessInteractiveBrowserTest.FullscreenElementInABAAndExitViaJS,SitePerProcessInteractiveBrowserTest.FullscreenElementInSubframe but they were merged together. This should never happen, because merging is done server side by looking at the reason data.
_mergeReason @ som-app.vulcanized.html:38241
_computeAlert @ som-app.vulcanized.html:38218
_computeAlertsSet @ som-app.vulcanized.html:38155
_computeAlerts @ som-app.vulcanized.html:38136
runMethodEffect @ som-app.vulcanized.html:3014
runComputedEffect @ som-app.vulcanized.html:2643
runEffectsForProperty @ som-app.vulcanized.html:2378
runEffects @ som-app.vulcanized.html:2344
runComputedEffects @ som-app.vulcanized.html:2621
_propertiesChanged @ som-app.vulcanized.html:3853
_flushProperties @ som-app.vulcanized.html:1688
_invalidateProperties @ som-app.vulcanized.html:3707
set @ som-app.vulcanized.html:4015
_alertsSetData @ som-app.vulcanized.html:38059
window.fetch.then.then @ som-app.vulcanized.html:38095
Promise resolved (async)
alertStreams.forEach @ som-app.vulcanized.html:38095
_updateAlerts @ som-app.vulcanized.html:38079
refresh @ som-app.vulcanized.html:37934
_refresh @ som-app.vulcanized.html:50264
handler @ som-app.vulcanized.html:1848
_fire @ som-app.vulcanized.html:6490
forward @ som-app.vulcanized.html:6852
click @ som-app.vulcanized.html:6822
_handleNative @ som-app.vulcanized.html:6280
=====
 
Labels: -Pri-1 Pri-0
Increasing the priority, since this increases the difficulty of sheriffing.
Cc: martiniss@chromium.org
martiniss@ was touching that code long time ago. CC him, in case he is familiar with recent changes.
Owner: zhangtiff@chromium.org
Status: Assigned (was: Untriaged)
Assigning to zhangtiff@ as an OWNER.
Labels: -Pri-0 Pri-1
It did refresh 4 minutes ago. Decreasing priority.
Now it hasn't been refreshed since 2:30 am PST (5 hours).

Comment 7 by bsep@chromium.org, Jan 29 2018

Ping! The list hasn't updated since 12:58 am PST. The bug queue is okay though.

Comment 8 by bsep@chromium.org, Jan 29 2018

Oh, I was looking at the time the failure occurred. It still hasn't updated in like 6 hours though.

Comment 9 by bsep@chromium.org, Jan 29 2018

Cc: mar...@chromium.org seanmccullough@chromium.org
This is the chromium tree I take it? Looking...
Cc: hinoka@chromium.org
Analyzer logs say:

Status 500 msg Post https://luci-milo.appspot.com/prpc/milo.Buildbot/GetCompressedMasterJSON: Call error 11: Deadline exceeded (timeout)

hinoka@: anything on the milo end look odd?

Comment 13 by no...@chromium.org, Jan 29 2018

Cc: zhangtiff@chromium.org
Owner: no...@chromium.org
Status: Started (was: Assigned)

Comment 14 by no...@chromium.org, Jan 29 2018

Cc: -zhangtiff@chromium.org
Owner: zhangtiff@chromium.org
HTTP 500s in comment #12 are unrelated to SOM. Those requests are coming from luci-migration app.

Comment 15 by no...@chromium.org, Jan 29 2018

Cc: zhangtiff@chromium.org
Status: Assigned (was: Started)
Project Member

Comment 17 by bugdroid1@chromium.org, Jan 29 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/35b6729b1a56e0c2d4f88bcf7258da71d636f9a9

commit 35b6729b1a56e0c2d4f88bcf7258da71d636f9a9
Author: Nodir Turakulov <nodir@google.com>
Date: Mon Jan 29 21:21:00 2018

[milo] fix error message format

Forgot %d in format string.

Bug:  806700 
Change-Id: I0fff27210e0a8b89f0ad3364de0b88cd2d3b10f2
Reviewed-on: https://chromium-review.googlesource.com/891798
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/35b6729b1a56e0c2d4f88bcf7258da71d636f9a9/milo/buildsource/buildbot/buildstore/buildbucket.go

Project Member

Comment 18 by bugdroid1@chromium.org, Jan 29 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/d3692e44683e3133b0fbb81bbf72e8d476148154

commit d3692e44683e3133b0fbb81bbf72e8d476148154
Author: Ryan Tseng <hinoka@google.com>
Date: Mon Jan 29 22:33:50 2018

[milo] Set min active instances

Add in automatic scaling factors for Milo.

Milo's default service generally has about 9-12 instances active, so this just
codifies the minimum, and there shouldn't be a difference.

Bug:806700
Change-Id: Ifbd21dc2697b12f9784b0a7bc90ada851b20d777
Reviewed-on: https://chromium-review.googlesource.com/892019
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>

[modify] https://crrev.com/d3692e44683e3133b0fbb81bbf72e8d476148154/milo/frontend/appengine/app.yaml

Comment 19 by bsep@chromium.org, Jan 29 2018

I'm not sure if the fix is supposed to have taken effect yet, but the sheriff-o-matic hasn't updated thus far.
The sheriff-o-matic still hasn't updated so far. How's it going?

Comment 21 by no...@chromium.org, Jan 30 2018

Ryan, did you trying bisecting in which Milo version the problem started to occur? we can narrow down the list of CLs that caused this.
Project Member

Comment 22 by bugdroid1@chromium.org, Jan 30 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/be601b65d15544d56302f511dcb02ad971625199

commit be601b65d15544d56302f511dcb02ad971625199
Author: Ryan Tseng <hinoka@google.com>
Date: Tue Jan 30 19:19:22 2018

[milo] Reduce parallel requests from 8 to 4

Bug:  806700 
Change-Id: Ie003fc77894d2e4217a913dc6ae2e3b093064d3a
Reviewed-on: https://chromium-review.googlesource.com/891631
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>

[modify] https://crrev.com/be601b65d15544d56302f511dcb02ad971625199/milo/frontend/appengine/app.yaml

Comment 23 by no...@chromium.org, Jan 30 2018

Labels: -Pri-1 Pri-0
Owner: no...@chromium.org
Status: Started (was: Assigned)
Owner: seanmccullough@chromium.org
I'll try changing the size of the worker pool to increase concurrency and get the overall time down, and see if it still stays under RAM constraints.
Issue 807635 has been merged into this issue.
Project Member

Comment 27 by bugdroid1@chromium.org, Jan 31 2018

Summary: Sheriff-o-Matic is not refreshed (stale) (was: Sheriff-o-Matic is not refreshed)
Changing the title, so that new sheriffs have more chances to notice.
Just pushed this fix to prod. PTAL
Status: Verified (was: Started)
Looks good, thank you!

Sign in to add a comment