New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 715286 link

Starred by 1 user

Issue metadata

Status: Duplicate
Merged: issue 716218
Owner: ----
Closed: Apr 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

Limit the size of core dumps that could be transferred in rsync/scp ?

Project Member Reported by nxia@chromium.org, Apr 25 2017

Issue description

We had a CQ outage because offloading lots of big Chrome crash dump files overloaded the lab network. Is it possible to limit the size of core dumps that could be transferred in rsync/scp ?

more context about the outage at  crbug.com/713856 

 

Comment 1 by nxia@chromium.org, Apr 25 2017

Daniel Erat
11:17 AM Today

this seems much more important to me than adding alerts about the frequency of chrome crashes, and should probably be the top priority here, right? sometimes chrome is going to be very crashy. the CQ needs to avoid overloading the network infrastructure when artifacts are larger than expected.


Ningning Xia
12:11 PM Today
each core dump file is ~ 300Mb. What will be the good limit size?

	
Ningning Xia
12:11 PM Today
maybe we need a configurable flag to turn on/off the offloading of dump files when this kind of crisis happens again?

	
Daniel Erat
12:19 PM Today
what about if we just offload at most one core dump (while logging the existence of others)? i'm guessing/hoping that the normal state is for there to be zero chrome crashes, so if we have multiple dumps, it might be safe to assume that chrome is in a crash loop.

would only offloading one dump have prevented the meltdown, or would we still have been in trouble due to the number of bots trying to offload?
	
Ningning Xia
2:44 PM Today
I'm not sure how easy it is to identify whether crashes are same and only upload one if there're duplicated. maybe symbolized stacks as James Cook mentioned above will help

Comment 2 by aut...@google.com, Apr 25 2017

Labels: Hotlist-Fixit
Mergedinto: 716218
Status: Duplicate (was: Untriaged)

Sign in to add a comment