New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 713856 link

Starred by 1 user

Issue metadata

Status: Archived
Owner:
Last visit > 30 days ago
Closed: Apr 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 0
Type: Bug

Blocked on:
issue 713968

Blocking:
issue 713004



Sign in to add a comment

network overloaded by offloading big size files to GS

Project Member Reported by nxia@chromium.org, Apr 20 2017

Issue description

The kilobyte transfer rate of gsoffloader has spiked a lot since 6am on April 12. It means the number of large files to offload has increased since then.

We looked into some big folder examples, they all include big crash dump files. we suspect the bad chrome version generates big crash files and causes network overload. And the network connectivity issue have been affecting the CQ/Canary/release builds a lot (see crbug.com/713004)

jobs: http://shortn/_30BwQteEGu
kilobytes: http://shortn/_WwvfnYqEGk
bytes offloaded per job http://shortn/_iApXsTMdYc


 

Comment 1 by nxia@chromium.org, Apr 20 2017

I looked into the first test failure reported in crbug.com/713004

the size of the test log is 1.1G

http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=113290478
There are multiple issues surrounding the problem that excessive crashes in the lab due to a product bug can bring the lab network (+ other stuff) down. issue 489845 has collected some past outages due to this, and has some ideas about what needs to be done.

Comment 3 by nxia@chromium.org, Apr 20 2017

We're going to exclude 'chrome*core' files in collecting_logs for now. 

Comment 4 by nxia@chromium.org, Apr 21 2017

Blocking: 713004
Cc: semenzato@chromium.org
Cc: hidehiko@chromium.org

Comment 7 by ihf@chromium.org, Apr 21 2017

Blockedon: 713968
The chrome crash is a double free and tracked in issue 713968.

Comment 8 by nxia@chromium.org, Apr 24 2017

Cc: -jrbarnette@chromium.org chingcodes@chromium.org
bytes offloaded per job http://shortn/_iApXsTMdYc has dropped to the normal number, downgrade this to p1 and cc the current deputy.
(note: it may just have dropped because we aren't running anything, so let's not declare victory yet :) )

Comment 10 by aut...@google.com, Apr 25 2017

Owner: nxia@chromium.org
Cc: dshi@chromium.org
+dshi is gonna be working on a ddoc in this area
Status: Fixed (was: Untriaged)
At this point, the GS Offloader dashboard says we're back to
roughly the level prior to the outage.  As best we can tell,
this outage is over.

Labels: VerifyIn-61

Comment 14 by dchan@chromium.org, Jan 22 2018

Status: Archived (was: Fixed)

Sign in to add a comment