New issue
Advanced search Search tips

Issue 735651 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Dec 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux
Pri: 1
Type: ----



Sign in to add a comment

compile failure on chromium.perf/Linux Builder

Project Member Reported by martiniss@chromium.org, Jun 21 2017

Issue description

compile failure on chromium.perf/Linux Builder

Builders failed on: 
- Linux Builder: 
  https://build.chromium.org/p/chromium.perf/builders/Linux%20Builder


Everything seems to fail compilation. I'm not sure what's happening on the bot, it's very confusing.

package build is also hanging for 2 hours, and then timing out. 
 
Project Member

Comment 1 by bugdroid1@chromium.org, Jun 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/ca9c93fa7b512d6c1ecdf6b27f029b7137ed9107

commit ca9c93fa7b512d6c1ecdf6b27f029b7137ed9107
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Wed Jun 21 21:27:54 2017

Revert "Zip gomacc logs before upload"

This reverts commit 3b86d055d9ad254c1ccadd1820d49216cc4d5895.

Reason for revert: Looks to be causing failures on perf
https://build.chromium.org/p/chromium.perf/builders/Win%20Builder

Original change's description:
> Zip gomacc logs before upload
> 
> When gomacc logs are generated, the number of the files is large,
> often 100. Uploading one by one takes 10~20 minutes, it's too slow.
> 
> Zip them and upload the zipped file.
> 
> BUG= 734896 
> 
> Change-Id: Ib409e909633a8b600472cb1e223213677678fb10
> Reviewed-on: https://chromium-review.googlesource.com/541181
> Commit-Queue: Shinya Kawanaka <shinyak@chromium.org>
> Reviewed-by: Dirk Pranke <dpranke@chromium.org>
> Reviewed-by: Yoshisato Yanagisawa <yyanagisawa@chromium.org>
> Reviewed-by: Takuto Ikuta <tikuta@google.com>

TBR=ukai@chromium.org,dpranke@chromium.org,yyanagisawa@chromium.org,shinyak@chromium.org,tikuta@google.com,tikuta@chromium.org

Change-Id: I4c094b76e2bede6a18e0002153c8c4da360c10fe
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  734896 ,  735651 
Reviewed-on: https://chromium-review.googlesource.com/544096
Reviewed-by: Benjamin Pastene <bpastene@chromium.org>
Commit-Queue: Benjamin Pastene <bpastene@chromium.org>

[modify] https://crrev.com/ca9c93fa7b512d6c1ecdf6b27f029b7137ed9107/scripts/slave/goma_utils.py

Comment 2 by dtu@chromium.org, Jun 22 2017

The compile failures only seem to occur on some bots. Maybe clobbering the lot would resolve it?
There's a 180 GB folder called thinlto-cache on the bots. I think it's trying to zip that up and hanging.

I'll remove the files from the machines.
Owner: friedman@chromium.org
Status: Assigned (was: Available)
Actually, friedman will delete it. We need to delete /b/c/b/Linux_Builder/src/out/Release/thinlto-cache on these machines:

slave85-c1
slave86-c1
slave87-c1
slave88-c1
slave89-c1
slave90-c1
slave91-c1
slave92-c1
slave93-c1
slave94-c1
slave95-c1
slave96-c1
slave97-c1
slave98-c1
slave99-c1
slave100-c1

Owner: martiniss@chromium.org
Actually, I'll just respawn the ccompute machines. It'll be a full clobber, but these bots are hosed anyways.
Cc: p...@chromium.org
Ok, found the suspect CL. https://codereview.chromium.org/2939373002

cc-ing author. 
I tried respawning the bots, by running a script, but that didn't fix it. They now have about 40 GBs of files on them, instead of 180.

Comment 8 by p...@chromium.org, Jun 22 2017

As I mentioned to martiniss@ over chat we probably need to exclude those files from the zip. Reverting until we can figure out how to do that.
Project Member

Comment 10 by bugdroid1@chromium.org, Jun 22 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/d2a7a5b9f3b572edea5c650930ff17edd31d30ee

commit d2a7a5b9f3b572edea5c650930ff17edd31d30ee
Author: Peter Collingbourne <pcc@google.com>
Date: Thu Jun 22 22:56:07 2017

Exclude thinlto-cache directory from package archives.

This directory contains a cache of object files created by the linker
during ThinLTO. It does not need to be included in the archive because
it is used only by the linker and can reach many GBs in size.

Bug:  735651 
Change-Id: Id8ec87957ece509010871f7c7829bff48be80a47
R=dpranke@chromium.org,martiniss@chromium.org
Reviewed-on: https://chromium-review.googlesource.com/544530
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Commit-Queue: Peter Collingbourne <pcc@chromium.org>

[modify] https://crrev.com/d2a7a5b9f3b572edea5c650930ff17edd31d30ee/scripts/common/chromium_utils.py

Project Member

Comment 11 by bugdroid1@chromium.org, Jun 23 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2be84dd0d41f8247d22bdc8690b5520d6c483a8d

commit 2be84dd0d41f8247d22bdc8690b5520d6c483a8d
Author: pcc <pcc@chromium.org>
Date: Fri Jun 23 00:46:58 2017

Revert of Enable ThinLTO for POSIX LTO by default on Linux. (patchset #2 id:20001 of https://codereview.chromium.org/2939373002/ )

Reason for revert:
Breaks chromium.perf builder.

BUG= 735651 

Original issue's description:
> Enable ThinLTO for POSIX LTO by default on Linux.
>
> This is a reland of the remaining part of
> https://codereview.chromium.org/2831213006 .
>
> The binary size regression has been addressed by disabling ThinLTO
> whole-program optimizations and improving the quality of unoptimized
> code.
>
> BUG= 660216 
> R=thakis@chromium.org,hans@chromium.org
>
> Review-Url: https://codereview.chromium.org/2939373002
> Cr-Commit-Position: refs/heads/master@{#480144}
> Committed: https://chromium.googlesource.com/chromium/src/+/5cb9983f12f300422c16ba812c2d5cdeb278cb74

TBR=hans@chromium.org,thakis@chromium.org
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG= 660216 

Review-Url: https://codereview.chromium.org/2953943002
Cr-Commit-Position: refs/heads/master@{#481748}

[modify] https://crrev.com/2be84dd0d41f8247d22bdc8690b5520d6c483a8d/build/toolchain/toolchain.gni

Project Member

Comment 12 by bugdroid1@chromium.org, Jun 23 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9a6cc80b32bb5522fe262e1f60a9995dabf9dcbb

commit 9a6cc80b32bb5522fe262e1f60a9995dabf9dcbb
Author: pcc <pcc@chromium.org>
Date: Fri Jun 23 17:35:17 2017

Enable ThinLTO for POSIX LTO by default on Linux, take 2.

Previous attempt: https://codereview.chromium.org/2939373002

It was reverted because it caused large thinlto-cache directories
to be created on the perf bots, which were also being unnecessarily
included in package archives. This has been addressed in two ways:

1) the bots have been taught to exclude thinlto-cache from
   package archives (see https://chromium-review.googlesource.com/c/544530/).
2) this change limits the size of the cache to 10% of
   available disk space.

BUG= 660216 , 735651 
R=dpranke@chromium.org,thakis@chromium.org
TBR=thomasanderson@chromium.org

Review-Url: https://codereview.chromium.org/2957533002
Cr-Commit-Position: refs/heads/master@{#481944}

[modify] https://crrev.com/9a6cc80b32bb5522fe262e1f60a9995dabf9dcbb/build/config/compiler/BUILD.gn
[modify] https://crrev.com/9a6cc80b32bb5522fe262e1f60a9995dabf9dcbb/build/toolchain/toolchain.gni
[modify] https://crrev.com/9a6cc80b32bb5522fe262e1f60a9995dabf9dcbb/chrome/installer/linux/debian/expected_deps_x64_jessie

Status: Fixed (was: Assigned)

Sign in to add a comment