New issue
Advanced search Search tips

Issue 721585 link

Starred by 2 users

Issue metadata

Status: Started
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug

Blocked on:
issue 723856



Sign in to add a comment

Better integration of gclient-new-workdir.py on file systems supporting copy-on-write

Project Member Reported by wychen@chromium.org, May 11 2017

Issue description

On file systems supporting copy-on-write, gclient-new-workdir.py should take advantage of it.

Possible benefits:
- Lower disk usage per workdir
- Faster copying time
- Skip file downloading by gclient
- Reuse artifacts for incremental build

The default file system on Goobuntu is ext4, so you'll have to opt-in and run the btrfs conversion first in order to enjoy these benefits though.

Related discussions:
https://groups.google.com/a/chromium.org/forum/#!topic/chromium-dev/YCG9n8H0XIc/discussion
 
Project Member

Comment 1 by bugdroid1@chromium.org, May 16 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/704be878162e5e0f776d77b182f4969103aa886b

commit 704be878162e5e0f776d77b182f4969103aa886b
Author: Wei-Yin Chen (陳威尹) <wychen@chromium.org>
Date: Tue May 16 21:25:25 2017

Add reflink support for gclient-new-workdir.py

On filesystems supporting copy-on-write, like Btrfs and ZFS, use
"cp --reflink" to achieve faster speed and lower disk space usage.

Option --reflink and --no-reflink can override the automatic detection.

When reflink is supported, toolchains, support libraries, and artifacts
would also be preserved. This means you can do incremental builds right
after creating a new workdir.

For Linux Chromium checkout, one additional workdir only takes 0.8~1 GB
disk space in Btrfs, including data and metadata, depending on how many
artifacts are there inside the out*/ directories.

Change-Id: I8b913efba9e5afaeea2adc1c10a561f63cb50282
Bug: 721585
Reviewed-on: https://chromium-review.googlesource.com/499567
Commit-Queue: Wei-Yin Chen (陳威尹) <wychen@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/704be878162e5e0f776d77b182f4969103aa886b/gclient-new-workdir.py

Comment 2 by aga...@chromium.org, May 17 2017

Components: -Infra Infra>SDK
Owner: wychen@chromium.org
Status: Started (was: Untriaged)

Comment 3 by wychen@chromium.org, May 17 2017

Blockedon: 723856
Project Member

Comment 4 by bugdroid1@chromium.org, May 23 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/56e4ad939b3c06e2915faea0be14d34473bc3602

commit 56e4ad939b3c06e2915faea0be14d34473bc3602
Author: Wei-Yin Chen (陳威尹) <wychen@chromium.org>
Date: Tue May 23 20:30:23 2017

Add messages in gclient-new-workdir.py

When the repo is copied in copy-on-write mode, print informative
messages afterwards to help users.

Also, skip "git clean" step in copy-on-write mode.

Bug: 721585, 723856
Change-Id: I3235398c960d59a4cf44cfe7dffc79ed008a5190
Reviewed-on: https://chromium-review.googlesource.com/512262
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Wei-Yin Chen (陳威尹) <wychen@chromium.org>

[modify] https://crrev.com/56e4ad939b3c06e2915faea0be14d34473bc3602/gclient-new-workdir.py

Project Member

Comment 5 by bugdroid1@chromium.org, May 24 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/56664461ae794acee098f85d8c33b438909952f2

commit 56664461ae794acee098f85d8c33b438909952f2
Author: Wei-Yin Chen (陳威尹) <wychen@chromium.org>
Date: Wed May 24 04:10:53 2017

Support btrfs snapshot in gclient-new-workdir.py

If the repo is a btrfs subvolume, take a snapshot instead of using
"cp --reflink". Taking a snapshot is much faster, and uses less
disk space for metadata.

Bug: 721585
Change-Id: I97753dc30d46ff56d84d65f8d815c19d09cf104b
Reviewed-on: https://chromium-review.googlesource.com/513586
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Wei-Yin Chen (陳威尹) <wychen@chromium.org>

[modify] https://crrev.com/56664461ae794acee098f85d8c33b438909952f2/gclient-new-workdir.py

704be878162e5e0f776d77b182f4969103aa886b completely broke `gclient-new-workdir.py` on Mac and Linux for me. I've filed a bug at  Issue 728903 .
Project Member

Comment 7 by bugdroid1@chromium.org, Jun 2 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/2fa1203a22622a151df86f34704068b93ba84ee3

commit 2fa1203a22622a151df86f34704068b93ba84ee3
Author: Wei-Yin Chen (陳威尹) <wychen@chromium.org>
Date: Fri Jun 02 20:14:19 2017

Fix regression of gclient-new-workdir.py

gclient-new-workdir.py should work on systems not supporting reflink now.

Bug:  728903 , 721585
Change-Id: I1385c4281bbf61d4ccae64c3595a39972fbe9d9e
Reviewed-on: https://chromium-review.googlesource.com/522232
Commit-Queue: Wei-Yin Chen (陳威尹) <wychen@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/2fa1203a22622a151df86f34704068b93ba84ee3/gclient-new-workdir.py

Project Member

Comment 8 by bugdroid1@chromium.org, Jan 23 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/fd4ad2416554c43f6ecd2c9c62ab5914f521c9ad

commit fd4ad2416554c43f6ecd2c9c62ab5914f521c9ad
Author: Henrique Ferreiro <hferreiro@igalia.com>
Date: Tue Jan 23 23:05:03 2018

Canonicalize paths before linking in gclient-new-workdir.py

This fixes the case where a new workdir is created from a repository
which was previously created with gclient-new-workdir.py and that
repository is later deleted, rendering the symbolic links created by
gclient-new-workdir.py invalid.

It also fixes support_cow() returning 'True' in that case because 'cp
--reflink' does not fail when copying a symlink.


Bug: 721585
Change-Id: I84f4d80b8698af0da346be559d46a328ec0d0e9b
Reviewed-on: https://chromium-review.googlesource.com/842402
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@chromium.org>

[modify] https://crrev.com/fd4ad2416554c43f6ecd2c9c62ab5914f521c9ad/gclient-new-workdir.py
[modify] https://crrev.com/fd4ad2416554c43f6ecd2c9c62ab5914f521c9ad/git_common.py

Project Member

Comment 9 by bugdroid1@chromium.org, Feb 21 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/aea45d21ee81dc1d8cc0d7c71b4a31797e8b5fbb

commit aea45d21ee81dc1d8cc0d7c71b4a31797e8b5fbb
Author: Henrique Ferreiro <hferreiro@igalia.com>
Date: Wed Feb 21 23:38:58 2018

os.realpath => os.path.realpath in gclient-new-workdir.py

Bug: 721585
Change-Id: Ic6d47201e93cd89a1b58871ef0e3a1b5b708eb47
Reviewed-on: https://chromium-review.googlesource.com/924151
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Aaron Gable <agable@chromium.org>

[modify] https://crrev.com/aea45d21ee81dc1d8cc0d7c71b4a31797e8b5fbb/gclient-new-workdir.py
[modify] https://crrev.com/aea45d21ee81dc1d8cc0d7c71b4a31797e8b5fbb/git_common.py

Sign in to add a comment