New issue
Advanced search Search tips

Issue 812979 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Feb 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 3
Type: Bug

Blocking:
issue 812982



Sign in to add a comment

cros-workon: rsync filter rules don't support .gitignore ! negation

Project Member Reported by nya@chromium.org, Feb 16 2018

Issue description

crrev.com/c/845474 updated cros-workon.eclass to set

 rsync --filter=":- .gitignore"

to honor .gitignore on copying files locally. However rsync's supported set of filter patterns is not the same as that of git, so it may cause unexpected results.

In fact, tfiga@ reported an issue about .gitignore using negative patterns, such as:

 gen*_pack.h
 !genX_pack.h

 

Comment 1 by tfiga@chromium.org, Feb 16 2018

This is generally related to the problem I noticed when trying to compile media-libs/arc-mesa-9999 (after cros_workon start). See  issue 812982 . The negative pattern was my attempt to fix the upstream bug, which didn't work.

Comment 2 by vapier@chromium.org, Feb 16 2018

Components: OS>Packages
Labels: Hotlist-GoodFirstBug
Status: Available (was: Untriaged)
Summary: cros-workon: rsync filter rules don't support .gitignore ! negation (was: cros-workon fails to honor .gitignore)
fortunately, a cs/ suggests we basically never use this feature.  however, i agree it would make sense to do so.

going through the rsync documentation, i don't think the existing rules/formats make this feasible.  i think the only way to pull this off would be to add support for the gitignore format directly to rsync and send that patch upstream.  considering the prevalence of git, and that rsync has support for .cvsignore already, i don't think it'd be a tough sell.

basically:
- add a new --git-exclude command line option
- extend the parser backend to support this format
- extend the merge-file filter logic to have a new "G" keyword (like there is a "C" keyword already for CVS)

then we'd be able to update cros-workon to use --filter=":G" and everything would be peachy.

Comment 3 by nya@chromium.org, Feb 19 2018

tfiga@ suggested another way to ask git to list ignored files and pass it to rsync. For example:

$ git ls-files --others --ignored --exclude-standard --directory

Comment 4 by vapier@chromium.org, Feb 19 2018

that might work if we did something like:
 ... --exclude-from=<(git ls-files --others --ignored --exclude-standard --directory) ...

Comment 5 by tfiga@chromium.org, Feb 19 2018

Or perhaps

git ls-files --others --ignored --exclude-standard --directory | rsync [...] --exclude-from=- [...]

as man seems to mention that you can give "-" to --exclude-from and the list will be read from stdin.

Comment 6 by vapier@chromium.org, Feb 19 2018

process substitution is preferable as you can run multiple commands to read from, and you won't have a pipeline to deal with.  both otherwise that seems like it'd work in this specific case.

Comment 7 by tfiga@chromium.org, Feb 19 2018

Fair enough. Thanks for explaining. Let me hack up a CL quickly.

To be honest I didn't even know such thing existed. (Yes, it's embarrassing. ;))

Comment 8 by tfiga@chromium.org, Feb 19 2018

Blocking: 812982

Comment 9 by tfiga@chromium.org, Feb 19 2018

Owner: tfiga@chromium.org
Status: Started (was: Available)
Project Member

Comment 10 by bugdroid1@chromium.org, Feb 19 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/3110e22b5b6ed23ef99b686372637301cbbf4129

commit 3110e22b5b6ed23ef99b686372637301cbbf4129
Author: Mike Frysinger <vapier@chromium.org>
Date: Mon Feb 19 10:36:03 2018

gitignore: delete

We don't keep any of these binary types in the tree, nor do we want
to.  So drop this gitignore rule that permits them.

BUG= chromium:812979 
TEST=precq passes

Change-Id: I180ecd6cd18baf46d1faa31d44d16fd557845715
Reviewed-on: https://chromium-review.googlesource.com/924348
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Shuhei Takahashi <nya@chromium.org>

[delete] https://crrev.com/3b8e1aae5207561b910f1cd44d0415361cd6c151/.gitignore

Project Member

Comment 12 by bugdroid1@chromium.org, Feb 21 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/b6f21b6b6e9164cfdd1d37fa9290592ee762a3d2

commit b6f21b6b6e9164cfdd1d37fa9290592ee762a3d2
Author: Tomasz Figa <tfiga@chromium.org>
Date: Wed Feb 21 12:27:06 2018

cros-workon: Use git ls-files for excluding files from copying

The --filter=":- <file>" switch of rsync, used currently, does not fully
handle .gitignore syntax and would behave incorrectly at least in
following two example cases:

a) negative matches

  gen*_pack.h
  !genX_pack.h

anything matching the first pattern, except anything matching the second
one too, should be ignored, but rsync reads the exclamation mark as a
part of file name pattern.

b) files checked in, but present in .gitignore - even though it's
questionable whether this is a correct repository setup, the affected
files are there in the checkout and so should be also included in the
source tree used for compilation.

Use git ls-files to query the list of ignored files instead and use the
output with --exclude-from rsync switch. This ensures that file
exclusion works exactly the same way as in git.

BUG= chromium:812979 
BUG= chromium:812982 
TEST=Create following directory stucture in src/third_party/arc-mesa
   x/y/ignored
   x/y/not_ignored
   y/ignored
   y/not_ignored
   y/.gitignore
 and put following contents in y/.gitignore
   ignored
 emerge-soraka arc-mesa copies all files except y/ignored.

Change-Id: I459a066c41f8effb2f7d655f324ab7af645bcd2e
Reviewed-on: https://chromium-review.googlesource.com/925222
Commit-Ready: Tomasz Figa <tfiga@chromium.org>
Tested-by: Tomasz Figa <tfiga@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/b6f21b6b6e9164cfdd1d37fa9290592ee762a3d2/eclass/cros-workon.eclass

Comment 13 by tfiga@chromium.org, Feb 21 2018

Status: Fixed (was: Started)

Sign in to add a comment