New issue
Advanced search Search tips

Issue 671683 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: May 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

[W3C auto-import] Test import fails in presubmit when imported file has Windows-style line endings.

Project Member Reported by qyears...@chromium.org, Dec 6 2016

Issue description

Example:

https://build.chromium.org/p/chromium.infra.cron/builders/w3c-test-autoroller/builds/7820/steps/update%20css/logs/stdio

Excerpt:
** Presubmit Warnings **
Are you sure that you want these files to contain Windows style line endings?
third_party/WebKit/LayoutTests/imported/csswg-test/css-ui-3/caret-color-009.html

Besides this, there are other possible things that could get committed to the upstream wpt or csswg-test repo that wouldn't pass our presubmit, e.g. end-of-line whitespace.

We've seen this kind of problem once or twice with other things that fail our presubmits; there are two types of solutions to this:
 1. Import the files exactly as they are, and disable Chromium presubmits for the imported/ directory.
 2. Apply a set of transformations on the files when importing, in order to ensure they should uphold our presubmits (e.g. remove end-of-line CR characters and whitespace).

The advantage of option 1 is that it's simpler; the advantage of option 2 is that it ensure that certain standards (e.g. line endings) are the same across the Chromium repo.
 
Labels: -Pri-3 Pri-1
Owner: qyears...@chromium.org
Status: Assigned (was: Unconfirmed)
This is blocking csswg-test import, so I want to solve it soon.

The easiest and least controversial fix would probably be to add a step in the import process that checks for CRLF endings, spits out a warning, then converts the file to use LF line endings.
Cc: r...@igalia.com
 Issue 662667  has been merged into this issue.
Cc: dpranke@chromium.org thakis@chromium.org
Update: Looking back, I did already decide earlier that excluding imported files from Chromium presubmit checks is probably reasonable: https://codereview.chromium.org/2374203003, and also when uploading CLs, the w3c-test-importer calls "git cl upload -f", which I would expect should mean "ignore presubmit warnings, only abort on presubmit errors".

The CRLF line ending presubmit check is still run because the black-list at the top of src/PRESUBMIT.py is not used when running some checks (including the CRLF check), and the upload still fails despite -f because it turns out that -f actually means "don't prompt, just fail", rather than "don't prompt and pass if there are just warnings" (https://cs.chromium.org/chromium/tools/depot_tools/presubmit_support.py?l=1281).

So:

 - If the -f flag was intended to mean "ignore warnings, force upload", then that should be changed.
 - If the black list at the top of src/PRESUBMIT.py was intended to apply to all of the checks, then that should be changed.

If either one of those changes are made, then w3c-test-importer should ignore CRLF line endings when importing tests, I believe.
Project Member

Comment 5 by bugdroid1@chromium.org, Dec 15 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools.git/+/516fe7f8818dd27a9a2161a79492371712e06849

commit 516fe7f8818dd27a9a2161a79492371712e06849
Author: Quinten Yearsley <qyearsley@google.com>
Date: Wed Dec 14 19:50:18 2016

Presubmit: if there are warnings and may_prompt is false, don't fail.

Rationale: The description of the -f flag to git cl upload is "force
yes to questions (don't prompt)", so when git cl upload -f is run,
I would expect it to abort on errors, but still continue on warnings.

When the -f is given, DoPresubmitChecks is called with may_prompt=False;
this CL would change the behavior of DoPresubmitChecks so that when
may_prompt is False and there are warnings but no errors, then that
means we will print warnings but not fail.

BUG= 671683 

Change-Id: Ie0f1ac1983d875226db8ad741cbce3dc0bc4eb96
Reviewed-on: https://chromium-review.googlesource.com/419148
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Quinten Yearsley <qyearsley@chromium.org>

[modify] https://crrev.com/516fe7f8818dd27a9a2161a79492371712e06849/presubmit_support.py
[modify] https://crrev.com/516fe7f8818dd27a9a2161a79492371712e06849/tests/presubmit_unittest.py

Status: Started (was: Assigned)
After the above change, the CSS changes are now being uploaded, but still not being committed -- they're failing in the chromium_presubmit try bot. The same presubmit warning is happening there, but is a fatal error there.

Example CSS import CL:
https://codereview.chromium.org/2593703002

One possible next step. Change top-level PRESUBMIT.py so that some checks *actually* aren't done for //third_party/WebKit. (Pan-project checks aren't done since this directory is in _EXCLUDED_PATHS, but _EXCLUDED_PATHS isn't used for all checks.)
Update: I made a CL https://codereview.chromium.org/2598743002 to convert line endings upon import, but then gsnedders@ noted that there's already a lint check upstream for line endings, so we're expecting this not to be an issue in the future, so once it's fixed upstream then this should be fixed.
Project Member

Comment 8 by bugdroid1@chromium.org, Dec 26 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools.git/+/e9c794ea4e4d1c62729757ed20c3dffb5a877914

commit e9c794ea4e4d1c62729757ed20c3dffb5a877914
Author: Quinten Yearsley <qyearsley@google.com>
Date: Mon Dec 19 18:35:32 2016

Make DoPresubmitChecks unit tests clearer.

This is a follow-up to https://chromium-review.googlesource.com/c/419148/;
the purpose is to confirm that the behavior is how we want it to be when
there are presubmit errors and warnings.

BUG= 671683 

Change-Id: I5b295c200d3db1a374e4294bdd78a777ae36c832
Reviewed-on: https://chromium-review.googlesource.com/420975
Commit-Queue: Quinten Yearsley <qyearsley@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>

[modify] https://crrev.com/e9c794ea4e4d1c62729757ed20c3dffb5a877914/tests/presubmit_unittest.py

Labels: -Pri-1 Pri-3
Summary: [W3C auto-import] Test import fails in presubmit when imported file has Windows-style line endings. (was: W3C auto-importer: Presubmit fails when imported file has Windows-style line endings.)
I made a change upstream to remove the CRLF endings, so this is no longer an immediate issue.
Cc: qyears...@chromium.org
Owner: ----
Status: Available (was: Started)
Components: Blink>Infra>Predictability
Components: -Blink>Infra
Cc: -qyears...@chromium.org
Owner: qyears...@chromium.org
Status: Fixed (was: Available)
I don't believe that this is an issue now (since I believe that the importer ignores presubmit warnings, and only halts on presubmit errors). Will re-open if this is actually an issue still.
Components: Blink>Infra>Ecosystem
Components: -Blink>Infra>Predictability

Sign in to add a comment