linux_chromium_rel_ng is having similar problems and is trying to build code that no longer exists:

https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/218917/steps/checkdeps%20%28retry%20summary%29/logs/stdio

E.g.:
step returned non-zero exit code: 1
@@@STEP_TEXT@<br/>failures:<br/>/b/build/slave/linux/build/src/third_party/cld/encodings/compact_lang_det/generated/compact_lang_det_generated_quadschrome.cc: encodings/compact_lang_det/cldutil.h<br/>@@@
@@@STEP_FAILURE@@@
@@@SET_BUILD_PROPERTY@failure_type@"TEST_FAILURE"@@@
@@@SET_BUILD_PROPERTY@failure_hash@"ca5c2310fb8fb93ef6d8fb2149e9b78e4f9c3946"@@@


Is this just as simple as adding a landmine?

IIRC the license checker does indeed fail on "directories that exist under third_party but don't contain any files", because it expects all those directories to have a README.chromium and an empty directory does not..

git normally deletes removed directories that *actually* become empty, but if they have some temp/generated file in them, or if the patch isn't being applied purely by git, then it might not work..

Thanks, Torne. Troopers, need some guidance on what to do here: force it through and manually clean the bots seems like the only viable option.

You can't really force it through the bot, since there's no D (delete) directive on the cld/ folder, apply_issue.py doesn't know to delete the folder.

That would put as at a bit of an impasse :)

Can we manually delete the dir on the bots after the patch lands, or perhaps upgrade apply_issue.py to delete empty directories so this doesn't happen?

Alternatively, maybe we can change the license check script to exclude dirs that are empty?

+thakis as fyi

You mean delete the dir while the build is running, after bot_update but before check licenses?

I'm not sure if this would work, but I think it's worth a try.  Can you upload the patch using "git cl upload --gerrit"

hinoka@: The command fails with this error:

ERROR: Gerrit commit-msg hook not available.
Traceback (most recent call last):
  File "/usr/local/google/home/andrewhayden/lib/depot_tools/git_cl.py", line 4843, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/usr/local/google/home/andrewhayden/lib/depot_tools/git_cl.py", line 4825, in main
    return dispatcher.execute(OptionParser(), argv)
  File "/usr/local/google/home/andrewhayden/lib/depot_tools/subcommand.py", line 252, in execute
    return command(parser, args[1:])
  File "/usr/local/google/home/andrewhayden/lib/depot_tools/git_cl.py", line 3558, in CMDupload
    return cl.CMDUpload(options, args, orig_args)
  File "/usr/local/google/home/andrewhayden/lib/depot_tools/git_cl.py", line 1394, in CMDUpload
    ret = self.CMDUploadChange(options, git_diff_args, change)
  File "/usr/local/google/home/andrewhayden/lib/depot_tools/git_cl.py", line 2430, in CMDUploadChange
    args))
  File "/usr/local/google/home/andrewhayden/lib/depot_tools/git_cl.py", line 2575, in set_description
    lines = [line.rstrip() for line in desc]
TypeError: 'NoneType' object is not iterable


I've checked that my depot_tools is up to date and it is, I'm at cf48206332e34297adaf8bfed76f0798392a6f3f from Tuesday, 26 April 2016.

Any other ideas?

PS, I checked on webview_licenses.py and it is calling tools/licenses.py:FindThirdPartyDirsWithFiles which already filters out empty dirs, so this is just a problem with the patch applier leaving behind some files somewhere (or in a temp directory, or something).

This CL looks like it still hasn't landed; do you need someone to take a look at this?

(Ping) Is there an update to this bug? Have we determined why the directories aren't empty?

I signed onto the bot in #0, build85-b4, and confirmed:
1) third_party/cld is still there, and is fully populated.
2) third_party/cld_2 is also there.
3) DEPS in chromium/ only references cld_2.

+hinoka@, what normally takes care of cleaning up directories that become unreferenced? gclient? bot_update?

The directories aren't empty because:
* apply_issue.py is incapable of deleting the directory because
** Rietveld is incapable of representing deleted directories because
*** "git diff" on a deleted directory does not actually represent the deleted directory

I thought that uploading to gerrit instead might help (since bot_update can also update from a gerrit ref), but "git cl upload --gerrit" doesn't work right now, so that's a non-starter.

I suspect in the past directory deletions have been done with a direct commit (after notifying a sheriff).

I don't think this is a diff-related problem. The directory is only included b/c it's in DEPS, so a diff wouldn't show a difference here.

The CL doesn't touch deps, it's just deleting a checked in non-DEPS folder.  (I assume a DEPS change will come afterwards?).  Also gclient is (now) capable of deleting unversioned deps directories so that's wouldn't have been an issue.

Ah nevermind, I assumed the previous one was DEPS'd too.

So there was talk about landmining the old directory. Is this still the thing to do?

Sorry, I've been unexpectedly out sick. I'm back now, and yes, the question is still about landmining. I will try adding a landmine into the CL and see if this fixes the bots. If not, I think we will need to force this through with help from the troopers to clean up the detritus.

Let's see if the landmine is effective. I'll rebase the CL shortly and give it a shot. If not I'll set up a meeting with the troopers and sheriff to land this manually and fix things up before the world catches on fire.

CL has been updated with a landmine, and new tryjobs have been started. Fingers crossed.

Landmind did not fix the issue:
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/67779/steps/webview_licenses%20%28with%20patch%29/logs/stdio

Infra: Can we set up a time that is mutually acceptable to force-submit this and then clean up the directory on the bots?

I looked into landmines a little deeper, all they do is clobber the "out/" directory, which isn't useful here.

Manually force submitting and cleaning up all of the bots is not really an option here. In addition to being incredibly time consuming, it will also allow for the possibility of this issue resurfacing on bots that were offline / unavailable during the cleanup indefinitely. I'll talk to hinoka@ and find a better solution.

I ran "apply_issue" locally. There was one remaining file:

third_party/cld/encodings/compact_lang_det/generated/compact_lang_det_generated_quadschrome.cc

When I deleted it:
$ git rm third_party/cld/encodings/compact_lang_det/generated/compact_lang_det_generated_quadschrome.cc

The directory was deleted, as expected.

The file is checked in:
https://chromium.googlesource.com/chromium/src/+/704e8bc7ebcd4f699df985b7b8685f651b3d4cf1/third_party/cld/encodings/compact_lang_det/generated/compact_lang_det_generated_quadschrome.cc

I don't know why your patch doesn't have it, but if this experiment translates to the bots, deleting that file in your patchset should make the entire "third_party/cld" directory go away.

"git cl upload" says that the file is too big to upload the change for. I am not sure how to fix that without the gerrit stuff mentioned before. I suppose I could... like... delete bits of it in a few CLs, but that seems kind of nuts. WDYT?

I see... 70k lines wow. As hacky as it seems, deleting it in parts might actually be the cleanest way to go given the circumstances.

If the file is really no longer referenced, you could also "git cl land" (direct push) the deletion of that one file and go ahead with your current patch.

Also while I'm grepping, I noticed that the following files reference "third_party/cld" but aren't in your CL (https://codereview.chromium.org/1959063004/):
- chrome/common/extensions/api/i18n.json
- chrome/common/extensions/api/tabs.json

Let me see what I can do, thanks.

Yeah, I saw the json files but I'll clean those up later. It's comments, and they've been outdated for *years* at this point (plain wrong). Trying to keep this to compile-time stuffs.

OK, let's be as safe as possible. First let's land this smaller patch to just kill the DEPS:
https://codereview.chromium.org/1972543003/

After that we can nuke the problematic file and be sure it won't break the tree.

New trooper checking in. There's a lot of context here; is there an update?

Seems like there's not work for a trooper to do. Assigning this to Andrew Hayden since it seems like his CLs should have this well in hand. Andrew, if it turns out you need more trooper help, please return this to the trooper queue, or file a more specific bug and have it block this one.

I need Nico to LGTM https://codereview.chromium.org/1972543003/.

Landed the DEPS removal change, now working on getting rid of the problematic file:
third_party/cld/encodings/compact_lang_det/generated/compact_lang_det_generated_quadschrome.cc

I figured out (sigh) that the reason --gerrit isn't working is because of recent changes to squashing:
https://groups.google.com/a/chromium.org/forum/#!topic/infra-announce/30PqVAOJdbc

If I add "--squash" to my "git cl upload --gerrit" it works, so I'm going to get this moving again with the full delete.

New review posted:
https://chromium-review.googlesource.com/#/c/345841

And for posterity and help to those who come after, tryjobs are done by ( issue 599931 ):
git cl set-commit --dry-run

Marking as resolved since we have a gerrit solution now.

Gerrit change is approved and ready to land:
https://chromium-review.googlesource.com/#/c/345841

But I have no "submit" button, despite being logged in and having committer access. Troopers, can you submit this? Or am I missing some step documented elsewhere?

Oh... Chromium doesn't use gerrit, which is why the submit button is disabled.

It looks like it's on gerrit because the patch is too big for rietveld? You'll need to land it manually I think (please coordinate with the sherrifs to do so). Landing via gerrit wouldn't trigger CQ anyway; it would be no different than a direct land.

Andrew, I'm assigning this to you, but I'm confused why you thought that chromium/src supports gerrit? It's never supported gerrit (though, indeed, we're working on doing that soon). Is there some documentation which states that to be the case?

AFAIK, there's currently no way to land a large patch like the one you have via the CQ in chromium/src. +tandrii who may know differently than I.

iannucci@ good questions. I wonder about the same.

I didn't know we even had Gerrit codereview enabled for Chromium repo.
Also, the problem is not just that submit is disabled, but also that git cl upload --gerrit isn't yet able to work with Chromium repo***

There is indeed no way yet to trigger CQ with Gerrit CL for Chromium.
So, please do:
[0. (optional) abandon your gerrit CL]
1. Notify sheriffs that you are gonna land your CL manually and that they should revert your CL at the sign of trouble. Note on the CL description that reverts should be landed manually bypassing CQ. 
2. On your local branch that you've uploaded to Rietveld do:
  git cl land
3. Watch out for any breakages on waterfall or tryjobs shortly afterwards.

*** it uploaded the CL against refs/for/refs/heads/master, while in really should have done that against refs/for/refs/pending/heads/master.

I thought it supported gerrit because of #10. Sorry for the confusion. So after a bunch of roundabout stuff, we're back to "git cl land", which is fine. I will coordinate with the sheriffs to get this done.

Change has landed as c0c0cab516d52d7e58397b7c2d61839b1be69d5b:
https://chromium.googlesource.com/chromium/src/+/c0c0cab516d52d7e58397b7c2d61839b1be69d5b

Done. I contacted two of the on-duty sheriffs today to let them know about this and got their approval prior to submitting.