New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 690010 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Mar 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

WebRTC and libyuv bots wasting lots of bandwidth due to unversioned file cleaning

Project Member Reported by kjellander@chromium.org, Feb 8 2017

Issue description

Right before Christmas 2016 WebRTC moved away from checking out Chromium and symlinking into it to reuse the build toolchain, BUILD.gn files for shared dependencies and other goodies. Instead we now use and auto-roll Git subtree mirrors in our DEPS file (e.g. mirrors for src/build, src/third_party, src/tools etc).
Libyuv did a similar switch last week.

This saves a lot of disk space for everyone and avoids many hacks.
However, I noticed today that all unversioned directories that end up inside a Git mirror directory are wiped during bot_update because of this line in gclient_scm.py: https://cs.chromium.org/chromium/tools/depot_tools/gclient_scm.py?rcl=83051152f32458b829cd787dac37d03ce938d542&l=698

Example: 
https://build.chromium.org/p/client.webrtc/builders/Linux64%20Release/builds/11295/steps/bot_update/logs/stdio

________ running 'git reset --hard HEAD' in '/b/c/b/Linux64_Release/src/build'
[0:00:04] HEAD is now at a5b79d98 Revert of Add a script to merge the PGC files (patchset #6 id:120001 of https://codereview.chromium.org/2575543002/ )
Checked out revision c4b2c5ac665a2804730c523a75ed1cea5249d4df
[0:00:04] _____ removing unversioned directory linux/debian_wheezy_amd64-sysroot/
[0:00:07] _____ removing unversioned directory linux/debian_wheezy_i386-sysroot/
[0:00:08] _____ removing unversioned directory linux/ubuntu_precise_amd64-sysroot/
[0:00:10] Finished.
----------------------------------------

...
________ running 'git reset --hard HEAD' in '/b/c/b/Linux64_Release/src/third_party'
[0:00:43] HEAD is now at 8694363380 Add README.md to dom/html-related directories in LayoutTests.
Checked out revision 6549709ce756f2688159ee86dbb18a0b6219a2c1
[0:00:50] _____ removing unversioned directory boringssl/src/
[0:00:50] _____ removing unversioned directory catapult/
[0:00:52] _____ removing unversioned directory colorama/src/
[0:00:52] _____ removing unversioned directory ffmpeg/
[0:00:52] _____ removing unversioned directory gflags/
[0:00:52] _____ removing unversioned directory gtest-parallel/
[0:00:52] _____ removing unversioned directory jsoncpp/source/
[0:00:52] _____ removing unversioned directory libFuzzer/src/
[0:00:52] _____ removing unversioned directory libjpeg_turbo/
[0:00:53] _____ removing unversioned directory libsrtp/
[0:00:53] _____ removing unversioned directory libvpx/source/libvpx/
[0:00:53] _____ removing unversioned directory libyuv/
[0:00:53] _____ removing unversioned directory llvm-build/
[0:00:53] _____ removing unversioned directory lss/
[0:00:53] _____ removing unversioned directory nss/
[0:00:53] _____ removing unversioned directory openh264/src/
[0:00:53] _____ removing unversioned directory openmax_dl/
[0:00:53] _____ removing unversioned directory usrsctp/usrsctplib/
[0:00:53] _____ removing unversioned directory yasm/source/patched-yasm/
[0:00:53] Finished.
----------------------------------------

A lot of bandwidth is saved by the Git cache, but not all, as for the sysroots in src/build/linux etc.
Same with Android's Google Play Services SDK.

Is there a good way to avoid this waste?
 

Comment 1 by aga...@chromium.org, Feb 13 2017

Owner: phajdan.jr@chromium.org
Status: Assigned (was: Untriaged)
+pawel who I believe made a change regarding "git reset --hard HEAD" recently
Cc: charujain@chromium.org
Pawel: can you respond to #1?
Cc: phajdan.jr@chromium.org
Owner: kjellander@chromium.org
Nevermind #2, this is not related to 'git reset --hard HEAD', it's the cleaning at https://cs.chromium.org/chromium/tools/depot_tools/gclient_scm.py?rcl=83051152f32458b829cd787dac37d03ce938d542&l=698 as mentioned in the bug description.

I've found a reasonable workaround for this though: I'll just create CLs for a few .gitignore files in src/third_party and src/build. They're needed when using these dirs as separate Git repos (but not in Chrome, since the top-level .gitignore covers the stuff that gets deleted here).

Doing that shouldn't affect Chromium: they'll just overlap with already existing entries in https://cs.chromium.org/chromium/src/.gitignore but nothing will break if the ones I'm adding is getting out of sync. It'll be in WebRTC and libyuv's interest to keep them at least somewhat up to date.
Status: Started (was: Assigned)
I posted https://codereview.chromium.org/2710593005/ in which I realized it's probably better to just move the entries out of the top-level .gitignore instead to avoid them getting out of sync. There are already 51 other .gitignore files in the src/ repo so adding 2 more won't make any difference.
Project Member

Comment 5 by bugdroid1@chromium.org, Feb 22 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/68d02518d39937443244001624fa500e00aad408

commit 68d02518d39937443244001624fa500e00aad408
Author: kjellander <kjellander@chromium.org>
Date: Wed Feb 22 19:52:16 2017

Add .gitignore files in build/ and third_party/

Having this, a lot of wasted bandwidth is avoided for projects
that use these directories as DEPS entries, since the top-level
Chromium .gitignore doesn't exist in those cases.

The src/ repo already contained 51 other .gitignore files anyway,
so it's hard to claim the top-level file has to be the only one.

BUG= 690010 

Review-Url: https://codereview.chromium.org/2710593005
Cr-Commit-Position: refs/heads/master@{#452167}

[modify] https://crrev.com/68d02518d39937443244001624fa500e00aad408/.gitignore
[add] https://crrev.com/68d02518d39937443244001624fa500e00aad408/build/.gitignore
[add] https://crrev.com/68d02518d39937443244001624fa500e00aad408/third_party/.gitignore

Labels: -Pri-3 Pri-2
Owner: iannucci@chromium.org
Status: Assigned (was: Started)
So... my .gitignore CL above was a little late to the party since https://chromium-review.googlesource.com/c/444308/ landed 5 days ago and took over the cleaning of these things. Now all these directories are wiped with a "Deleting residual after reset" message instead (example: [1]).

Adding the .gitignore files in build/ and third_party/ doesn't affect that cleaning, but I guess it would have solved this problem otherwise.

Robbie: could you help us on this? It has significant effect on our sync times, and it's loading Chrome infra unnecessary.

[1]: https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fclient.webrtc%2FLinux64_Debug%2F11850%2F%2B%2Frecipes%2Fsteps%2Fbot_update%2F0%2Fstdout
Cc: iannucci@chromium.org
Owner: kjellander@chromium.org
Actually, I mistakenly assumed we had rolled in the .gitignore files, but we haven't yet rolled in #452167. The latest roll CL was doing 452017:452148 in https://codereview.webrtc.org/2712753002/

So let's wait a few hours and and see, maybe it actually fixes it!
Cc: -iannucci@chromium.org
Owner: iannucci@chromium.org
Now that the .gitignore files are rolled in I can verify that they didn't have any effect.
The "Deleting residual after reset" functionality [1] is wiping these directories now. 

Robbie: Is there any way we could avoid this for projects using build/third_party Git subtree mirrors? I imagine we could avoid passing --delete_unversioned_trees, but that sounds a bit risky for other scenarios like when deps are added/removed etc, right? 

[1]: https://cs.chromium.org/chromium/tools/depot_tools/gclient_scm.py?rcl=53b8550acf5130d033253bc2743be99c12c51f02&l=314
Status: Fixed (was: Assigned)
So, it appears that this magically has resolved itself or was I verifying this in the wrong way. It actually DID solve the problem and now there are just a few entries that are wiped (mostly *.pyc files), so this is no longer an issue! Yay!

Sign in to add a comment