New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 111570 link

Starred by 21 users

Issue metadata

Status: WontFix
Owner: ----
Closed: May 2015
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: ----

Blocked on:
issue 224072
issue 241276

Blocking:
issue 363658



Sign in to add a comment

reduce size of src.git through filter-branch

Project Member Reported by c...@chromium.org, Jan 27 2012

Issue description

Summary: reduce size of src.git through filter-branch

According to erg@ (cc'd) on 10/5/2011:

Here is the final "easy" filter-branch command:

time git filter-branch --index-filter "git rm -r -f --cached
--ignore-unmatch --quiet ceee/ o3d/ gears/
third_party/{buildbot,boost,cygwin,cygwin_src,gnu,lighttpd,python_24,scons,svn}/
third_party/ffmpeg/binaries/chromium/
chrome/tools/test/reference_build/
chrome_frame/tools/test/reference_build/
webkit/tools/test/reference_build/
webkit/data/{layout_tests,layout_test_results}/
chrome/test/data/safe_browsing/{old,filter}/ tools/buildbot/
chrome/third_party/hunspell/
chrome/tools/{icudt38.dll,optipng.exe,sqlite.exe,sqlite3_analyzer.exe}
third_party/cld/cld.ncb
chrome/test/data/profiles/complex_theme/Default/Cached\ Theme\
Images/" --prune-empty HEAD

With all that removed, the pruned packfile is down to 326 megabytes
from 1.2 gigabytes. We could be doing better by filtering out some
churning binary data directories and readding the ToT version as a new
commit, but that might not be worth the effort.

Timing is important here. I'm pretty sure that filter-branched
repositories are no longer valid git-svn repositories so this will
have to happen after the svn server is put in read-only mode.

---

Here's a broken down list of everything it removes:

* Dead projects:
*** ceee/
*** o3d/
*** gears/
* Third party code that shouldn't be in here at all:
*** buildbot (included twice! third_party/ and tools/)
*** boost (was checked in but immediately removed!)
*** 3rd party binaries!
(cygwin,cygwin_surce,gnu,lighttpd,python_24,svn,chrome/tools/*.{dll,exe})
*** chrome/third_party/hunspell (several multimegabyte dictionaries
that are now pulled in through DEPS)
* Reference builds!
*** chrome, chrome_frame, webkit
* Layout tests and layout test results
* chrome/test/data/safe_browsing/{old,filter}/
*** Over 60 megabytes of safebrowsing test data that was deleted in early 2009.
* third_party/cld/cld.ncb
*** A 9 megabyte binary file that was removed in mid 2010.
* chrome/test/data/profiles/complex_theme/Default/Cached\ Theme\ Images/
*** A few megabytes that I recognized from the old theme system; not
that important but tossable.
 

Comment 1 by c...@chromium.org, Jan 27 2012

I believe this can be done on src.git with some coordination for the current src.git users.  We'll need to run tests on a src.git clone on the server (with simulated user experiences) to ensure things work before trying it on the real src.git repo.

Here's a mock-up of the process:
> 1. User clones chromium/src.git, runs git svn init, etc, makes some changes.
> 2. Admin clones the chromium/src.git repo.
> 3. Admin runs the filter-branch command on the cloned repo.  This generates
> many new SHA1 revs along with a new HEAD rev.
> 4. Admin pushes the new commits to chromium/src.git which updates HEAD on
> the master branch.
> 5. User fetches the latest HEAD which downloads all of the new commits,
> rebases, removes .git/svn, and runs git cl dcommit.

Evan made some extra notes:
> One final idea: you can create a "grafts" file, which tells git to add
> some history.  In the above example, you can make grafts say that F'
> is a child of both D' (which it is) *and* F.  This means that
> (simplifying) any git operations will only look back "as far as F'"
> and not get confused by the messed up history when doing merges etc.
> But I think rebase would still be broken.
> 
> It might be worth making a small script "cleanup-my-stuff" which
> creates graft files / rebases all your branches / clobbers your
> git-svn metadata that people can run.  See e.g. "git for-each-ref" for
> builtins made for iterating through branches.  It's pretty hard to
> make this kind of surgery (filter-branch) work automatically.
Labels: nomedia
Status: Available
triaging this bug to the available queue.  If you have time to do this any time soon, now is the time to grab the bug.

Comment 4 by e...@chromium.org, Apr 3 2012

Another library path to add to this list that was temporarily added and then immediately reverted: third_party/qcms/

(In fact, doing another pass across history would probably be a good idea since its been months and I bet there's even more fun stuff in our repository history now.)

Comment 5 by evan@chromium.org, Jun 6 2012

you could remove all the chrome-bot commits, though it likely doesn't impact space much

Comment 6 by evan@chromium.org, Jun 11 2012

Cc: -evan@chromium.org
(Un-ccing myself from bugs.)

Comment 7 by evan@chromium.org, Jun 11 2012

(Un-ccing myself from bugs.)

Comment 8 by e...@chromium.org, Jun 22 2012

qcms looks like a real library; it's reversion was temporary.
Other things we should be filtering and that I'm going to test over the weekend:

Old vcproj files in some (but not all) directories:
- {app,base,breakpad,gears,google_update,media,net,skia,testing,webkit}/**.vcproj
- chrome/{app,browser,common,plugin,renderer,test,tools,utility,views,worker}/**.vcproj
- {base,chrome,media,net,webkit}/**.sln

Same with old *.scons and SConscript files:
- {base,breakpad,build,chrome,media,net,printing,sandbox,third_party,webkit}/**.scons
- {base,breakpad,chrome,net,rlz,sandbox,sdch,skia,webkit}/**SConscript

Xcode files:
- **/project.pbxproj (The real file in an xcodeproj.)

More binary resources:
- chrome/app/theme/*.png (lots of old versions; moved elsewhere in tree)
- chrome/app/theme/{touch,ash}/

Will report how doing the above affected the repository on Monday. This should give me a filtered repository once again on which I can try to do some more improvements.

To think about:
- chrome/common/extensions/docs carries around a LOT of old zip files in its history.
- chrome/test/data/. Lots of old binary files not in the current version.

Comment 9 by e...@chromium.org, Jun 22 2012

Sigh. We have tens of megabytes of accidently committed high res photos in i/resources/aura/wallpapers/ to filter:

ui/resources/aura/wallpapers/{waltersoestbergen,wallpaper,vitaliprokopenko,stefanoronchi,pauloflop,olegzhukov,neilkremer,mikereyfman,michelbricteux,markbridger,mariomoreno,johannesvandonge,charlesdavey,a_dummy)*.jpg

Comment 10 by szager@google.com, Jun 22 2012

Just as a point of reference:

http://commondatastorage.googleapis.com/chromium-git-bundles/index.html

src-bare.zip is just the chromium/src.git repository.
src-full.zip is chromium/src.git plus all non-private dependencies (including all os-specific dependencies).

Thanks for your work on this!

Stefan

Comment 11 by e...@chromium.org, Jun 28 2012

So I have the packfile on src.git down from 1.8 gigs to 366 megabytes with the attached script. Roughly 20% of the original size.

This not breaking the tree requires that https://chromiumcodereview.appspot.com/10698009/ be committed first, where I rename a few test data directories that have large binaries and have historically churned quite a bit.

I think at this point, I'm approaching the theoretical limit of what can be done without major disruption. The only other thing I'd like to do is filter out the old mac pdf images, and that's sort of blocked on thakis completing the mac hidpi work.
git-prunechrome.sh
2.2 KB View Download

Comment 12 by cmp@google.com, Aug 25 2012

Labels: NewGit
Project Member

Comment 13 by bugdroid1@chromium.org, Aug 29 2012

The following revision refers to this bug:
    http://src.chromium.org/viewvc/chrome?view=rev&revision=153933

------------------------------------------------------------------------
r153933 | erg@chromium.org | 2012-08-29T19:07:03.795513Z

Changed paths:
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/Top Sites?r1=153933&r2=153932&pathrev=153933
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/ui/ui_test.h?r1=153933&r2=153932&pathrev=153933
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Cookies?r1=153933&r2=153932&pathrev=153933
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/PreferencesTemplate?r1=153933&r2=153932&pathrev=153933
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/startup_test.cc?r1=153933&r2=153932&pathrev=153933
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Full Text Index?r1=153933&r2=153932&pathrev=153933
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Extensions/mblmlcbknbnfebdfjnolmcapmdofhmme/Current Version?r1=153933&r2=153932&pathrev=153933
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default?r1=153933&r2=153932&pathrev=153933
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/complex_theme?r1=153933&r2=153932&pathrev=153933
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Extensions/mblmlcbknbnfebdfjnolmcapmdofhmme/1.1/i/agxjaHJvbWV0aGVtZXNyDAsSBEZpbGUYjNgCDA?r1=153933&r2=153932&pathrev=153933
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Extensions/mblmlcbknbnfebdfjnolmcapmdofhmme?r1=153933&r2=153932&pathrev=153933
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Extensions/mblmlcbknbnfebdfjnolmcapmdofhmme/1.1/i/agxjaHJvbWV0aGVtZXNyDAsSBEZpbGUYnMgCDA?r1=153933&r2=153932&pathrev=153933
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/feature_startup_test.cc?r1=153933&r2=153932&pathrev=153933
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Extensions/mblmlcbknbnfebdfjnolmcapmdofhmme/1.1/Cached Theme.pak?r1=153933&r2=153932&pathrev=153933
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/Preferences?r1=153933&r2=153932&pathrev=153933
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/History?r1=153933&r2=153932&pathrev=153933
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/gtk_theme?r1=153933&r2=153932&pathrev=153933
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/custom_frame_gtk_theme?r1=153933&r2=153932&pathrev=153933
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/Visited Links?r1=153933&r2=153932&pathrev=153933
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/Favicons?r1=153933&r2=153932&pathrev=153933
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Extensions/mblmlcbknbnfebdfjnolmcapmdofhmme/1.1/i/agxjaHJvbWV0aGVtZXNyDAsSBEZpbGUYwsgCDA?r1=153933&r2=153932&pathrev=153933
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/ui/ui_test.cc?r1=153933&r2=153932&pathrev=153933
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme?r1=153933&r2=153932&pathrev=153933
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/custom_frame?r1=153933&r2=153932&pathrev=153933
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Extensions/mblmlcbknbnfebdfjnolmcapmdofhmme/1.1/i/agxjaHJvbWV0aGVtZXNyDAsSBEZpbGUYw8gCDA?r1=153933&r2=153932&pathrev=153933
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/typical_history?r1=153933&r2=153932&pathrev=153933
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Extensions/mblmlcbknbnfebdfjnolmcapmdofhmme/1.1/i?r1=153933&r2=153932&pathrev=153933
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Extensions?r1=153933&r2=153932&pathrev=153933
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Extensions/mblmlcbknbnfebdfjnolmcapmdofhmme/1.1/i/agxjaHJvbWV0aGVtZXNyDAsSBEZpbGUY_c8CDA?r1=153933&r2=153932&pathrev=153933
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/History?r1=153933&r2=153932&pathrev=153933
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/README.txt?r1=153933&r2=153932&pathrev=153933
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Extensions/mblmlcbknbnfebdfjnolmcapmdofhmme/1.1?r1=153933&r2=153932&pathrev=153933
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Visited Links?r1=153933&r2=153932&pathrev=153933
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/Cookies?r1=153933&r2=153932&pathrev=153933
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/Full Text Index?r1=153933&r2=153932&pathrev=153933
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme?r1=153933&r2=153932&pathrev=153933
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default?r1=153933&r2=153932&pathrev=153933
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Extensions/mblmlcbknbnfebdfjnolmcapmdofhmme/1.1/i/agxjaHJvbWV0aGVtZXNyDAsSBEZpbGUYj9gCDA?r1=153933&r2=153932&pathrev=153933
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Extensions/mblmlcbknbnfebdfjnolmcapmdofhmme/1.1/manifest.json?r1=153933&r2=153932&pathrev=153933

Make history easily filterable.

We're renaming the two profile directories that we want to keep, and dropping
GTK specific tests in anticipation of the changeover later this year.

Every change to the history, theme or profile format required a new giant
binary file added to our repository. There are tens of versions of the history
database alone, each weighing at over 10 megabytes. In the proposed pruned git
repository, these are the largest files.

This standardizes both remaining test profile directories on the same set of
binaries (they're currently out of sync), and moves them to new path names so
we can purge the previous paths from history.

BUG= 111570 
TEST=none

Review URL: https://chromiumcodereview.appspot.com/10698009
------------------------------------------------------------------------
Project Member

Comment 14 by bugdroid1@chromium.org, Aug 29 2012

The following revision refers to this bug:
    http://src.chromium.org/viewvc/chrome?view=rev&revision=153952

------------------------------------------------------------------------
r153952 | erg@chromium.org | 2012-08-29T19:38:41.079769Z

Changed paths:
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/feature_startup_test.cc?r1=153952&r2=153951&pathrev=153952
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme?r1=153952&r2=153951&pathrev=153952
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/ui/ui_test.cc?r1=153952&r2=153951&pathrev=153952
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/ui/ui_test.h?r1=153952&r2=153951&pathrev=153952
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme?r1=153952&r2=153951&pathrev=153952
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/startup_test.cc?r1=153952&r2=153951&pathrev=153952

Revert 153933 - Make history easily filterable.

This breaks tests that use hand rolled paths instead of the main path methods.

We're renaming the two profile directories that we want to keep, and dropping
GTK specific tests in anticipation of the changeover later this year.

Every change to the history, theme or profile format required a new giant
binary file added to our repository. There are tens of versions of the history
database alone, each weighing at over 10 megabytes. In the proposed pruned git
repository, these are the largest files.

This standardizes both remaining test profile directories on the same set of
binaries (they're currently out of sync), and moves them to new path names so
we can purge the previous paths from history.

BUG= 111570 
TEST=none

Review URL: https://chromiumcodereview.appspot.com/10698009

TBR=erg@chromium.org
------------------------------------------------------------------------
Project Member

Comment 15 by bugdroid1@chromium.org, Aug 31 2012

The following revision refers to this bug:
    http://src.chromium.org/viewvc/chrome?view=rev&revision=154448

------------------------------------------------------------------------
r154448 | erg@chromium.org | 2012-08-31T16:58:28.526437Z

Changed paths:
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/Top Sites?r1=154448&r2=154447&pathrev=154448
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/ui/ui_test.h?r1=154448&r2=154447&pathrev=154448
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Cookies?r1=154448&r2=154447&pathrev=154448
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/PreferencesTemplate?r1=154448&r2=154447&pathrev=154448
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/startup_test.cc?r1=154448&r2=154447&pathrev=154448
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Full Text Index?r1=154448&r2=154447&pathrev=154448
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Extensions/mblmlcbknbnfebdfjnolmcapmdofhmme/Current Version?r1=154448&r2=154447&pathrev=154448
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default?r1=154448&r2=154447&pathrev=154448
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/complex_theme?r1=154448&r2=154447&pathrev=154448
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Extensions/mblmlcbknbnfebdfjnolmcapmdofhmme/1.1/i/agxjaHJvbWV0aGVtZXNyDAsSBEZpbGUYjNgCDA?r1=154448&r2=154447&pathrev=154448
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/history_unittest.cc?r1=154448&r2=154447&pathrev=154448
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Extensions/mblmlcbknbnfebdfjnolmcapmdofhmme/1.1/i/agxjaHJvbWV0aGVtZXNyDAsSBEZpbGUYnMgCDA?r1=154448&r2=154447&pathrev=154448
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Extensions/mblmlcbknbnfebdfjnolmcapmdofhmme?r1=154448&r2=154447&pathrev=154448
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/feature_startup_test.cc?r1=154448&r2=154447&pathrev=154448
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Extensions/mblmlcbknbnfebdfjnolmcapmdofhmme/1.1/Cached Theme.pak?r1=154448&r2=154447&pathrev=154448
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/Preferences?r1=154448&r2=154447&pathrev=154448
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/History?r1=154448&r2=154447&pathrev=154448
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/gtk_theme?r1=154448&r2=154447&pathrev=154448
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/custom_frame_gtk_theme?r1=154448&r2=154447&pathrev=154448
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/Visited Links?r1=154448&r2=154447&pathrev=154448
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/Favicons?r1=154448&r2=154447&pathrev=154448
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Extensions/mblmlcbknbnfebdfjnolmcapmdofhmme/1.1/i/agxjaHJvbWV0aGVtZXNyDAsSBEZpbGUYwsgCDA?r1=154448&r2=154447&pathrev=154448
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/ui/ui_test.cc?r1=154448&r2=154447&pathrev=154448
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme?r1=154448&r2=154447&pathrev=154448
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/themes/browser_theme_pack_unittest.cc?r1=154448&r2=154447&pathrev=154448
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/custom_frame?r1=154448&r2=154447&pathrev=154448
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Extensions/mblmlcbknbnfebdfjnolmcapmdofhmme/1.1/i/agxjaHJvbWV0aGVtZXNyDAsSBEZpbGUYw8gCDA?r1=154448&r2=154447&pathrev=154448
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/typical_history?r1=154448&r2=154447&pathrev=154448
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Extensions/mblmlcbknbnfebdfjnolmcapmdofhmme/1.1/i?r1=154448&r2=154447&pathrev=154448
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Extensions?r1=154448&r2=154447&pathrev=154448
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Extensions/mblmlcbknbnfebdfjnolmcapmdofhmme/1.1/i/agxjaHJvbWV0aGVtZXNyDAsSBEZpbGUY_c8CDA?r1=154448&r2=154447&pathrev=154448
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/History?r1=154448&r2=154447&pathrev=154448
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/README.txt?r1=154448&r2=154447&pathrev=154448
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Extensions/mblmlcbknbnfebdfjnolmcapmdofhmme/1.1?r1=154448&r2=154447&pathrev=154448
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Visited Links?r1=154448&r2=154447&pathrev=154448
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/Cookies?r1=154448&r2=154447&pathrev=154448
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/Full Text Index?r1=154448&r2=154447&pathrev=154448
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme?r1=154448&r2=154447&pathrev=154448
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default?r1=154448&r2=154447&pathrev=154448
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Extensions/mblmlcbknbnfebdfjnolmcapmdofhmme/1.1/i/agxjaHJvbWV0aGVtZXNyDAsSBEZpbGUYj9gCDA?r1=154448&r2=154447&pathrev=154448
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Extensions/mblmlcbknbnfebdfjnolmcapmdofhmme/1.1/manifest.json?r1=154448&r2=154447&pathrev=154448

Now with unit test fixes. [Try 2 with fixes for unit tests.]

Make history easily filterable.

We're renaming the two profile directories that we want to keep, and dropping
GTK specific tests in anticipation of the changeover later this year.

Every change to the history, theme or profile format required a new giant
binary file added to our repository. There are tens of versions of the history
database alone, each weighing at over 10 megabytes. In the proposed pruned git
repository, these are the largest files.

This standardizes both remaining test profile directories on the same set of
binaries (they're currently out of sync), and moves them to new path names so
we can purge the previous paths from history.

BUG= 111570 
TEST=none
TBR=pkotwicz,brettw
First Review URL: https://chromiumcodereview.appspot.com/10698009

Review URL: https://chromiumcodereview.appspot.com/10897033
------------------------------------------------------------------------

Comment 16 by c...@chromium.org, Mar 6 2013

Cc: -nsylvain@chromium.org

Comment 17 by e...@chromium.org, Mar 27 2013

Blockedon: chromium:224072
Project Member

Comment 18 by bugdroid1@chromium.org, May 10 2013

------------------------------------------------------------------------
r199526 | erg@chromium.org | 2013-05-10T20:20:29.617322Z

Changed paths:
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/perf_ui_test_suite.h?r1=199526&r2=199525&pathrev=199526
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/top_sites_impl.cc?r1=199526&r2=199525&pathrev=199526
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Full Text Index?r1=199526&r2=199525&pathrev=199526
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/History Provider Cache?r1=199526&r2=199525&pathrev=199526
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/top_sites_impl.h?r1=199526&r2=199525&pathrev=199526
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/History?r1=199526&r2=199525&pathrev=199526
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/top_sites_cache.cc?r1=199526&r2=199525&pathrev=199526
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/chrome_tests.gypi?r1=199526&r2=199525&pathrev=199526
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/history_unittest.cc?r1=199526&r2=199525&pathrev=199526
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/generate_profile.cc?r1=199526&r2=199525&pathrev=199526
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/feature_startup_test.cc?r1=199526&r2=199525&pathrev=199526
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Visited Links?r1=199526&r2=199525&pathrev=199526
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/top_sites_likely_impl.cc?r1=199526&r2=199525&pathrev=199526
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/generate_profile.h?r1=199526&r2=199525&pathrev=199526
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/History-journal?r1=199526&r2=199525&pathrev=199526
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/top_sites_likely_impl.h?r1=199526&r2=199525&pathrev=199526
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Extensions/mblmlcbknbnfebdfjnolmcapmdofhmme/1.1/Cached Theme.pak?r1=199526&r2=199525&pathrev=199526
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/History Index 2013-01?r1=199526&r2=199525&pathrev=199526
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/tools/profiles/generate_profile.cc?r1=199526&r2=199525&pathrev=199526
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/top_sites.h?r1=199526&r2=199525&pathrev=199526
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/run_all_perfuitests.cc?r1=199526&r2=199525&pathrev=199526
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/History Index 2013-01-journal?r1=199526&r2=199525&pathrev=199526
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/ui/ui_test.cc?r1=199526&r2=199525&pathrev=199526
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme?r1=199526&r2=199525&pathrev=199526
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/ui/ui_test.h?r1=199526&r2=199525&pathrev=199526
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Cookies?r1=199526&r2=199525&pathrev=199526
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/history_types.h?r1=199526&r2=199525&pathrev=199526
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/startup_test.cc?r1=199526&r2=199525&pathrev=199526
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/perf_ui_test_suite.cc?r1=199526&r2=199525&pathrev=199526
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/PreferencesTemplate?r1=199526&r2=199525&pathrev=199526

Rebuild test history databases when starting up performance_ui_tests.

Previously, whenever the theme or history database format were modified, a
large binary database or pak file had to be committed. This was only done
intermittently, messing up some perf expectations. Since last June, this
has added over 100 megabytes to our repository weight.

By generating the SQLite databases and Cached Theme.pak files at startup
of the one test suite that uses them, we're guaranteed that we'll always
be using the current versions and won't add upgrade costs to our
performance_ui_tests. We'll also not continuously add to new binary
resources to the permanent git archive.

BUG= 111570 
R=brettw@chromium.org, phajdan.jr@chromium.org, rdsmith@chromium.org, sky@chromium.org, yoz@chromium.org

Review URL: https://codereview.chromium.org/14273023
------------------------------------------------------------------------
Project Member

Comment 19 by bugdroid1@chromium.org, May 10 2013

------------------------------------------------------------------------
r199558 | erg@chromium.org | 2013-05-10T22:39:12.690596Z

Changed paths:
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/ui/ui_test.cc?r1=199558&r2=199557&pathrev=199558
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/ui/ui_test.h?r1=199558&r2=199557&pathrev=199558
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/Archived History-journal?r1=199558&r2=199557&pathrev=199558
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Cookies?r1=199558&r2=199557&pathrev=199558
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/Favicons-journal?r1=199558&r2=199557&pathrev=199558
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/startup_test.cc?r1=199558&r2=199557&pathrev=199558
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/PreferencesTemplate?r1=199558&r2=199557&pathrev=199558
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Full Text Index?r1=199558&r2=199557&pathrev=199558
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/Top Sites-journal?r1=199558&r2=199557&pathrev=199558
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/History Provider Cache?r1=199558&r2=199557&pathrev=199558
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/README.txt?r1=199558&r2=199557&pathrev=199558
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/history_unittest.cc?r1=199558&r2=199557&pathrev=199558
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/top_sites_likely_impl.cc?r1=199558&r2=199557&pathrev=199558
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/generate_profile.h?r1=199558&r2=199557&pathrev=199558
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/feature_startup_test.cc?r1=199558&r2=199557&pathrev=199558
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/History Provider Cache?r1=199558&r2=199557&pathrev=199558
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/History?r1=199558&r2=199557&pathrev=199558
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/top_sites.h?r1=199558&r2=199557&pathrev=199558
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/Visited Links?r1=199558&r2=199557&pathrev=199558
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/History-journal?r1=199558&r2=199557&pathrev=199558
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme?r1=199558&r2=199557&pathrev=199558
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/History Index 2013-02?r1=199558&r2=199557&pathrev=199558
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/history_types.h?r1=199558&r2=199557&pathrev=199558
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/perf_ui_test_suite.cc?r1=199558&r2=199557&pathrev=199558
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/top_sites_impl.cc?r1=199558&r2=199557&pathrev=199558
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/perf_ui_test_suite.h?r1=199558&r2=199557&pathrev=199558
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/History?r1=199558&r2=199557&pathrev=199558
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/top_sites_cache.cc?r1=199558&r2=199557&pathrev=199558
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/top_sites_impl.h?r1=199558&r2=199557&pathrev=199558
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/History Index 2013-02-journal?r1=199558&r2=199557&pathrev=199558
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/chrome_tests.gypi?r1=199558&r2=199557&pathrev=199558
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/generate_profile.cc?r1=199558&r2=199557&pathrev=199558
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Visited Links?r1=199558&r2=199557&pathrev=199558
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/History-journal?r1=199558&r2=199557&pathrev=199558
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/top_sites_likely_impl.h?r1=199558&r2=199557&pathrev=199558
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Extensions/mblmlcbknbnfebdfjnolmcapmdofhmme/1.1/Cached Theme.pak?r1=199558&r2=199557&pathrev=199558
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default?r1=199558&r2=199557&pathrev=199558
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/History Index 2013-01?r1=199558&r2=199557&pathrev=199558
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/tools/profiles/generate_profile.cc?r1=199558&r2=199557&pathrev=199558
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/Archived History?r1=199558&r2=199557&pathrev=199558
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/run_all_perfuitests.cc?r1=199558&r2=199557&pathrev=199558
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/Favicons?r1=199558&r2=199557&pathrev=199558
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/History Index 2013-01-journal?r1=199558&r2=199557&pathrev=199558
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/Top Sites?r1=199558&r2=199557&pathrev=199558

Revert "Rebuild test history databases when starting up performance_ui_tests."

This reverts commit 45a5daefb13453c704c364fa1cfc3bdb5774e9ac. There are weird
resource bundle related failures on Windows debug builds.

BUG= 111570 
------------------------------------------------------------------------
Project Member

Comment 20 by bugdroid1@chromium.org, May 14 2013

------------------------------------------------------------------------
r200094 | erg@chromium.org | 2013-05-14T22:58:49.006837Z

Changed paths:
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/History Index 2013-01?r1=200094&r2=200093&pathrev=200094
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/tools/profiles/generate_profile.cc?r1=200094&r2=200093&pathrev=200094
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/top_sites.h?r1=200094&r2=200093&pathrev=200094
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/History Index 2013-01-journal?r1=200094&r2=200093&pathrev=200094
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/run_all_perfuitests.cc?r1=200094&r2=200093&pathrev=200094
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme?r1=200094&r2=200093&pathrev=200094
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/ui/ui_test.cc?r1=200094&r2=200093&pathrev=200094
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/ui/ui_test.h?r1=200094&r2=200093&pathrev=200094
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Cookies?r1=200094&r2=200093&pathrev=200094
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/history_types.h?r1=200094&r2=200093&pathrev=200094
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/PreferencesTemplate?r1=200094&r2=200093&pathrev=200094
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/perf_ui_test_suite.cc?r1=200094&r2=200093&pathrev=200094
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/startup_test.cc?r1=200094&r2=200093&pathrev=200094
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/perf_ui_test_suite.h?r1=200094&r2=200093&pathrev=200094
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/top_sites_impl.cc?r1=200094&r2=200093&pathrev=200094
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Full Text Index?r1=200094&r2=200093&pathrev=200094
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/History Provider Cache?r1=200094&r2=200093&pathrev=200094
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/top_sites_cache.cc?r1=200094&r2=200093&pathrev=200094
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/top_sites_impl.h?r1=200094&r2=200093&pathrev=200094
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/History?r1=200094&r2=200093&pathrev=200094
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/chrome_tests.gypi?r1=200094&r2=200093&pathrev=200094
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/history_unittest.cc?r1=200094&r2=200093&pathrev=200094
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/generate_profile.cc?r1=200094&r2=200093&pathrev=200094
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/generate_profile.h?r1=200094&r2=200093&pathrev=200094
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/feature_startup_test.cc?r1=200094&r2=200093&pathrev=200094
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Visited Links?r1=200094&r2=200093&pathrev=200094
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/top_sites_likely_impl.cc?r1=200094&r2=200093&pathrev=200094
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/History-journal?r1=200094&r2=200093&pathrev=200094
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/top_sites_likely_impl.h?r1=200094&r2=200093&pathrev=200094
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Extensions/mblmlcbknbnfebdfjnolmcapmdofhmme/1.1/Cached Theme.pak?r1=200094&r2=200093&pathrev=200094

Rebuild test history databases when starting up performance_ui_tests.

[In this reland, manually load the browser dll as a resource on Windows.]

Previously, whenever the theme or history database format were modified, a
large binary database or pak file had to be committed. This was only done
intermittently, messing up some perf expectations. Since last June, this
has added over 100 megabytes to our repository weight.

By generating the SQLite databases and Cached Theme.pak files at startup
of the one test suite that uses them, we're guaranteed that we'll always
be using the current versions and won't add upgrade costs to our
performance_ui_tests. We'll also not continuously add to new binary
resources to the permanent git archive.

BUG= 111570 
TBR=sky@chromium.org

First Review: https://codereview.chromium.org/14273023

Review URL: https://codereview.chromium.org/14585015
------------------------------------------------------------------------
Project Member

Comment 21 by bugdroid1@chromium.org, May 15 2013

------------------------------------------------------------------------
r200312 | erg@chromium.org | 2013-05-15T17:52:17.371557Z

Changed paths:
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Visited Links?r1=200312&r2=200311&pathrev=200312
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/top_sites_likely_impl.cc?r1=200312&r2=200311&pathrev=200312
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/generate_profile.h?r1=200312&r2=200311&pathrev=200312
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/History-journal?r1=200312&r2=200311&pathrev=200312
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/top_sites_likely_impl.h?r1=200312&r2=200311&pathrev=200312
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default?r1=200312&r2=200311&pathrev=200312
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/History Provider Cache?r1=200312&r2=200311&pathrev=200312
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/History Index 2013-01?r1=200312&r2=200311&pathrev=200312
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/History Index 2013-01-journal?r1=200312&r2=200311&pathrev=200312
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/Top Sites?r1=200312&r2=200311&pathrev=200312
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/ui/ui_test.h?r1=200312&r2=200311&pathrev=200312
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Cookies?r1=200312&r2=200311&pathrev=200312
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/startup_test.cc?r1=200312&r2=200311&pathrev=200312
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/PreferencesTemplate?r1=200312&r2=200311&pathrev=200312
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/perf_ui_test_suite.h?r1=200312&r2=200311&pathrev=200312
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Full Text Index?r1=200312&r2=200311&pathrev=200312
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/Top Sites-journal?r1=200312&r2=200311&pathrev=200312
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/top_sites_impl.h?r1=200312&r2=200311&pathrev=200312
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/top_sites_cache.cc?r1=200312&r2=200311&pathrev=200312
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/history_unittest.cc?r1=200312&r2=200311&pathrev=200312
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/feature_startup_test.cc?r1=200312&r2=200311&pathrev=200312
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Extensions/mblmlcbknbnfebdfjnolmcapmdofhmme/1.1/Cached Theme.pak?r1=200312&r2=200311&pathrev=200312
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/tools/profiles/generate_profile.cc?r1=200312&r2=200311&pathrev=200312
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/History?r1=200312&r2=200311&pathrev=200312
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/top_sites.h?r1=200312&r2=200311&pathrev=200312
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/Archived History?r1=200312&r2=200311&pathrev=200312
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/Favicons?r1=200312&r2=200311&pathrev=200312
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/Visited Links?r1=200312&r2=200311&pathrev=200312
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/run_all_perfuitests.cc?r1=200312&r2=200311&pathrev=200312
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/History-journal?r1=200312&r2=200311&pathrev=200312
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme?r1=200312&r2=200311&pathrev=200312
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/ui/ui_test.cc?r1=200312&r2=200311&pathrev=200312
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/History Index 2013-02?r1=200312&r2=200311&pathrev=200312
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/history_types.h?r1=200312&r2=200311&pathrev=200312
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/Archived History-journal?r1=200312&r2=200311&pathrev=200312
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/Favicons-journal?r1=200312&r2=200311&pathrev=200312
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/perf_ui_test_suite.cc?r1=200312&r2=200311&pathrev=200312
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/top_sites_impl.cc?r1=200312&r2=200311&pathrev=200312
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/History Provider Cache?r1=200312&r2=200311&pathrev=200312
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/History?r1=200312&r2=200311&pathrev=200312
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/Default/History Index 2013-02-journal?r1=200312&r2=200311&pathrev=200312
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/chrome_tests.gypi?r1=200312&r2=200311&pathrev=200312
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme/README.txt?r1=200312&r2=200311&pathrev=200312
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/generate_profile.cc?r1=200312&r2=200311&pathrev=200312

Revert "Rebuild test history databases when starting up performance_ui_tests."

This reverts commit fd99334e881859262199c9b8a98f9fbb003b20e1.

BUG= 111570 
------------------------------------------------------------------------
Project Member

Comment 22 by bugdroid1@chromium.org, May 15 2013

------------------------------------------------------------------------
r200330 | erg@chromium.org | 2013-05-15T19:11:28.007605Z

Changed paths:
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/top_sites.h?r1=200330&r2=200329&pathrev=200330
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/History Index 2013-01-journal?r1=200330&r2=200329&pathrev=200330
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/run_all_perfuitests.cc?r1=200330&r2=200329&pathrev=200330
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/ui/ui_test.cc?r1=200330&r2=200329&pathrev=200330
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_default_theme?r1=200330&r2=200329&pathrev=200330
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/ui/ui_test.h?r1=200330&r2=200329&pathrev=200330
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Cookies?r1=200330&r2=200329&pathrev=200330
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/history_types.h?r1=200330&r2=200329&pathrev=200330
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/PreferencesTemplate?r1=200330&r2=200329&pathrev=200330
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/perf_ui_test_suite.cc?r1=200330&r2=200329&pathrev=200330
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/startup_test.cc?r1=200330&r2=200329&pathrev=200330
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Full Text Index?r1=200330&r2=200329&pathrev=200330
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/top_sites_impl.cc?r1=200330&r2=200329&pathrev=200330
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/perf_ui_test_suite.h?r1=200330&r2=200329&pathrev=200330
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/History Provider Cache?r1=200330&r2=200329&pathrev=200330
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/top_sites_cache.cc?r1=200330&r2=200329&pathrev=200330
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/top_sites_impl.h?r1=200330&r2=200329&pathrev=200330
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/History?r1=200330&r2=200329&pathrev=200330
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/chrome_tests.gypi?r1=200330&r2=200329&pathrev=200330
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/history_unittest.cc?r1=200330&r2=200329&pathrev=200330
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/generate_profile.cc?r1=200330&r2=200329&pathrev=200330
   A http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/generate_profile.h?r1=200330&r2=200329&pathrev=200330
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/feature_startup_test.cc?r1=200330&r2=200329&pathrev=200330
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Visited Links?r1=200330&r2=200329&pathrev=200330
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/top_sites_likely_impl.cc?r1=200330&r2=200329&pathrev=200330
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/History-journal?r1=200330&r2=200329&pathrev=200330
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/top_sites_likely_impl.h?r1=200330&r2=200329&pathrev=200330
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/Extensions/mblmlcbknbnfebdfjnolmcapmdofhmme/1.1/Cached Theme.pak?r1=200330&r2=200329&pathrev=200330
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/profiles/profile_with_complex_theme/Default/History Index 2013-01?r1=200330&r2=200329&pathrev=200330
   D http://src.chromium.org/viewvc/chrome/trunk/src/chrome/tools/profiles/generate_profile.cc?r1=200330&r2=200329&pathrev=200330

Revert "Revert "Rebuild test history databases when starting up performance_ui_tests.""

This reverts the revert and puts the performance_ui_tests fix in the tree. It
will still break on the Windows XP dbg 2 performance bot, but this is for the
infrastructure people to see what's going on there.

BUG= 111570 
------------------------------------------------------------------------
Project Member

Comment 23 by bugdroid1@chromium.org, May 15 2013

------------------------------------------------------------------------
r200351 | erg@chromium.org | 2013-05-15T21:18:57.952247Z

Changed paths:
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/perf_ui_test_suite.cc?r1=200351&r2=200350&pathrev=200351

Add debugging statements for problem that only happens on bots.

The Windows XP (2) dbg perf bot is not printing any output. Everything else
works, including other debug versions of Windows and Windows XP
non-debug. Committing this on the recommendation of the current troopers.

BUG= 111570 
TBR=agable@chromium.org

Review URL: https://codereview.chromium.org/15037005
------------------------------------------------------------------------

Comment 24 by e...@chromium.org, May 16 2013

Blockedon: chromium:241276
Project Member

Comment 25 by bugdroid1@chromium.org, May 16 2013

------------------------------------------------------------------------
r200620 | erg@chromium.org | 2013-05-16T20:33:11.833367Z

Changed paths:
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/perf/perf_ui_test_suite.cc?r1=200620&r2=200619&pathrev=200620

Revert "Add debugging statements for problem that only happens on bots."

This removes the stderr debugging statements that suggested that this is
really a bot problem.

BUG= 111570 
TBR=agable@chromium.org

Review URL: https://codereview.chromium.org/15044012
------------------------------------------------------------------------
Labels: -Build-Infrastructure Infra

Comment 27 by e...@chromium.org, Jul 11 2013

Cc: iannucci@chromium.org
This is the most recent version of my script for pruning the repository after pulling out the test profiles from history.

iannucci, agable told me that you're also working on a similar pruning system. I've been working on a similar system for a year now. I assume whatever system is run in the end should prune the union of our two systems.
git-prunechrome.sh
2.8 KB View Download
Labels: TaskForce-GreenTree

Comment 29 by e...@chromium.org, Aug 15 2013

Here's a new approach: Using BFG Repo Cleaner (http://rtyley.github.io/bfg-repo-cleaner/)

I was able to shrink the current repository down to ~500 megabytes in about 10 minutes when using this tool. My hand crafted scripts clock in at 615 megabytes in 3 days time for comparison.

There are some trade offs:

- On the plus side, BFG prunes the history of any file matching the list of suffixes. So while my scripts only cleaned paths where files used to be (like chrome/theme/app/*.png, where there are no images any more), BFG keeps any png files in HEAD and then drops any such file which isn't. This makes pruning the history of any binary file types that we want gone both safer and more thorough.
- On the down side, its directory matching algorithm only works on a single directory name and not on full repository paths. This is part of what makes it fast.
- As a mostly good thing, deleting the past history of every directory named 'data' is probably safe, and the pruning benefits will scale as more and more of the old pyauto data gets deleted from chrome/test/data/. I'm slightly worried about over-pruning the history of directories named 'data', but AFAICT, they're all test data (mostly binary and large, too).

I've attached a report of the files in the BFG pruned repository, to kind of guide where to go from here. Note that the largest blob is a 16 meg CRX in chrome/test/data/. The next biggest binary is chrome/test/data/import/firefox/linux.zip, weighing in at 12 megs. If you look at the first couple of pages, it's mostly dominated by large, binary test data, all of which exist in ToT.

Finally, here's the command line I ran with:

java -jar ~/Downloads/bfg-1.10.0.jar --delete-files "*.{la,a,52,50,crx,xib,png,pdf,jpg,zip,ico,icns,jar,dmg,pbxproj,pak,gif,pcm,inc,raw,pdb,psd,pkg,jpeg,dylib,dll,DLL,exe,EXE,vcproj,ncb,so,sln,scons,nib,graffle,yuv,rgb,ogg,mp4,webm,wav}" --delete-folders "{apple_webkit,bzip2,cygwin,cygwin_src,ceee,gnu,hyphen,o3d,findbugs,gles2_book,gles_book_examples,glew,gears,gnu,hunspell,boost,python_24,lighttpd,libunwind,layout_test_results,scons,svn,initial,old,filter,profile_with_complex_theme,profile_with_default_theme,complex_theme,gtk_theme,custom_frame,custom_frame_gtk_theme,complex_theme,typical_history,layout_tests,MesaLib,mozilla,buildbot,chrome_linux,chrome_mac,data,buildbot}" . 
bigtosmall.txt
5.7 MB View Download

Comment 30 by c...@chromium.org, Aug 15 2013

Blockedon: chromium:263848
Marking blocked on the "switch Chrome to Git" bug.
Blocking: chromium:263848
Blockedon: -chromium:263848
Labels: Infra-Chrome
Labels: -Pri-2 Pri-3
Let's wait until after Chrome-to-Git flag day to address.

Comment 35 by e...@chromium.org, Feb 6 2014

Are your sure? AIUI, when everything switches to git, everyone will have to pull down a new repository anyway, so shouldn't we lump together events where developers will have to make new checkouts?

Comment 36 by szager@google.com, Feb 6 2014

Nope, there will be no new repository.  We will continue with src.git as currently constituted.

Probably best to do the history-rewrite sooner rather than later in general, to minimise the number of people affected. Presumably flag day will be when /everyone/ has to stop using SVN, and start using Git, so it would be good to fix the repo before those new users perform their initial clones.

The process described by @erg in #29 takes only 10 minutes, so it would make sense to try to incorporate it into the flag day plan.

Comment 38 by c...@chromium.org, Feb 6 2014

Hi Roberto, what that view doesn't take into account is that switching Chrome to Git is already very complicated.  I will actively remove any blockers from this project that are absolutely not needed to complete the overall task in order to simplify the project.  That increases the chances for it to complete at all sooner.

I had earlier considered the cost of deferring this rewrite until after the switch.  The disruption could be that those affected will need to fetch ~200-300MB of repo data (the new history) and rebase their branches on top of the new history.  I can't see how that is impossibly disruptive to everyone since fetching and rebasing are a core part of the main workflow already.

I tagged this topic for the team to discuss in person.  We'll report back here after that.

Comment 39 by c...@chromium.org, Apr 15 2014

Blocking: chromium:363658

Comment 40 by c...@chromium.org, Apr 15 2014

Blocking: -chromium:263848
If the progress of switch to git finished, then we have no chance to filter the chance anymore( cause we will use git as a official repository but not a mirror). About filter history, I have two require ment

1.remove historical binary/big data. We do not need to make sure the history to be rebuidable, if we want to do that,  we can use read only svn. We just need to make sursure meaningful changes to be presevrd.
2.filter the git's author/email pair confirm the git standard.

Comment 42 by mmoss@chromium.org, May 22 2015

Status: WontFix
I'm pretty sure this ship has sailed, and shallow clones are generally the preferred "fix" for when people don't want to fetch many GBs of repo data.

Sign in to add a comment