Commit Graph

287 Commits

Author SHA1 Message Date
Elijah Newren
301aea9993 filter-repo: pre-compile all regexes
Repeatedly using non-compiled regexes is rather wasteful of resources.
Pre-compile these and use the cached versions.

I ran
   git filter-repo --invert-paths --path configure.ac --dry-run
and then for timing ran

   cat .git/filter-repo/fast-export.original | time git filter-repo \
     --invert-paths --path configure.ac --dry-run --stdin

on the git.git repository (with tags of blobs and tags of tags deleted).
Comparing the timings before and after this change, I see about a 13%
overall speedup just from caching the regexes.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:37 -07:00
Elijah Newren
cbacb6cd82 filter-repo: simplify import in lib-usage examples
Python wants filenames with underscores instead of hyphens and with a
.py extension.  We really want the main file named git-filter-repo, but
we can add a git_filter_repo.py symlink.  Doing so dramatically
simplifies the steps needed to import it as a library in external python
scripts.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-26 07:56:03 -07:00
Elijah Newren
6dba1f200c filter-repo: avoid string->datetime->string round trips
Most filtering operations are not interested in the time that commits
were authored or committed, or when tags were tagged.  As such,
translating the string representation of the date into a datetime object
is wasted effort, and causes us to waste more time later as we have to
translate it back into a string.

Instead, provide string_to_date() and date_to_string() functions so that
callers can perform the translation if wanted, and let the normal case
be fast.

Provides a small but noticable speedup when just filtering based on
paths; about a 3.5% improvement in execution time for writing the new
history.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-26 07:56:03 -07:00
Elijah Newren
b363a1574f filter-repo: notify user how long filtering took
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-26 07:56:03 -07:00
Elijah Newren
c75492ca07 filter-repo: get new names of commits asynchronously
We have to ask fast-import for the new names of commits, but doing so
immediately upon dumping out the commit related information requires
context switches and waiting for fast-import to parse and handle more
information.  We don't need to know the new name of the commit until we
run across a subsequent commit that referenced it in the commit message
by its old ID.

So, speed things up dramatically by waiting until we need the new name
of the commit message (or the fast-import output pipe we are
communicating with should be getting kind of full) before blocking on
reading new commit hashes.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-26 07:56:03 -07:00
Elijah Newren
7698b33a05 filter-repo: small refactoring for fast_import_pipes
Treat fast_import_pipes more like the other parameters to
FastExportFilter.run(), both for consistency, and because it will allow
us to more easily defer doing blocking reads for new commit names until
we actually need to know the new commit hashes corresponding to old
commit ids.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-26 07:56:03 -07:00
Elijah Newren
2ba1131753 filter-repo: disconnect from origin upon invocation
Once we rewrite history, our history will be unrelated to our original
upstream.  As such, migrate refs/remotes/origin/* to refs/heads/*
(our sanity check already verified that if both names exist they are
equal; if the user used --force then just delete the remote tracking
branch and leave the local branch as is), and then delete the 'origin'
remote.

This has a few advantages:
  * People expect to work with refs/heads/*, not refs/remotes/origin/*,
    and will be more likely to write filters based on those.
  * People will probably need to push the new history somewhere when
    they are done, and it's easier if we have it in refs/heads/* than
    if it's under refs/remotes/origin/*.
  * It encourages people to use good hygiene and not mix old and new
    histories.  If users really want, they can push the repo (or parts
    thereof) back over their original history, but they should have to
    take extra steps to do it instead of just a 'git push'.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-26 07:56:03 -07:00
Elijah Newren
5ac24698fb filter-repo: check that git version is new enough to work
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-26 07:56:03 -07:00
Elijah Newren
a7110de5f5 filter-repo: make tagger optional
Apparently, despite the git-fast-import.txt documentation, tagger is
optional for both fast-export and fast-import.  I suspect this is
because there are several (old) tags in the linux.git repository that
have no tagger, so if we want to test rewriting linux.git history then
we need to make it optional for filter-repo too.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-26 07:56:03 -07:00
Elijah Newren
911f234e3d filter-repo: update README.md
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-26 07:56:03 -07:00
Elijah Newren
73e91edecc filter-repo: add text removal (or replacement) via file of expressions
Make it easy for users to search and replace text throughout the
repository history.  Instead of inventing some new syntax, reuse the
same syntax used by BFG repo filter's --replace-text option, namely,
a file with one expression per line of the form

    [regex:|glob:|literal:]$MATCH_EXPR[==>$REPLACEMENT_EXPR]

Where "$MATCH_EXPR" is by default considered to be literal text, but
could be a regex or a glob if the appropriate prefix is used.  Also,
$REPLACEMENT_EXPR defaults to '***REMOVED***' if not specified.  If
you want a literal '==>' to be part of your $MATCH_EXPR, then you
must also manually specify a replacement expression instead of taking
the default.  Some examples:

    sup3rs3kr3t
    (replaces 'sup3rs3kr3t' with '***REMOVED***')

    HeWhoShallNotBeNamed==>Voldemort
    (replaces 'HeWhoShallNotBeNamed' with 'Voldemort')

    very==>
    (replaces 'very' with the empty string)

    regex:(\d{2})/(\d{2})/(\d{4})==>\2/\1/\3
    (replaces '05/17/2012' with '17/05/2012', and vice-versa)

    The format for regex is as from
    re.sub(<pattern>, <repl>, <string>) from
    https://docs.python.org/2/library/re.html
    The <string> comes from file contents of the repo, and you specify
    the <pattern> and <repl>.

    glob:Copy*t==>Cartel
    (replaces 'Copyright' or 'Copyleft' or 'Copy my st' with 'Cartel')

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-26 07:56:03 -07:00
Elijah Newren
4ee915e4dd filter-repo: fix fast-import crash with renaming
When non-merge commits have files in the _files_tweaked set (they were
modified by a blob or commit callback), they may become empty.  However,
new_1st_parent is more accurately named
new_1st_parent_if_would_become_non_merge; it will always be None for
non-merge commits.  So we need to get the correct parent.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-26 07:56:03 -07:00
Elijah Newren
dd438dc455 filter-repo: add mailmap handling
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-26 07:56:03 -07:00
Elijah Newren
a5d4d70876 filter-repo: add some testcases making use of filter-repo as a library
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-26 07:56:03 -07:00
Elijah Newren
d615d71411 filter-repo: simplify FastExportFilter and nuke unused code
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-26 07:56:03 -07:00
Elijah Newren
5ea234153c filter-repo: add RepoFilter.finish() function for code readability
When we only have an output and no input of our own, filter.run() seems
weird to call, especially since it'll only be closing a handle and waiting
for fast-import to finish.  Add a finish() synonym for such a case to make
external code callers more legible.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-26 07:56:03 -07:00
Elijah Newren
72b69b3dbe filter-repo: support --source and --target options
This will allow exporting from one repo into a different repo, and
combined with chained RepoFilter instances from commit 81016821a1
(filter-repo: allow chaining of RepoFilter instances, 2019-01-07), will
even allow things like splicing separate repositories together.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-26 07:55:39 -07:00
Elijah Newren
d0640bad7a filter-repo: perform sanity checks before setting up output processes
We do not want to kill fast-import processes unused; it's better
to abort before those processes are created when we know we need to.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-26 07:55:13 -07:00
Elijah Newren
81016821a1 filter-repo: allow chaining of RepoFilter instances
Allow each instance to be just input or just output so that we can splice
repos together or split one into multiple different repos.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-26 07:55:13 -07:00
Elijah Newren
59f3947857 filter-repo: allow importing into a bare repository
If we are using --stdin, it should be okay to import into a bare repo,
but the checks were enforcing that we were in a clone with a packfile.
Relax the check to work within a bare repo as well.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-26 07:55:13 -07:00
Elijah Newren
6fffed6bb1 filter-repo: handle blob callbacks without excessive empty-pruning checks
If we have blob callbacks, we cannot pass --no-data to fast-export.  Also,
with blob callbacks, any file the callback modifies could match the
modification done to the file by a subsequent commit, possibly making the
later commit empty.  As such, we keep a record of all filenames modified
(by blob or commit callbacks), and then check all these filenames for all
subsequent commits to see if it causes empty commits.  In particular, if
files other than these are modified in a non-merge commit, we know that
the commit will not become empty so we can bypass the empty-pruning
checks.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-26 07:55:13 -07:00
Elijah Newren
dbdb18170b filter-repo: perf hack -- avoid expensive empty pruning checks
If a commit was a non-merge commit previously, then since we do not do
any kind of blob modifications (or funny parent grafting), there is no
way for a filemodify instruction to introduce the same version of the
file that already existed in the parent, as such the only check we need
to do to determine whether a commit becomes empty is whether
file_changes is empty.  Subsequent more expensive checks can be skipped.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-26 07:55:13 -07:00
Elijah Newren
d0037275af filter-repo: allow RepoFilter.run to be passed callbacks
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-26 07:55:13 -07:00
Elijah Newren
55c2c32d7c filter-repo: group high-level repo filtering functions into a class
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-26 07:55:13 -07:00
Elijah Newren
4e2110136e filter-repo: group repo analysis functions into a class
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-26 07:55:13 -07:00
Elijah Newren
9887dd5cbe filter-repo: move sanity_check to put analyze functions before filtering ones
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-03-12 14:19:46 -07:00
Elijah Newren
fc90cf8ca9 filter-repo: collect various short functions into a GitUtils helper class
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-03-12 14:19:46 -07:00
Elijah Newren
2f3a445875 filter-repo: restructure argument parsing for re-use
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-03-12 14:19:46 -07:00
Elijah Newren
9bb4188e83 filter-repo: perf hack -- do minimal amount of quoting required by fast-import
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-03-12 14:19:46 -07:00
Elijah Newren
da5895ecc3 filter-repo: restructure empty pruning
Split a lot of the logic out into separate functions, and avoid
flattening parents when the original commit history itself had
redundant parents (such as --no-ff merges).

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-03-12 14:19:46 -07:00
Elijah Newren
1c3bc2fa1e filter-repo: track skipped/pruned commits
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-03-12 14:19:46 -07:00
Elijah Newren
70e6f848ed filter-repo: modify parse_optional_parent_ref to return original parent too
commits may not have any parents at all.  As such,
parse_optional_parent_ref() is used expecting that it will sometimes
return None.

Now, when commits are skipped, we have a scheme to translate anyone that
depends on such commits to instead depend on the nearest ancestor of
such commits.  If the entire ancestry of a commit was skipped along with
a comit, then that commit will be translated to None, which is
indistinguishable from there having been no parent to begin with.
Sometimes our scheme needs to distinguish between a commit that started
with no parents and one which ended up with no parents, so we need a way
to tell these apart.

Also, not knowing the original parent makes it hard for us to
determine if the original had the same weird topology that the current
commit does.  For example, it is possible for a merge commit to have
one parent be the ancestor of another (particularly when --no-ff is
passed to git merge), or even for a merge commit to have the same
commit used as both parents (if you use low-level commands to create
a crazy commit).  There are cases where the pruning of some commits
could cause either of these situations to arise, and it's useful to be
able to distinguish between intentionally "weird" history and history
that has been made weird due to other pruning, because the latter we
may have reason to do additional pruning on.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-03-12 14:19:46 -07:00
Elijah Newren
ab1b43f480 filter-repo: add a couple minor clarifications
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-03-12 14:19:46 -07:00
Elijah Newren
69147fe120 filter-repo: fix crazy timezone issues
Oh, boy, timezone +051800 exists in the wild.  Is that 0518 hours and 00
minutes?  Or 05 hours and 1800 minutes?  Or 051 hours and 800 minutes?
Attempt to do something sane with these broken commits that fast-import
barfs on.  Also, fix an old bug in the handling of ahead-of-UTC timezones.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-03-12 14:19:46 -07:00
Elijah Newren
03507e57f5 filter-repo: buffer subprocess stdout to significantly improve performance
Apparently, the default for subprocess stdout is unbuffered; switching
it to buffered yields a huge 40% speedup.  Doing this also exposes the
need to add fi_input.flush() calls, highlighting another performance
issue.  We may be able to have fewer such calls with some refactoring,
but that is a bigger separate change.  Just having them highlighted to
remind about them as a performance issue is good for now.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-03-12 14:19:46 -07:00
Elijah Newren
9ebd3117ca filter-repo: notify user when we start writing reports
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-03-12 14:19:46 -07:00
Elijah Newren
554c7e39af filter-repo: switch --analyze to use rev-list|diff-tree pipeline
As suggested by Peff, use rev-list & diff-tree to get the information we
need, instead of relying on fast-export (with some out-of-tree patches)
to get that information.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-03-12 14:19:46 -07:00
Elijah Newren
beff0b958f filter-repo: be more thorough about path quoting, and handle non-ascii
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-03-12 14:19:46 -07:00
Elijah Newren
becc29a9bd filter-repo: show progress parsing blob sizes
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-03-12 14:19:46 -07:00
Elijah Newren
aa7eebbc88 filter-repo: add ProgressWriter class and switch FastExportFilter to it
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-03-12 14:19:46 -07:00
Elijah Newren
a2540f4087 filter-repo: add packed sizes to --analyze reports
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-03-12 14:19:46 -07:00
Elijah Newren
7048be2849 filter-repo: split analysis reports into separate files
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-03-12 14:19:46 -07:00
Elijah Newren
37c92d9352 filter-repo: handle tags pointing at commits pruned along with their history
If a tag points at a commit whose changes are all filtered out and thus
becomes empty and gets pruned, and all of its ancestors are likewise
pruned, then there is no need for the tag; just nuke it.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-03-12 14:19:46 -07:00
Elijah Newren
77d5e93135 filter-repo: add some preventative sanity checks
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-03-12 14:19:46 -07:00
Elijah Newren
9b88f3f094 filter-repo: ensure we parse all merge parents, even if some became pruned
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-03-12 14:19:46 -07:00
Elijah Newren
424faa3103 filter-repo: add optional newline to make --dry-run output easier to parse
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-03-12 14:19:46 -07:00
Elijah Newren
c8a96d4684 filter-repo: add --subdirectory-filter and --to-subdirectory-filter
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-03-12 14:19:46 -07:00
Elijah Newren
e36a62c2c7 filter-repo: add tag renaming
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-03-12 14:19:46 -07:00
Elijah Newren
f813469ff8 filter-repo: start revamping the --help page
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-03-12 14:19:46 -07:00
Elijah Newren
5f55960dfe filter-repo: add README.md explaing new filter-repo tool
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-03-12 14:19:46 -07:00