Commit Graph

40 Commits

Author SHA1 Message Date
Elijah Newren
35052f673d filter-repo (python3): replace strings with bytestrings
This is by far the largest python3 change; it consists basically of
  * using b'<str>' instead of '<str>' in lots of places
  * adding a .encode() if we really do work with a string but need to
    get it converted to a bytestring
  * replace uses of .format() with interpolation via the '%' operator,
    since bytestrings don't have a .format() method.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
4c05cbe072 filter-repo (python3): bytes() instead of chr() or string join
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
c3072c7f01 filter-repo (python3): convert StringIO->BytesIO and __str__->__bytes__
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
9b3134b68c filter-repo (python3): ensure file reads and writes are done in bytes
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
8b8d6b4b43 filter-repo (python3): ensure stdin and args are bytes instead of strings
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
effcd5b9ff filter-repo (python3): convert run_coverage
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
e5955f397f filter-repo (python3): shebang and imports
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
805ce360fa filter-repo: simplify API for parent handling in Commit object
While the underlying fast-export and fast-import streams explicitly
separate 'from' commit (first parent) and 'merge' commits (all other
parents), foisting that separation into the Commit object for
filter-repo forces additional places in the code to deal with that
distinction.  It results in less clear code, and especially does not
make sense to push upon folks who may want to use filter-repo as a
library.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
bec6bd8d3c filter-repo: add testcase with funny characters
Use UTF-8 chars in user names, filenames, branch names, tag names, and
file contents.  Also include invalid UTF-8 in file contents; should be
able to handle binary data.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
0ca3988953 filter-repo: specify sorting order in greater detail
The sorting order of entries written to files in the analysis directory
didn't specify a secondary sort, thus making the order dependent on the
random-ish sorting order of dictionaries and making it inconsistent
between python versions.  While the secondary order didn't matter much,
having a defined order makes it slightly easier to define a single
testcase that can work across versions.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
4cb3bc3459 filter-repo: mark messages for translation
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
7bc87c7f0b filter-repo: enable internationalization
Assuming filter-repo will be merged into git.git, use "git" for the
TEXTDOMAIN, and assume its build system will replace "@@LOCALEDIR@@"
appropriately.

Note that the xgettext command used to grab string translations is
nearly identical to the one for C files in git.git; just use
--language=python instead and add --join-existing to avoid overwriting
the po/git.pot file.  In other words, use the command:
    xgettext -o../git/po/git.pot --join-existing --force-po \
        --add-comments=TRANSLATORS: \
        --msgid-bugs-address="Git Mailing List <git@vger.kernel.org>" \
        --from-code=UTF-8 --language=python \
        --keyword=_ --keyword=N_ --keyword="Q_:1,2" \
        git-filter-repo

To create or update the translation, go to git.git/po and run either of:
    msginit --locale=XX
    msgmerge --add-location --backup=off -U XX.po git.pot

Once you've updated the translation, within git.git just build as
normal.  That's all that's needed.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
365afadfd2 filter-repo: add a test just to get line coverage of an unusual error
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
cd9ea5af9b filter-repo: fix incremental imports and add a test
The AncestryGraph setup assumed we had previously seen all commits which
would be used as parents; that interacted badly with doing an
incremental import.  Add a function which can be used to record external
commits, each of which we'll treat like a root commit (i.e. depth 1 and
having no parents of its own).  Add a test to prevent regressions.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
69f7224d95 filter-repo: add tests for invalid fast-import directives
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
3e4e0ed2ed filter-repo: add tests for other startup errors and help option
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
30228bdde2 filter-repo: add tests triggering callback sanity checks
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
dd4f3bd111 filter-repo: add tests checking error paths in mailmap parsing
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
8a1358cc30 filter-repo: add tests checking whether we start in a clean clone
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
e913ccbe8d filter-repo: add coverage for some corner cases and unusual constructs
There are a number of things not present in "normal" imports that we
nevertheless support and need to be tested:
  * broken timezone adjustment (+051800->+0261; observed in the wild
    in real repos, and adjustment prevents fast-import from dying)
  * commits missing an author (observed in the wild in a real repo;
    just sets author to committer)
  * optional additional linefeeds in the input allowed by
    git-fast-import but usually not written by git-fast-export
  * progress and checkpoint objects
  * progress, checkpoint, and 'everything' callbacks

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
8a8a15b998 filter-repo: add tests for --debug and --dry-run
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
0e4d48158f filter-repo: add a script to generate a line coverage report
This also generates line coverage statistics for t/t9391/*.py, but the
point is line coverage of git-filter-repo.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
7558cb2198 filter-repo: add more thorough test of --replace-text
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
8960eaeb8c filter-repo: add testcases for commit message rewriting
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
e19682bc19 filter-repo: add a fairly thorough test for --analyze
The test does check the exact output for the report, meaning if the
output is changed at all this test will need to be updated, but it at
least makes sure we are getting all the right kinds of information.  I
do not expect the output format will change very often.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
a8111d3521 filter-repo: add tests using refs/replace/
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
2b2ab0422e filter-repo: add testcases for --[to-]subdirectory-filter
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
842b6c3811 filter-repo: add --tag-rename testcase
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
5ef6aeba16 filter-repo: add various --path-rename tests
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
ef4b96e7be filter-repo: add API backward compatibility warning
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
5ba62ba4e8 filter-repo: add testcases dealing with topology changes
Pruning of commits which become empty can result in a variety of
topology changes: a merge may have lost all its ancestors corresponding
to one of (or more) of its parents, a merge may end up merging a commit
with itself, or a merge may end up merging a commit with its own
ancestor.  Merging a commit with itself makes no sense, so we'd rather
prune down to one parent and hopefully prune the merge commit, but we do
need to worry about whether the are changes in the commit and whether
the original merge commit also merged something with itself.  We have
similar cases for dealing with a merge of some commit with its own
ancestor: if the original topology did the same, or the merge commit has
additional file changes, then we cannot remove the commit.  But,
otherwise, the commit can be pruned.

Add testcases covering the variety of changes that can occur to make
sure we get them right.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
49732e8b5f filter-repo: add testcases dealing with commit pruning
There are several cases to worry about with commit pruning; commits
that start empty and had no parent, commits that start empty and
had a parent which may or may not get pruned, commits which had
changes but became empty, commits which were merges but lost a line
of ancestry and have no changes of their own, etc.  Add testcases
covering these cases, though most topology related ones will be
deferred to a later set of tests.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:37 -07:00
Elijah Newren
4635102d0f filter-repo: add more path-related testcases
Add some testcases for multiple --path arguments, for --path-glob, and
for --path-regex.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:37 -07:00
Elijah Newren
2431904f26 filter-repo: add command line parameters for passing body of callbacks
Many of the callback functions might only be a single line, and as such
instead of forcing the user to write a full blown program with an import
and everything, let them just specify the body of the callback function
as a command line parameter.  Add several tests of this functionality as
well.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:37 -07:00
Elijah Newren
cbacb6cd82 filter-repo: simplify import in lib-usage examples
Python wants filenames with underscores instead of hyphens and with a
.py extension.  We really want the main file named git-filter-repo, but
we can add a git_filter_repo.py symlink.  Doing so dramatically
simplifies the steps needed to import it as a library in external python
scripts.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-26 07:56:03 -07:00
Elijah Newren
6dba1f200c filter-repo: avoid string->datetime->string round trips
Most filtering operations are not interested in the time that commits
were authored or committed, or when tags were tagged.  As such,
translating the string representation of the date into a datetime object
is wasted effort, and causes us to waste more time later as we have to
translate it back into a string.

Instead, provide string_to_date() and date_to_string() functions so that
callers can perform the translation if wanted, and let the normal case
be fast.

Provides a small but noticable speedup when just filtering based on
paths; about a 3.5% improvement in execution time for writing the new
history.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-26 07:56:03 -07:00
Elijah Newren
73e91edecc filter-repo: add text removal (or replacement) via file of expressions
Make it easy for users to search and replace text throughout the
repository history.  Instead of inventing some new syntax, reuse the
same syntax used by BFG repo filter's --replace-text option, namely,
a file with one expression per line of the form

    [regex:|glob:|literal:]$MATCH_EXPR[==>$REPLACEMENT_EXPR]

Where "$MATCH_EXPR" is by default considered to be literal text, but
could be a regex or a glob if the appropriate prefix is used.  Also,
$REPLACEMENT_EXPR defaults to '***REMOVED***' if not specified.  If
you want a literal '==>' to be part of your $MATCH_EXPR, then you
must also manually specify a replacement expression instead of taking
the default.  Some examples:

    sup3rs3kr3t
    (replaces 'sup3rs3kr3t' with '***REMOVED***')

    HeWhoShallNotBeNamed==>Voldemort
    (replaces 'HeWhoShallNotBeNamed' with 'Voldemort')

    very==>
    (replaces 'very' with the empty string)

    regex:(\d{2})/(\d{2})/(\d{4})==>\2/\1/\3
    (replaces '05/17/2012' with '17/05/2012', and vice-versa)

    The format for regex is as from
    re.sub(<pattern>, <repl>, <string>) from
    https://docs.python.org/2/library/re.html
    The <string> comes from file contents of the repo, and you specify
    the <pattern> and <repl>.

    glob:Copy*t==>Cartel
    (replaces 'Copyright' or 'Copyleft' or 'Copy my st' with 'Cartel')

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-26 07:56:03 -07:00
Elijah Newren
dd438dc455 filter-repo: add mailmap handling
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-26 07:56:03 -07:00
Elijah Newren
a5d4d70876 filter-repo: add some testcases making use of filter-repo as a library
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-26 07:56:03 -07:00
Elijah Newren
17a2f7102d filter-repo: add some basic tests, with git-style test-lib.sh
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-03-12 14:19:38 -07:00