Commit Graph

212 Commits

Author SHA1 Message Date
Elijah Newren
ca5818056d filter-repo (python3): oct strings in python3 use "0o" instead of "0"
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
c3072c7f01 filter-repo (python3): convert StringIO->BytesIO and __str__->__bytes__
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
0279e3882d filter-repo (python3): error messages should be strings instead of bytes
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
9b3134b68c filter-repo (python3): ensure file reads and writes are done in bytes
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
8b8d6b4b43 filter-repo (python3): ensure stdin and args are bytes instead of strings
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
effcd5b9ff filter-repo (python3): convert run_coverage
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
6e78788feb filter-repo (python3): more flush()ing needed under python3
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
ad3c839263 filter-repo (python3): handle conversion of glob to regex
python3 forces a couple issues for us with the conversion of globs to
regexes:
  * fnmatch.translate() will ONLY operate on unicode strings, not
    bytestrings.  Super lame.
  * newer versions of python3 modified the regex style used by
    fnmatch.translate() causing us to need extra logic to 'fixup'
    the regex into the form we want.
Split the code for translating the glob to a regex out into a separate
function which now houses more complicated logic to handle these extra
conditions.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
1a8e247ba7 filter-repo (python3): add a decode() function
We need a function to transform byte strings into unicode strings for
printing error messages and occasional other uses.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
2562f0270c filter-repo (python3): revert "workaround python<2.7.9 exec bug"
Commit ca32c5d9afe2 ("filter-repo: workaround python<2.7.9 exec bug",
2019-04-30) put in a workaround for python versions prior to 2.7.9, but
which was incompatible with python3.  Revert it as one step towards
migrating to python3.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
468ef568cf filter-repo (python3): xrange() -> range()
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
511a8f52f8 filter-repo (python3): iteritems() -> items()
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
e5955f397f filter-repo (python3): shebang and imports
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
4d0264ab72 filter-repo: workaround python<2.7.9 exec bug
Python issue 21591 will cause SyntaxError messages to by thrown if using
python versions prior to 2.7.9.  Use the workaround identified in the
bug report: use the exec statement instead of the exec function, even if
this will need to be reverted for python3.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-30 23:50:20 -07:00
Elijah Newren
068dd66b70 filter-repo: Use extra LF on progress messages
Extra LFs are permitted in git-fast-import syntax, and they serve to
make it easier to read the stream (from --dry-run or --debug), if they
are so inclined.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 10:39:15 -07:00
Elijah Newren
5f06c19c55 filter-repo: ensure we properly quote a filename
When we invoked the 'ls' command of fast-import, we just passed the
filename as-is.  That will work for most filenames, but some have to
be quoted.  Make sure we do so.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 10:39:15 -07:00
Elijah Newren
805ce360fa filter-repo: simplify API for parent handling in Commit object
While the underlying fast-export and fast-import streams explicitly
separate 'from' commit (first parent) and 'merge' commits (all other
parents), foisting that separation into the Commit object for
filter-repo forces additional places in the code to deal with that
distinction.  It results in less clear code, and especially does not
make sense to push upon folks who may want to use filter-repo as a
library.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
bec6bd8d3c filter-repo: add testcase with funny characters
Use UTF-8 chars in user names, filenames, branch names, tag names, and
file contents.  Also include invalid UTF-8 in file contents; should be
able to handle binary data.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
0ca3988953 filter-repo: specify sorting order in greater detail
The sorting order of entries written to files in the analysis directory
didn't specify a secondary sort, thus making the order dependent on the
random-ish sorting order of dictionaries and making it inconsistent
between python versions.  While the secondary order didn't matter much,
having a defined order makes it slightly easier to define a single
testcase that can work across versions.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
4cb3bc3459 filter-repo: mark messages for translation
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
7bc87c7f0b filter-repo: enable internationalization
Assuming filter-repo will be merged into git.git, use "git" for the
TEXTDOMAIN, and assume its build system will replace "@@LOCALEDIR@@"
appropriately.

Note that the xgettext command used to grab string translations is
nearly identical to the one for C files in git.git; just use
--language=python instead and add --join-existing to avoid overwriting
the po/git.pot file.  In other words, use the command:
    xgettext -o../git/po/git.pot --join-existing --force-po \
        --add-comments=TRANSLATORS: \
        --msgid-bugs-address="Git Mailing List <git@vger.kernel.org>" \
        --from-code=UTF-8 --language=python \
        --keyword=_ --keyword=N_ --keyword="Q_:1,2" \
        git-filter-repo

To create or update the translation, go to git.git/po and run either of:
    msginit --locale=XX
    msgmerge --add-location --backup=off -U XX.po git.pot

Once you've updated the translation, within git.git just build as
normal.  That's all that's needed.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
0e8a11aa60 filter-repo: add a reminder comment in case I ever use --full-tree
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
a31a381fb8 filter-repo: delete complex code
Over a decade ago, I added code to deal with splitting and splicing
repositories where you weren't always dealing with first parents and
linear histories, and in particular where the mainline tended to be the
second parent (because there was no integrator or special central
gatekeeper like gerrit or github; instead, everyone pushed directly to
the main repository after locally testing, and integration happened via
everyone running 'git pull').  When attempting to splice repositories
the fact that fast-export always gave changes relative to the first
parent caused some grief with my splitting and splicing efforts.

It has been over a decade, I don't know of a good testcase of this
functionality separate from the live repositories I lost access to over
six years ago, git-subtree was released in the meantime which I'm
certain handled the task better, git-fast-export since gained a
--full-tree option which might have provided a better way to attack the
problem  (though with splicing repos you often want work with additive
changes rather than recreating from scratch), and I just don't
quite understand the code anymore anyway.  I think it had some
fundamental limitations that I knew my usecase avoided, but I don't
remember the details (and I'm not certain if this is true).

Even though code coverage hits all but one of the lines, I'd rather
rewrite any needed functionality if the usecase arises, and in view of
what facilities exist today rather than what I was working with a decade
ago.  So, just nuke this code.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
cb9e3dd92c filter-repo: remove rename-related unused code
The original idea was to add --path-rename-(glob|regex) options, but
I like the general flexibility of --filename-callback better for
special cases and keeping the number of command line options at least
slightly in check.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
4b26fcd50c filter-repo: add 'pragma: no cover' to intentionally unreachable lines
There are several lines equivalent to BUG() calls in git that are
supposed to be unreachable, and which exist just to make debugging the
fundamental system problem or refactoring of the code slightly easier by
trying to give a more immediate notification of a problem.  If these
error cases are ever hit and happen to be wrong, then the individual
will at worst get a stacktrace and the program will abort...but that
might arguably be even more helpful.  Since there is no harm in avoiding
the work of finding ways to break the system to force these lines to be
covered, simply exclude them from line coverage counting.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
365afadfd2 filter-repo: add a test just to get line coverage of an unusual error
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
cd9ea5af9b filter-repo: fix incremental imports and add a test
The AncestryGraph setup assumed we had previously seen all commits which
would be used as parents; that interacted badly with doing an
incremental import.  Add a function which can be used to record external
commits, each of which we'll treat like a root commit (i.e. depth 1 and
having no parents of its own).  Add a test to prevent regressions.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
69f7224d95 filter-repo: add tests for invalid fast-import directives
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
3e4e0ed2ed filter-repo: add tests for other startup errors and help option
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
30228bdde2 filter-repo: add tests triggering callback sanity checks
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
dd4f3bd111 filter-repo: add tests checking error paths in mailmap parsing
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
8a1358cc30 filter-repo: add tests checking whether we start in a clean clone
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
8ecd7a0d88 filter-repo: fix bug in checking for uncommitted but staged changes
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
e913ccbe8d filter-repo: add coverage for some corner cases and unusual constructs
There are a number of things not present in "normal" imports that we
nevertheless support and need to be tested:
  * broken timezone adjustment (+051800->+0261; observed in the wild
    in real repos, and adjustment prevents fast-import from dying)
  * commits missing an author (observed in the wild in a real repo;
    just sets author to committer)
  * optional additional linefeeds in the input allowed by
    git-fast-import but usually not written by git-fast-export
  * progress and checkpoint objects
  * progress, checkpoint, and 'everything' callbacks

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
5c80474713 filter-repo: fix an incorrect exit
While most users of filter-repo will just use it as a tool and
RepoFilter.run() is the final function, filter-repo can be used as a
library with additional work being done after calling that function.
So, simply return from that function when it is done rather than calling
sys.exit.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
8e79692ea3 filter-repo: remove unnecessary argument to everything_callback
The everything_callback took two arguments, the first being a string
that was the name of the type of the second argument.  There was no
point for this argument; someone can just compare type(second) to the
relevant classes.  Remove it.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
6e0d846398 filter-repo: fix printing of _IDs
The only times this is ever printed is when debugging filter-repo
itself, or when trying to add tests to get to 100% line coverage.  But
the printing was broken when objects were skipped (which caused a
mapping from int -> None).  Fix the format specifier to handle this
case too.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
3f0bfc2925 filter-repo: avoid relaying progress or checkpoint directives
We don't expect to ever get progress or checkpoint directives in normal
operation, but the --stdin flag makes it a possibility.  In such a case,
the progress directives could actually break our parsing since
git-fast-import will just print it to its stdout, which is what we read
from to find new commit names so we can do commit message hash updating.
So, pass these along to a progress_callback, but don't dump them by
default.  Also, it is not clear checkpoint directives make sense given
that we'll be filtering and only getting a subset of history (and I'm
dubious on checkpoint's utility in general anyway as fast-import is
relatively quick), so pass these along to a callback but don't use them
by default.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
8a8a15b998 filter-repo: add tests for --debug and --dry-run
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
690c824fad filter-repo: avoid coverage testing renames from fast-export
We don't run fast-export with rename detection, even though we have
code for handling it, because we decided to use a rev-list|diff-tree
pipeline instead.  The code was manually tested and determined to be
working and it might be useful in the future so I don't want to just
outright delete it, but since we know we can't trigger it right now,
add a
   # pragma: no cover
on these lines so it doesn't show up on coverage reports.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
0e4d48158f filter-repo: add a script to generate a line coverage report
This also generates line coverage statistics for t/t9391/*.py, but the
point is line coverage of git-filter-repo.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
4beba5df40 filter-repo: fix comments in _parse_literal_command
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
7558cb2198 filter-repo: add more thorough test of --replace-text
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
8e7ab2fbfe filter-repo: fix tracking of referenced-but-removed commits
Since there were multiple places in the code where we returned early
knowing that we didn't have a translation of old_hash to a new_hash, we
need to update _commits_referenced_but_removed from each of them.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
5c72ac5343 filter-repo: replace dead code with an assertion
Due to the invariants we maintain with _commit_renames and
_commit_short_old_hashes (the latter always gets an extra entry in
either a key or a value whenever _commit_renames gains a new key/value
pair), there were a few lines of code that we could not ever reach.
Replace them with an assertion that the condition used for them is
never true.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
8960eaeb8c filter-repo: add testcases for commit message rewriting
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
e19682bc19 filter-repo: add a fairly thorough test for --analyze
The test does check the exact output for the report, meaning if the
output is changed at all this test will need to be updated, but it at
least makes sure we are getting all the right kinds of information.  I
do not expect the output format will change very often.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
a8111d3521 filter-repo: add tests using refs/replace/
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
2b2ab0422e filter-repo: add testcases for --[to-]subdirectory-filter
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00
Elijah Newren
1f0e57bada filter-repo: avoid pruning annotated tags that we have seen
The former logic for keeping track of whether we had seen annotated tags
(and thus whether they were interesting and should avoid being pruned)
was just plain buggy.  I do not know if it was that bad from the start
or there was other surrounding code that made it different that got lost
in one of my history rewrites, but fix it.  I'll include tests of it
with --subdirectory-filter shortly.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-04-29 09:56:38 -07:00