Commit Graph

451 Commits

Author SHA1 Message Date
Elijah Newren
6584dd760d filter-repo: add some docstrings for a few functions
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
2bd86a64bb filter-repo: remove superfluous everything_callback
I introduced this over a decade ago thinking it would come in handy in
some special case, and the only place I used it was in a testcase that
existed almost solely to increase code coverage.  Modify the testcase to
instead demonstrate how it is trivial to get the effects of the
everything_callback without it being present.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
ef2343ac05 filter-repo: clean up RepoFilter callbacks
The specially constructed callbacks in RepoFilter.run() were
superfluous; we already had special callback functions.  Instead of
creating new local functions that call the real callbacks and then do
one extra step, just put the extra wanted code into the real callbacks.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
e97b195229 filter-repo: avoid accidental output after 'done' directive
Using fast-import's feature done capability, any output sent to it after
the 'done' directive will be ignored.  We do not intend to send any such
information, but there have been a couple cases where an accident while
refactoring the code resulted in some information being sent after the
done directive.  To avoid having to debug that again, just close the
output stream after sending the 'done' directive to ensure that we get
an immediate and clear error if we ever run into such a situation again.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
8e482d18a5 filter-repo: ensure compatibility with upcoming git-2.22
The upcoming git-2.22 release will not have the --reencode option to
fast-export; however, since we default to --reencode=yes and that was
the default behavior in all existing versions of git (only to change in
git-2.23), we can just silently leave the option off if we detect we are
running with this version.  However, the diff-tree --combined-all-paths
option from git-2.22 is still mandatory; we cannot run with git versions
older than that (well, with -rc or built-from-source versions, but that
won't matter to most users).

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
3999349be4 filter-repo: fix perf regression; avoid excessive translation
Translating "Parsed %d commits" a hundred thousand times (once per
commit), turned out to be somewhat expensive -- especially since we
were only going to print it out once every few thousand commits.
Translate it once and cache the result, shaving off about 20% of
execution time for a simple rewrite of a test repository (rails).

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
2472d1c93f filter-repo: implement --paths-from-file
This allows the user to put a whole bunch of paths they want to keep (or
want to remove) in a file and then just provide the path to it.  They
can also use globs or regexes (similar to --replace-text) and can also
do renames.  In fact, this allows regex renames, despite the fact that I
never added a --path-rename-regex option.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
9744c57106 filter-repo: change --path-rename to work on matches instead of prefixes
Using an exact path (file or directory) for --path-rename instead of a
prefix removes an ugly caveat from the documentation, makes it operate
similarly to --path, and will make it easier to reuse common code when I
add the --paths-from-file option.  Switch over, and replace the
startswith() check by a call to filename_matches().

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
092d0163d4 filter-repo: implement --use-base-name
This new flag allows people to filter files solely based on their
basename rather than on their full path within the repo, making it
easier to e.g. remove all .DS_Store files or keep all README.md
files.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
fd0b58ecdc filter-repo: minor code simplification
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
a475dce65e filter-repo (README): add a section with information about limitations
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
6e7d36edc1 filter-repo (README): add a section with information about internals
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
955c6cf79a filter-repo (README): link to examples of library usage
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
d94c8ea4c2 filter-repo (README): include callback examples
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
a451261f81 filter-repo (README): include user-and-email-based filtering examples
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
c3d7fe8feb filter-repo (README): include refname-based filtering examples
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
b04ba30e46 filter-repo (README): include content-based filtering examples
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
15c7bb52f1 filter-repo (README): include path-based filtering examples
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-16 10:06:06 -07:00
Elijah Newren
09d54a26b6 filter-repo (README): add a "big picture" overview to rewriting in general
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-16 10:06:06 -07:00
Elijah Newren
128c2a22ae filter-repo (README): separate background information more clearly
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-16 10:06:06 -07:00
Elijah Newren
5ed97e999c filter-repo: rename FileChanges to FileChange
This class only represents one FileChange, so fix the misnomer and make
it clearer to others the purpose of this object.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-16 09:02:40 -07:00
Elijah Newren
346f2ba891 filter-repo: make reencoding of commit messages togglable
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-11 23:33:34 -07:00
Elijah Newren
88c2900e5d filter-repo: add --fake-missing-tagger to unconditionally used flags
For most repos, --fake-missing-tagger will be a no-op.  But if a repo
out there has a missing tagger, then fast-import will choke on it unless
one is inserted; ask fast-export to do that.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-11 23:33:34 -07:00
Elijah Newren
76b71fe92d filter-repo: allow rewriting of hashes in commit messages to be toggled
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-11 23:33:34 -07:00
Elijah Newren
49ff9a74e8 filter-repo: improve the flow of help for program arguments
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-11 13:58:22 -07:00
Elijah Newren
23db0417c8 filter-repo (README): wording fixes, clarifications, and improvements
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-11 13:58:22 -07:00
Elijah Newren
1fa8c2c70b filter-repo: add --replace-refs option
This adds the ability to automatically add new replacement refs for each
rewritten commit (as well as delete or update replacement refs that
existed before the run).  This will allow users to use either new or old
commit hashes to reference commits locally, though old commit hashes
will need to be unabbreviated.  The only requirement for this to work,
is that the person who does the rewrite also needs to push the replace
refs up where other users can grab them, and users who want to use them
need to modify their fetch refspecs to grab the replace refs.

However, other tools external to git may not understand replace refs...

Tools like Gerrit and GitHub apparently do not yet natively understand
replace refs.  Trying to view "commits" by the replacement ref will
yield various forms of "Not Found" in each tool.  One has to instead try
to view it as a branch with an odd name (including "refs/replace/"), and
often branches are accessed via a different URL style than commits so it
becomes very non-obvious to users how to access the info associated with
an old commit hash.

  * In Gerrit, instead of being able to search on the sha1sum or use a
    pre-defined URL to search and auto-redirect to the appropriate code
    review with
      https://gerrit.SITE.COM/#/q/${OLD_SHA1SUM},n,z
    one instead has to have a special plugin and go to a URL like
      https://gerrit.SITE.COM/plugins/gitiles/ORG/REPO/+/refs/replace/${OLD_SHA1SUM}
    but then the user isn't shown the actual code review and will need
    to guess which link to click on to get to it (and it'll only be
    there if the user included a Change-Id in the commit message).
  * In GitHub, instead of being able to go to a URL like
      https://github.SITE.COM/ORG/REPO/commit/${OLD_SHA1SUM}
    one instead has to navigate based on branch using
      https://github.SITE.COM/ORG/REPO/tree/refs/replace/${OLD_SHA1SUM}
    but that will show a listing of commits instead of information about
    a specific commit; the user has to manually click on the first commit
    to get to the desired location.

For now, providing replace refs at least allows users to access
information locally using old IDs; perhaps in time as other external
tools will gain a better understanding of how to use replace refs, the
barrier to history rewrites will decrease enough that big projects that
really need it (e.g. those that have committed many sins by commiting
stupidly large useless binary blobs) can at least seriously contemplate
the undertaking.  History rewrites will always have some drawbacks and
pain associated with them, as they should, but when warranted it's nice
to have transition plans that are more smooth than a massive flag day.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-11 13:24:00 -07:00
Elijah Newren
52f98b6ae5 filter-repo: move post-run ref updating into a separate function
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-11 13:24:00 -07:00
Elijah Newren
2ddbe10034 filter-repo (README): split empty pruning into two sections
Keeping empty pruning as a single section likely makes users only think
about pruning of non-merge commits which become empty.  Since merge
commits can lose parents or become degenerate, it is worth creating a
second section for this; besides, that matches the separate options we
provide to users to control the features.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-11 13:22:20 -07:00
Elijah Newren
2c8f763426 filter-repo: allow users to adjust pruning of empty & degenerate commits
We have a good default for pruning of empty commits and degenerate merge
commits: only pruning such commits that didn't start out that way (i.e.
that couldn't intentionally have been empty or degenerate).  However,
users may have reasons to want to aggressively prune such commits (maybe
they used BFG repo filter or filter-branch previously and have lots of
cruft commits that they want remoed), and we may as well allow them to
specify that they don't want pruning too, just to be flexible.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-11 13:01:19 -07:00
Elijah Newren
3961a82ba4 filter-repo: fix pruning of former merge commits that have become empty
If a commit was a merge in the original repo, and its ancestors on at
least one side have all been filtered away, and the commit has no
filechanges relative to its remaining parent (if any), then this commit
should be pruned.  We had a small logic error preventing such pruning;
fix it by checking len(parents) instead of len(orig_parents).

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-11 13:00:32 -07:00
Elijah Newren
6a6d21aff5 filter-repo: handle implicit parents
fast-import syntax declares how to specify the parents of a commit with
'from' and possibly 'merge' directives, but it oddly also allows parents
to be implicitly specified via branch name.  The documentation is easy
to misread:

  "Omitting the from command in the first commit of a new branch will
   cause fast-import to create that commit with no ancestor."

Note that the "in the first commit of a new branch" is key here.  It is
reinforced later in the document with:

  "Omitting the from command on existing branches is usually desired, as
   the current commit on that branch is automatically assumed to be the
   first ancestor of the new commit."

Desirability of operating this way aside, this raises an interesting
question: what if you only have one branch in some repository, but that
branch has more than one root commit?  How does one use the fast-import
format to import such a repository?  The fast-import documentation
doesn't state as far as I can tell, but using a 'reset' directive
without providing a 'from' reference for it is the way to go.

Modify filter-repo to understand implicit 'from' commits, and to
appropriately issue 'reset' directives when we need additional root
commits.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-11 13:00:32 -07:00
Elijah Newren
89e5c43805 filter-repo: include additional worktrees in sanity startup check
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
aa63db17bc filter-repo: fix overwide program description
Let's keep it just under 80 chars.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
696a4f8d4f filter-repo: merge branch 'python3-conversion' into master
Notable items:
  * We use bytestrings _everywhere_.  This is incredibly annoying to
    me, as I think users will be tempted to use "normal" strings in
    callback functions and get surprised when things compare as
    unequal, but I did like 3-4 python3 conversions with different
    amounts in bytestrings and regular strings, and I always hit
    real world repositories with alternate encodings on user names
    and commit messages (despite commit messages not necessarily
    having a special 'encoding' field).  Further, I was always
    risking munging data the user didn't want by trying to 'decode'
    the bytestrings into unicode, and I was probably slowing down
    performance.  So, in the end I gave up and everything must be a
    bytestring.
  * The performance of the python2 version of filter-repo drifted
    slightly over time with additional features and more robust
    checking (particularly the become-empty and become-degenerate
    pruning), though largely still providing the same performance
    as I highlighted in my BFG/filter-branch/filter-repo comparison.
    There certainly weren't any factors of 2 difference.  A pleasant
    surprise was that the python2->python3 conversion appears to have
    only made a difference of a couple percent to performance and
    some tests were faster and others slower than the python2 version.
    So performance seems to be a wash.

  * The individual commits on python3-conversion do not work
    independently, but rather demonstrate separate aspects of what work
    was needed in the large conversion to python3.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
35052f673d filter-repo (python3): replace strings with bytestrings
This is by far the largest python3 change; it consists basically of
  * using b'<str>' instead of '<str>' in lots of places
  * adding a .encode() if we really do work with a string but need to
    get it converted to a bytestring
  * replace uses of .format() with interpolation via the '%' operator,
    since bytestrings don't have a .format() method.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
385b0586ca filter-repo (python3): bytestr splicing and iterating is different
Unlike how str works, if we grab an array index of a bytestr we get an
integer (corresponding to the ASCII value) instead of a bytestr of
length 1.  Adjust code accordingly.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
12602dae9c filter-repo (python3): f.readline() instead of f.next() and StopIteration
File iterators, at least when opened in binary mode, apparently operately
differently in python3.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
4c05cbe072 filter-repo (python3): bytes() instead of chr() or string join
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
ca5818056d filter-repo (python3): oct strings in python3 use "0o" instead of "0"
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
c3072c7f01 filter-repo (python3): convert StringIO->BytesIO and __str__->__bytes__
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
0279e3882d filter-repo (python3): error messages should be strings instead of bytes
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
9b3134b68c filter-repo (python3): ensure file reads and writes are done in bytes
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
8b8d6b4b43 filter-repo (python3): ensure stdin and args are bytes instead of strings
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
effcd5b9ff filter-repo (python3): convert run_coverage
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
6e78788feb filter-repo (python3): more flush()ing needed under python3
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
ad3c839263 filter-repo (python3): handle conversion of glob to regex
python3 forces a couple issues for us with the conversion of globs to
regexes:
  * fnmatch.translate() will ONLY operate on unicode strings, not
    bytestrings.  Super lame.
  * newer versions of python3 modified the regex style used by
    fnmatch.translate() causing us to need extra logic to 'fixup'
    the regex into the form we want.
Split the code for translating the glob to a regex out into a separate
function which now houses more complicated logic to handle these extra
conditions.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
1a8e247ba7 filter-repo (python3): add a decode() function
We need a function to transform byte strings into unicode strings for
printing error messages and occasional other uses.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
2562f0270c filter-repo (python3): revert "workaround python<2.7.9 exec bug"
Commit ca32c5d9afe2 ("filter-repo: workaround python<2.7.9 exec bug",
2019-04-30) put in a workaround for python versions prior to 2.7.9, but
which was incompatible with python3.  Revert it as one step towards
migrating to python3.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00
Elijah Newren
468ef568cf filter-repo (python3): xrange() -> range()
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-08 08:57:51 -07:00