Commit Graph

272 Commits

Author SHA1 Message Date
Elijah Newren
88c1269d5a filter-repo: ensure branches are updated as we go
When we prune a commit for being empty, there is no update to the branch
associated with the commit in the fast-import stream.  If the parent
commit had been associated with a different branch, then the branch
associated with the pruned commit would not be updated without
additional measures.  In the past, we resolved this by recording that
the branch needed an update in _seen_refs.  While this works, it is a
bit more complicated than just issuing an immediate Reset.  Also, note
that we need to avoid calling callbacks on that Reset because those
could rename branches (again, if the commit-callback already renamed
once) causing us to not update the intended branch.

There was actually one testcase where the old method didn't work: when a
branch was pruned away to nothing.  A testcase accidentally encoded the
wrong behavior, hiding this problem.  Fix the testcase to check for
correct behavior.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-06-22 22:32:04 -06:00
Elijah Newren
aaeadac6df filter-repo: fix explicit ref deletion via reset directive
We previously did this incorrectly, but due to our assumptions of
full-history rewriting and deleting of unseen refs, we got away with it.
Fix this for partial history rewrites.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-06-22 22:32:04 -06:00
Elijah Newren
e34dff11a1 filter-repo: remove dead code
Commit 1f0e57bada ("filter-repo: avoid pruning annotated tags that we
have seen", 2019-03-07) left behind the setting of a variable,
full_ref, that is no longer used.  Remove it.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-06-22 22:32:04 -06:00
Elijah Newren
aa9ab9df9f filter-repo: refine choice of when to skip blobs
We can pass --no-data to fast-export in one additional case.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-06-22 22:31:49 -06:00
Elijah Newren
7d42c2093c filter-repo: limit splicing repos warning to test that splices repos
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-06-08 09:05:31 -07:00
Elijah Newren
b6a35f8dcd filter-repo: implement --strip-blobs-with-ids
Add a flag allowing for specifying a file filled with blob-ids which
will be stripped from the repository.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
d958b0345c filter-repo: add basic built-in docs covering callbacks and examples
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
89f9fbbb6d filter-repo: partial repo filtering considerations
Fix a few issues and add a token testcase for partial repo filtering.
Add a note about how I think this is not a particularly interesting or
core usecase for filter-repo, even if I have put some good effort into
the fast-export side to ensure it worked.  If there is a core usecase
that can be addressed without causing usability problems (particularly
the "don't mix old and new history" edict for normal rewrites), then
I'll be happy to add more testcases, document it better, etc.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
1a887c5c13 filter-repo: more careful handling of --source and --target
Make several fixes around --source and --target:
  * Explain steps we skip when source or target locations are specified
  * Only write reports to the target directory, never the source
  * Query target git repo for final ref values, not the source
  * Make sure --debug messages avoid throwing TypeErrors due to mixing
    strings and bytes
  * Make sure to include entries in ref-map that weren't in the original
    target repo
  * Don't:
     * worry about mixing old and new history (i.e. nuking refs
       that weren't updated, expiring reflogs, gc'ing)
     * attempt to map refs/remotes/origin/* -> refs/heads/*
     * disconnect origin remote
  * Continue (but only in target repo):
     * fresh-clone sanity checks
     * writing replace refs
     * doing a 'git reset --hard'

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
587f727d19 filter-repo: implement --strip-blobs-bigger-than
Add a flag for filtering out blob based on their size, and allow the
size to be specified using 'K', 'M', or 'G' suffixes.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
598661dcf4 filter-repo: make logic to get blob sizes reusable
Create a new function, GitUtils.get_blob_sizes() to hold some logic
that used to be at the beginning of RepoAnalyze.gather_data().  This
will allow reuse of this functionality within RepoFilter.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
1b106eeac9 filter-repo: fix ref-map generation bug when commit at ref tip is pruned
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
c73c0304b0 filter-repo: pass more canonical ordering of files to fast-import
Although fast-import can take file changes in any order, trying to debug
by comparing the original fast-export stream to the filtered version is
difficult if the files are randomly reordered.  Sometimes we aren't
comparing the filtered version to the original but just looking at the
stream passed to fast-import, in which case having the files in sorted
order may help.

Our accumulation of file_changes into a dict() in order to check for
collisions when renaming had the unfortunate side effect of sorting
files by internals of dictionary ordering.  Although the files started
in sorted order, we don't in general want to use the original order
because renames can cause filenames to become out-of-order.  Just apply
a simple sort at the end.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
41c91150ec filter-repo: mark another incompatibility with fast-export's -M and -C
I suspect at some point someone will try to pass -M or -C to
fast-export; may as well leave a note in the code about another place
that's incompatible while I'm thinking about it.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
6fb7da0f0a filter-repo: rename to --prune-empty and --prune-degenerate
Imperative form sounds better than --empty-pruning and
--degenerate-pruning, and it probably works better with command line
completion.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
ece8a74df9 filter-repo (README): add instructions on parent rewriting
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
4c25fe7a37 filter-repo: handle reset to specific ref and deletion
The reset directive can specify a commit hash for the 'from' directive,
which can be used to reset to a specify commit, or, if the hash is all
zeros, then it can be used to delete the ref.  Support such operations.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
0b70b72150 filter-repo: provide extra metadata to some callbacks
For other programs importing git-filter-repo as a library and passing a
blob, commit, tag, or reset callback to RepoFilter, pass a second
parameter to these functions with extra metadata they might find useful.
For simplicity of implementation, this technically changes the calling
signature of the --*-callback functions passed on the command line, but
we hide that behind a _do_not_use_this_variable parameter for now, leave
it undocumented, and encourage folks who want to use it to write an
actual python program that imports git-filter-repo.  In the future, we
may modify the --*-callback functions to not pass this extra parameter,
or if it is deemed sufficiently useful, then we'll rename the second
parameter and document it.

As already noted in our API compatibilty caveat near the top of
git-filter-repo, I am not guaranteeing API backwards compatibility.
That especially applies to this metadata argument, other than the fact
that it'll be a dict mapping strings to some kind of value.  I might add
more keys, rename them, change the corresponding value, or even remove
keys that used to be part of metadata.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
c58e83ea49 filter-repo: fix obvious comment typo
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
ad73b5ed5f filter-repo: minor cleanups of RepoFilter function names
Fix visibility of several functions, and make the callbacks have a more
consistent naming.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
27f08be754 filter-repo: consolidate filtering functions into RepoFilter
Location of filtering logic was previously split in a confusing fashion
between FastExportFilter and RepoFilter.  Move all filtering logic from
FastExportFilter into RepoFilter, and rename the former to
FastExportParser to reflect this change.

One downside of this change is that FastExportParser's _parse_commit
holds two pieces of information (orig_parents and had_file_changes)
which are not part of the commit object but which are now needed by
RepoFilter.  Adding those bits of info to the commit object does not
make sense, so for now we pass an auxiliary dict with the
commit_callback that has these two fields.  This information is not
passed along to external commit_callbacks passed to RepoFilter, though,
which seems suboptimal.  To be fair, though, commit_callbacks to
RepoFilter never had access to this information so this is not a new
shortcoming, it just seems more apparent now.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
6584dd760d filter-repo: add some docstrings for a few functions
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
2bd86a64bb filter-repo: remove superfluous everything_callback
I introduced this over a decade ago thinking it would come in handy in
some special case, and the only place I used it was in a testcase that
existed almost solely to increase code coverage.  Modify the testcase to
instead demonstrate how it is trivial to get the effects of the
everything_callback without it being present.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
ef2343ac05 filter-repo: clean up RepoFilter callbacks
The specially constructed callbacks in RepoFilter.run() were
superfluous; we already had special callback functions.  Instead of
creating new local functions that call the real callbacks and then do
one extra step, just put the extra wanted code into the real callbacks.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
e97b195229 filter-repo: avoid accidental output after 'done' directive
Using fast-import's feature done capability, any output sent to it after
the 'done' directive will be ignored.  We do not intend to send any such
information, but there have been a couple cases where an accident while
refactoring the code resulted in some information being sent after the
done directive.  To avoid having to debug that again, just close the
output stream after sending the 'done' directive to ensure that we get
an immediate and clear error if we ever run into such a situation again.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
8e482d18a5 filter-repo: ensure compatibility with upcoming git-2.22
The upcoming git-2.22 release will not have the --reencode option to
fast-export; however, since we default to --reencode=yes and that was
the default behavior in all existing versions of git (only to change in
git-2.23), we can just silently leave the option off if we detect we are
running with this version.  However, the diff-tree --combined-all-paths
option from git-2.22 is still mandatory; we cannot run with git versions
older than that (well, with -rc or built-from-source versions, but that
won't matter to most users).

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
3999349be4 filter-repo: fix perf regression; avoid excessive translation
Translating "Parsed %d commits" a hundred thousand times (once per
commit), turned out to be somewhat expensive -- especially since we
were only going to print it out once every few thousand commits.
Translate it once and cache the result, shaving off about 20% of
execution time for a simple rewrite of a test repository (rails).

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
2472d1c93f filter-repo: implement --paths-from-file
This allows the user to put a whole bunch of paths they want to keep (or
want to remove) in a file and then just provide the path to it.  They
can also use globs or regexes (similar to --replace-text) and can also
do renames.  In fact, this allows regex renames, despite the fact that I
never added a --path-rename-regex option.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
9744c57106 filter-repo: change --path-rename to work on matches instead of prefixes
Using an exact path (file or directory) for --path-rename instead of a
prefix removes an ugly caveat from the documentation, makes it operate
similarly to --path, and will make it easier to reuse common code when I
add the --paths-from-file option.  Switch over, and replace the
startswith() check by a call to filename_matches().

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
092d0163d4 filter-repo: implement --use-base-name
This new flag allows people to filter files solely based on their
basename rather than on their full path within the repo, making it
easier to e.g. remove all .DS_Store files or keep all README.md
files.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
fd0b58ecdc filter-repo: minor code simplification
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
a475dce65e filter-repo (README): add a section with information about limitations
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
6e7d36edc1 filter-repo (README): add a section with information about internals
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
955c6cf79a filter-repo (README): link to examples of library usage
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
d94c8ea4c2 filter-repo (README): include callback examples
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
a451261f81 filter-repo (README): include user-and-email-based filtering examples
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
c3d7fe8feb filter-repo (README): include refname-based filtering examples
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
b04ba30e46 filter-repo (README): include content-based filtering examples
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-30 22:07:48 -07:00
Elijah Newren
15c7bb52f1 filter-repo (README): include path-based filtering examples
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-16 10:06:06 -07:00
Elijah Newren
09d54a26b6 filter-repo (README): add a "big picture" overview to rewriting in general
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-16 10:06:06 -07:00
Elijah Newren
128c2a22ae filter-repo (README): separate background information more clearly
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-16 10:06:06 -07:00
Elijah Newren
5ed97e999c filter-repo: rename FileChanges to FileChange
This class only represents one FileChange, so fix the misnomer and make
it clearer to others the purpose of this object.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-16 09:02:40 -07:00
Elijah Newren
346f2ba891 filter-repo: make reencoding of commit messages togglable
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-11 23:33:34 -07:00
Elijah Newren
88c2900e5d filter-repo: add --fake-missing-tagger to unconditionally used flags
For most repos, --fake-missing-tagger will be a no-op.  But if a repo
out there has a missing tagger, then fast-import will choke on it unless
one is inserted; ask fast-export to do that.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-11 23:33:34 -07:00
Elijah Newren
76b71fe92d filter-repo: allow rewriting of hashes in commit messages to be toggled
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-11 23:33:34 -07:00
Elijah Newren
49ff9a74e8 filter-repo: improve the flow of help for program arguments
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-11 13:58:22 -07:00
Elijah Newren
23db0417c8 filter-repo (README): wording fixes, clarifications, and improvements
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-11 13:58:22 -07:00
Elijah Newren
1fa8c2c70b filter-repo: add --replace-refs option
This adds the ability to automatically add new replacement refs for each
rewritten commit (as well as delete or update replacement refs that
existed before the run).  This will allow users to use either new or old
commit hashes to reference commits locally, though old commit hashes
will need to be unabbreviated.  The only requirement for this to work,
is that the person who does the rewrite also needs to push the replace
refs up where other users can grab them, and users who want to use them
need to modify their fetch refspecs to grab the replace refs.

However, other tools external to git may not understand replace refs...

Tools like Gerrit and GitHub apparently do not yet natively understand
replace refs.  Trying to view "commits" by the replacement ref will
yield various forms of "Not Found" in each tool.  One has to instead try
to view it as a branch with an odd name (including "refs/replace/"), and
often branches are accessed via a different URL style than commits so it
becomes very non-obvious to users how to access the info associated with
an old commit hash.

  * In Gerrit, instead of being able to search on the sha1sum or use a
    pre-defined URL to search and auto-redirect to the appropriate code
    review with
      https://gerrit.SITE.COM/#/q/${OLD_SHA1SUM},n,z
    one instead has to have a special plugin and go to a URL like
      https://gerrit.SITE.COM/plugins/gitiles/ORG/REPO/+/refs/replace/${OLD_SHA1SUM}
    but then the user isn't shown the actual code review and will need
    to guess which link to click on to get to it (and it'll only be
    there if the user included a Change-Id in the commit message).
  * In GitHub, instead of being able to go to a URL like
      https://github.SITE.COM/ORG/REPO/commit/${OLD_SHA1SUM}
    one instead has to navigate based on branch using
      https://github.SITE.COM/ORG/REPO/tree/refs/replace/${OLD_SHA1SUM}
    but that will show a listing of commits instead of information about
    a specific commit; the user has to manually click on the first commit
    to get to the desired location.

For now, providing replace refs at least allows users to access
information locally using old IDs; perhaps in time as other external
tools will gain a better understanding of how to use replace refs, the
barrier to history rewrites will decrease enough that big projects that
really need it (e.g. those that have committed many sins by commiting
stupidly large useless binary blobs) can at least seriously contemplate
the undertaking.  History rewrites will always have some drawbacks and
pain associated with them, as they should, but when warranted it's nice
to have transition plans that are more smooth than a massive flag day.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-11 13:24:00 -07:00
Elijah Newren
52f98b6ae5 filter-repo: move post-run ref updating into a separate function
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-11 13:24:00 -07:00
Elijah Newren
2ddbe10034 filter-repo (README): split empty pruning into two sections
Keeping empty pruning as a single section likely makes users only think
about pruning of non-merge commits which become empty.  Since merge
commits can lose parents or become degenerate, it is worth creating a
second section for this; besides, that matches the separate options we
provide to users to control the features.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-05-11 13:22:20 -07:00