Commit Graph

82 Commits

Author SHA1 Message Date
Elijah Newren
2bcf83aa7b filter-repo: avoid dying on tags; strip/rewrite by default
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:44:55 -08:00
Elijah Newren
af081d0fce filter-repo: add automatic rewriting of commit hashes in commit messages
Commit messages often refer to past commits; while rewriting commits we
would also like to update these commit messages to refer to the new
commit names.

In the case that a commit message references another commit which was
dropped by the filtering process, we have no way to rewrite the commit
message to reference a valid commit hash.  Instead of dying, note the
suboptimal commit in the suboptimal-issues file.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:44:55 -08:00
Elijah Newren
f95308c5eb filter-repo: add handling of 'original-oid' directive
This will be used later to help with commit message rewriting (so that
commits can continue to refer to other commits in their history, using
the new rewritten hashes for those commits), and perhaps also in
removing blobs by id.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:44:55 -08:00
Elijah Newren
057947f6ff filter-repo: prune commits that started empty if they now have no parents
If ancient history that pre-dated some subdirectory had a few empty
commits, we would rather those all got pruned as well.  Empty commits
from the original repository should only be retained if they have at
least one retained parent.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:44:55 -08:00
Elijah Newren
e3fde7689c filter-repo: record suboptimality notes about changing merges to non-merges
When the pruning of empty commits causes a culling of parents of a merge
commit, so that the merge commit drops to just one parent, the commit
likely becomes misleading since the commit is no longer a merge commit
but the message probably implies it is.  (e.g. "Merge branch maint into
master").  There's nothing we can do to automatically fix this, but we
can note it as a suboptimal issue in the filtering process.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:44:55 -08:00
Elijah Newren
9e02ac95e4 filter-repo: record metadata for remapping for refs and commits
Our filtering process will rewrite (and drop) commits, causing refs to
also get updated.  A useful debugging aid for users is to write metadata
showing the mapping from old commit IDs to new commit IDs, and from the
hash that old refs pointed and the hash that the new ones do.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:44:55 -08:00
Elijah Newren
04260a3aa4 filter-repo: parse fast-export --reference-excluded-parents output
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:44:55 -08:00
Elijah Newren
d13f7e9178 filter-repo: fix detection of merge becoming empty commit
In the previous commit, we detected when an entire line of history back to
a common ancestor of the merge became empty commits, and avoided having a
commit be merged with itself.  This commits looks through the changes
specified in the commit, which are always specified relative to the first
parent, so that if the first parent side was the empty one we can still
detect if the merge commit adds no extra changes relative to its remaining
parent.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:44:55 -08:00
Elijah Newren
70505e00f9 filter-repo: avoid merging a commit with one of its own ancestors
Pruning of empty commits can cause an entire line of history to become
empty and be pruned, resulting in a merge commit that merges some commit
with one of its ancestors.  In such a case, we should remove the
unnecessary parent(s) -- which can and will often result in the merge
commit being empty so we can remove it as well.

Currently, if the side that becomes empty is the first parent side, then
we do not detect if the commit becomes empty, due to the way that
fast-export lists changes in a merge commit relative to first parent only.
A subsequent commit will address this.

Note that the callbacks could theoretically insert additional commits or
reparent our commit on top of something else, meaning that the ancestry
graph might need post-callback updates.  However, in any extreme case
where that mattered, we would more or less need full updates to the
ancestry graph to be made for all the new commits from the callback as
well, and once we expect the callback to handle any ancestry graph
updates it can handle modifying it for the current commit.  However, it
is hard to come up with a case where it matters, since for the most part
we just want to know whether our filtering causes commits to become
empty and knowing the source repo we are exporting from is sufficient
information without knowing any new commits inserted or reparenting that
happens elsewhere.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:44:55 -08:00
Elijah Newren
fa515c8d10 filter-repo: protect against truncated fast-import input
Use the 'feature done' ability to mark when the fast-import stream is
finished, so that an aborted run (due to running into some kind of bug
while filtering, whether a bug in the code, or an error in the repo or
flags specified for the case under consideration) won't cause the repo
to be rewritten.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:44:55 -08:00
Elijah Newren
e5a3a134b1 filter-repo: retain refs that happen to point at commits that become empty
It may be that the only time a reference is shown in the fast-export stream
is for a commit which will become empty due to the filtering.  We do not
want such refs to be left out and thus not be updated; we want them to
instead be set to the nearest non-empty ancestor.  Only if it has no
non-empty ancestor would we want it to be stripped out.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:44:55 -08:00
Elijah Newren
064e2c0ef4 filter-repo: add --quiet option
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:44:55 -08:00
Elijah Newren
e6731225f8 filter-repo: allow importing into an empty repo
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:44:55 -08:00
Elijah Newren
0ecfad479e filter-repo: add parsing of more types of fast-export commands
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:44:55 -08:00
Elijah Newren
a7531af120 filter-repo: filtering note with empty commits and --dry-run
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:44:55 -08:00
Elijah Newren
3914c1377b filter-repo: add --stdin option
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:44:55 -08:00
Elijah Newren
cf460406d7 filter-repo: code cleanup
Slightly re-order the code to make input, output, and filtering sections
distinct.  Also, avoid running `git fast-import` at all when we're in
--dry-run mode.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:44:55 -08:00
Elijah Newren
3f8ce81aa2 filter-repo: prune parents made redundant by filtering
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:44:54 -08:00
Elijah Newren
f103735e01 filter-repo: implement --debug
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:44:27 -08:00
Elijah Newren
9499c78b94 filter-repo: implement --dry-run
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:44:27 -08:00
Elijah Newren
40f90c9cb8 filter-repo: move git_dir determination into function for future re-use
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:44:27 -08:00
Elijah Newren
4fed91af18 filter-repo: allow FastExportFilter to take file-like objects
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:44:27 -08:00
Elijah Newren
636a3cf575 filter-repo: add basic path filtering
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:44:27 -08:00
Elijah Newren
a427a80322 filter-repo: skeleton of new tool
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:44:26 -08:00
Elijah Newren
e2b8b68d3a filter-repo: use subprocess explicitly; make it easier to wrap debug version
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:44:04 -08:00
Elijah Newren
3457348c63 filter-repo: remove excessive hashes
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:44:03 -08:00
Elijah Newren
badd03105b filter-repo: fix really old typo
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:43:31 -08:00
Elijah Newren
2977725634 filter-repo: enable usage with --no-data
Regex is kinda sloppy, someone should slap me and fix that.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:43:31 -08:00
Elijah Newren
a6b008eaac filter-repo: don't default to creating bare repositories for importing into
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:43:31 -08:00
Elijah Newren
7e0b7a9fc2 filter-repo: only require two arguments to record_id_rename
We always called record_id_rename with handle_transitivity set to True, and
I do not know of a use case that would do otherwise so let's just hardcode
that value.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:43:31 -08:00
Elijah Newren
a8b39e58b3 filter-repo: rename _TimeZone to FixedTimeZone for external usage
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:43:31 -08:00
Elijah Newren
e7311b6db9 filter-repo: reinstate the id_offset
My idea to use --export-marks and --import-marks to avoid the need for the
id_offset was not tested and apparently a bad idea.  When splicing together
multiple repositories, the second will croak if we pass it --import-marks
with a file having sha1sums that don't exist in that repository.

I'm afraid this might conflict with the --import-marks stuff used in collab
so I've only enabled it for streams beyond the first.  So there might be an
issue using --import-marks on a second or later fast-export output stream,
but I can't think of a use case for that...

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:43:31 -08:00
Elijah Newren
fbd3a04b7f filter-repo: some wording and line-wrap cleanups
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:43:31 -08:00
James Foucar
f74761e803 filter-repo: lots of documentation additions
Signed-off-by: Elijah Newren <enewren@sandia.gov>
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:43:29 -08:00
Elijah Newren
4d5a9cf552 filter-repo: add a few documentation files
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:42:23 -08:00
Elijah Newren
9a85c6a1ae filter-repo: fix parsing of filechanges with skipped blobs
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:42:23 -08:00
Elijah Newren
d2d6d79db0 filter-repo: fix __all__ declaration
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:42:23 -08:00
Elijah Newren
adc3d52d26 filter-repo: add parsing of progress and checkpoint fast-export objects
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:42:23 -08:00
Elijah Newren
48aaedfc32 filter-repo: add parsing of (annotated) tag objects
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:42:23 -08:00
Elijah Newren
a644632a83 filter-repo: provide default args for get_commit_count()
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:42:23 -08:00
Elijah Newren
86a86fc074 filter-repo: remove the id_offset
Filtering input from multiple repositories can still be done; however, to
avoid overloading of mark numbers, one should pass --export-marks=<file>
to the first git fast-export and pass --import-marks=<file> to the second.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:42:23 -08:00
Elijah Newren
bfbc07d3a7 filter-repo: encapsulate input line advancement
Have all callers of input.readline() be done through _advance_nextline()

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:42:23 -08:00
Elijah Newren
85b1980d17 filter-repo: avoid using mark ids referred to in an --import-marks file
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:42:23 -08:00
Elijah Newren
a20bf1957f filter-repo: cleanups to gathering the commit count
Two things:
  * rename get_total_commits -> get_commit_count
  * accept rev-list arguments

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:42:23 -08:00
Elijah Newren
d099d2628b filter-repo: automatically drop commits whose changes are filtered out
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:42:23 -08:00
Elijah Newren
4998de6751 filter-repo: handle ahead-of-UTC timezones too
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:42:23 -08:00
Elijah Newren
307a31fd54 filter-repo: have author_date and committer_date be datetime objects
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:42:23 -08:00
Elijah Newren
35fdb05c3c filter-repo: streamline common/simple cases to require fewer calls and args
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:42:23 -08:00
Elijah Newren
cb29d84f48 filter-repo: fix skipping of blob files
Make sure commits don't reference skipped blob files.

Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:42:23 -08:00
Elijah Newren
aba66f6d42 filter-repo: duct tape and bailing wire...
Signed-off-by: Elijah Newren <newren@gmail.com>
2019-02-09 14:42:23 -08:00