[RFC] kbfiles: an extension to track binary files with less wasted bandwidth

[Andrew Pritchard]

> 1) I strongly believe you need some better docs: just trying to piece this
>   thing together from 'hg help' is not enough, which is why I wrote usage.txt
>   for bfiles. Please steal that and hack it up so it describes largefiles.
>   Your post to this list that started this thread a week or two ago was also
>   excellent: I suggest you crib from it liberally.

As of a few hours before you sent this message, I added a usage.txt
and updated a lot of the documentation.  I'm not aware of any
documentation that is still missing or wrong, so if you notice any,
mention it.

> 2) test-lockout.t is failing. I tried Mercurial 1.7, 1.8, and 1.9 -- different
>   failures, but it failed with all of them. Let me know if you cannot reproduce
>   and I'll give more detail.

Without seeing the failure, I'm going to guess that it is limited to
output styling of the "this repository uses largefiles" message when
interacting with a server.  If this is the case, this is because
you're using hg-stable or a revision of Mercurial before f4522df38c65,
where support is added for explicitly handling and formatting errors
in remote repositories.

> 3) The copyright attribution is inaccurate

Fixed

> 4) I see you made liberal use of my hgtest.py module. Too bad: I'm pretty sure
>   Matt won't like that. The best thing about hgtest.py is that it offended Matt
>   so deeply that he implemented the fine new unified test system in
> Mercurial 1.7.
>   The worst thing about it is that converting tests from hgtest.py to
> unified is
>   a slow, painful, tedious manual process. I've converted many of the
> tests I wrote for
>   bfiles, but not all.

I'm not really sure what to do about this; there are a lot of tests,
and I agree that converting them all would be painful.  Still, maybe
it's necessary.

> 5) Has anyone reviewed the changes to bfiles since you forked to see
> if there's anything there that needs to be in largefiles?

The first commit after the common ancestor is
http://hg.gerg.ca/hg-bfiles/rev/069cdf479b30.  I'll sift through them
if you don't get to it first.

> 6) It would be nice to make migration from bfiles painless and transparent.
>   My first suggestion: make the '.hglf' prefix configurable. Then bfiles
>   users can just set it to .hgbfiles and not have to go through a painful
>   repository conversion just to remap the standin filenames.

The problem with a configurable prefix is that it makes a huge mess
out of interaction between clients and servers, in that clients and
servers would both have to agree on the prefix; if they didn't, all
sorts of crazy and confusing things could happen.  Specifically, Kiln
likes to (and in a future where largefiles is bundle with Mercurial,
lots of other hosting providers will like to; and in a few changesets,
hg serve will probably like to) refuse pushes that reference
largefiles it doesn't know about.  If the prefix is mismatched between
client and server, it would be possible to push changes without their
corresponding largefiles, or even worse the server could reject
innocent pushes that happen to have the server's configured prefix.
And, even more troubling for hosting providers, their clients may each
use their own prefixes, giving them no way to figure out what the
prefix is.  If you really want to use your bfiles repositories as-is,
it's easy to write a several-line extension that just sets
lfutil.shortname in extsetup(ui), but I'm planning on adding a
conversion script anyway (I think it's likely to be nothing more than
a filemap for the convert extension).

> I still very much want to see a repo that *accurately* preserves the
> history of bfiles + kbfiles + largefiles.

I can work on extracting the commits in the Kiln extensions repo that
touch the bfiles code and dump them into a repository on
http://developers.kilnhg.com.  For the moment, actually combining
these into a single repo is not at the top of my to-do list, but at
least all the history will be available.