2011/10/14 Justin Holewinski <span dir="ltr"><<a href="mailto:justin.holewinski@gmail.com">justin.holewinski@gmail.com</a>></span><br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div class="gmail_quote"><div class="im">On Fri, Oct 14, 2011 at 11:06 AM, Na'Tosha Bard <span dir="ltr"><<a href="mailto:natosha@unity3d.com" target="_blank">natosha@unity3d.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
2011/10/14 Justin Holewinski <span dir="ltr"><<a href="mailto:justin.holewinski@gmail.com" target="_blank">justin.holewinski@gmail.com</a>></span><br><div class="gmail_quote"><div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<br><br><div class="gmail_quote"><div><div></div><div>On Wed, Oct 12, 2011 at 1:05 PM, Justin Holewinski <span dir="ltr"><<a href="mailto:justin.holewinski@gmail.com" target="_blank">justin.holewinski@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
I'm excited for the new largefiles extension, and I have been trying it out on some local test repositories. I realize the extension (in its "official" form) is very new and subject to change, but I have a question on how the large files are (or are going to be) stored on the server.<div>
<br></div><div>Let's say I have two local repositories S and C. S represents the "server" repository, and C represents a client clone. If I add large files to C and push to S, the large files appear to be stored in $HOME/.largefiles, not in the .hg directory for S.</div>
</blockquote></div></div></div></blockquote></div><div><br>I believe this makes sense, based on the current implementation. <br></div><div><div> </div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div class="gmail_quote"><div><div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div> It looks like S just contains hashes for the files, which makes sense.</div>
</blockquote></div></div></div></blockquote></div><div><br>This is correct -- the hashes are sitting in S/.hglf -- correct?<br></div></div></blockquote><div><br></div></div><div>They are stored in .hg/store/data/~2ehglf. There appear to be .i files that store all past revision hashes, and the .i files are stored in a structure that mirrors the repository structure. The client repo has the .hglf directory. If I run "hg update" on the server repo, then I get the .hglf directory, and "hg update null" removes it.</div>
</div></blockquote><div><br>The client repo seems correct. The stuff happening on the server sounds like garbage we need to fix.<br> <br></div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div class="gmail_quote">
<div></div><div>Also, this may be a bug: in my test repository, I have all of the largefiles in an assets/ directory. If I run "hg update" on the server, this directory is created. But if I run "hg update null", then the contents of assets/ are deleted, but the directory still remains, unlike other directories that contain only normally-versioned files.</div>
</div></blockquote><div><br>That directory probably won't show up on the server unless you run hg update, because by default the server version has no working copy, right?<br><br></div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div class="gmail_quote"><div class="im">
<div></div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div class="gmail_quote"><div></div><div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div class="gmail_quote"><div><div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div> Incidentally, will there be a config option for this, for users that wish to sandbox all hg-related files in a separate directory?</div>
</blockquote></div></div></div></blockquote></div><div><br>Every large-files enabled repo will have it's own set of standins maintained in repo/.hglf -- I don't see any reason why this should be able to be moved out of the repository because it is repo-specific. Also the standins are very small text files, so why do they need to be elsewhere?<br>
</div></div></blockquote><div><br></div></div><div>I was actually referring to the opposite: will I be able to configure the server to store all largefiles blobs in the .hg directory, or some other user-configurable directory?</div>
</div></blockquote><div><br>I don't believe it is supported yet, but I believe we should add it.<br> <br></div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div class="gmail_quote"><div class="im">
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div class="gmail_quote"><div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div class="gmail_quote"><div><div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div></div><div>Now, let's say I create a new repository accessible over SSH, called S2. If I push C to S2, the largefiles seem to be stored in *both* $HOME/.largefiles (in the SSH account) and the .hg directory for S2. Things are getting a bit inconsistent.</div>
</blockquote></div></div></div></blockquote></div><div><br>That does sound inconsistent -- to me, anyway. There shouldn't be any largefiles in S2/.hg -- there should only be the textfiles with the SHA1 sums in S2/.hglf -- is the S2/.hglf directory there?<br>
</div></div></blockquote><div><br></div></div><div>There is no .hglf directory, the largefiles appear to be stored in .hg/largefiles (in the server repo):</div><div><br></div><div><font face="'courier new', monospace">$ ls -l .hg/largefiles</font></div>
<div><div><font face="'courier new', monospace">total 71584</font></div><div><font face="'courier new', monospace">-rw------- 2 hg hg 71576 2011-10-12 12:49 161215d4d5bcd9e1e92bb9865a1c5dfd5363229a</font></div>
<div><font face="'courier new', monospace">-rw------- 2 hg hg 24376104 2011-10-12 12:49 21dd947c014a25729adca78a1b7eee30ded378e3</font></div><div><font face="'courier new', monospace">-rw------- 2 hg hg 7821 2011-10-12 12:49 2204a70d38f16a931bfbfc1e81f3b18572e1bd69</font></div>
<div><font face="'courier new', monospace">-rw------- 2 hg hg 24376128 2011-10-12 12:49 311990273ded89150c5eb3a84399323189c74425</font></div><div><font face="'courier new', monospace">-rw------- 2 hg hg 7813 2011-10-12 12:49 3f727d8d67f5a7bb7adea5beae0a02cc3ef4dd15</font></div>
<div><font face="'courier new', monospace">-rw------- 2 hg hg 24376116 2011-10-12 12:49 9605bf1cdd8cbba8879ebca4e7ea68d9a949b569</font></div><div><font face="'courier new', monospace">-rw------- 2 hg hg 71567 2011-10-12 12:49 a4572db237727e4d26c9c3230774f1705872221b</font></div>
</div><div class="im"><div><br></div><div> </div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div class="gmail_quote"><div>
<br></div><div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div class="gmail_quote"><div><div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div></div><div>I have not tested HTTP/HTTPS, but what is the expected behavior in this case? There may not be a writable home directory in this case.</div><div><br></div><div><br></div><div>More specifically, what are the planned methods for storing large files on mercurial servers?</div>
</blockquote><div><br></div></div></div><div>Ping? Any comment from the largefiles devs on the planned server storage model?</div></div></blockquote></div><div><br>I'm not really sure we have a concrete plan yet. This extension (at least in this form) is very new.<br>
</div></div></blockquote><div><br></div></div><div>Is it still going to be released with Mercurial 2.0?</div></div></blockquote><div><br>Yes.<br> <br></div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div class="gmail_quote"><div class="im"><div></div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div class="gmail_quote">
<div>
Some of us are expecting to use largefiles with Kiln, which just implements the server-side stuff already. Some people will be migrating from the old bfiles extension, which means they already have a central share set up somewhere (but I assume some conversion will be necessary). Greg is preparing a way for users to migrate from bfiles to largefiles, so he might have some idea on this.<br>
<br>The built-in serving ability of largefiles was developed by the team at FogCreek, so hopefully one of them can reply with what their vision was.<br><br>My initial thought is that:<br><br>The $HOME/.largefiles cache should be configurable server-side, if it is not already<br>
Each repo should only contain the hashes in repo/.hglf -- when largefiles are uploaded, they should probably all go directly to the cache.<br></div></div></blockquote><div><br></div></div><div>That makes sense to me, as long as the cache path is configurable. :)</div>
</div></blockquote><div><br>Let's wait for the other Largefiles devs to weigh in on the issue before we make a plan.<br><br>Cheers,<br>Na'Tosha <br></div></div><br clear="all"><br>-- <br><div><div><span style="color: rgb(153, 153, 153);"><b>Na'Tosha Bard</b></span></div>
<div><font color="#999999">Build & Infrastructure Developer | Unity Technologies</font></div><div><font color="#999999"><br></font></div><div><font color="#999999"><b>E-Mail:</b> <a href="mailto:natosha@unity3d.com" target="_blank">natosha@unity3d.com</a></font></div>
<div><font color="#999999"><b>Skype:</b> natosha.bard</font></div></div><br>