<div dir="ltr">Hello Iulian,<br>hello mercurial-dev,<br><br>this is another message in which I collect information<br>about a particular issue we encountered in<br>the command server GSoC project that starts to be scattered<br>
across several places.<br><br>== TL;DR ==<br><br>how much data will the c-hglib hg_log() function buffer at a time?<br>As much as needed to represent a changeset, no matter how big.<br><br><br>== Long version ==<br><br>The c-hglib API function that will invoke `hg log`<br>
cannot cache all data at once; for a software project<br>that spans over years, the amount of data in the history<br>can be simply too big to fit in memory.<br><br>So how big is the atomic quantity of data that c-hglib will cache?<br>
<br>We started thinking about a fixed number of bytes, say 4096.<br>But recently a better idea came out and gained consensus,<br>which is likely to be adopted in the definitive implementation:<br><br>the c-hglib equivalent of `hg log` will cache<br>
enough data to represent a single changeset,<br>no matter how big it is.<br><br>The rationale behind that is: if you cannot fit<br>one of your commit objects into your memory,<br>you have worse problems to worry about than<br>
the one we're addressing here.<br><br><br>== Digression: Level 0, Level 1 ==<br><br>With respect to the three "levels" of the API [1],<br>the hg_rawread() level 0 function will still read in chunks<br>of 4 KB; but the level 1 function hg_log() will cache<br>
more flexibly depending on the size of the current changeset.<br><br><br>== References ==<br><br>Here I list all contributions that led to the above design decision.<br><br>mpm on June 27th[2]:<br>:::: <br>:::: char buf[4096]; hg_handle *handle;<br>
:::: <br>:::: handle = hg_open("some/repo");<br>:::: hg_rawcommand(handle, "hg log -v");<br>:::: while(hg_rawread(handle, buf, 4096))<br>:::: printf("got: %s", buf);<br>:::: printf("exit code is: %d", hg_exitcode(handle));<br>
:::: hg_close(handle); <br><br>mpm on July 2nd [IRC]:<br>:::: <br>:::: You cannot return an exit code from hg_rawcommand. [...]<br>:::: Ok, let's say I do hg_rawcommand("this command will run<br>:::: for four days and output 500G of data")..<br>
:::: when does this API call return?<br>:::: Can I convince you that the answer should not be<br>:::: "four days from now after trying to buffer 500G of data"?<br>:::: [...]<br>:::: The client library cannot/should not internally buffer<br>
:::: unknown huge masses of data.<br><br>kevin on Aug 4th[3]:<br>::::<br>:::: Again, for things like `hg log`, you'll want to be able<br>:::: to read the output (and errors) long before<br>:::: you could possibly get the exit code.<br>
<br>ggherdov on Aug 4th[4]:<br>:::: <br>:::: But when moving on to level 1,<br>:::: a function like hg_log() would be all-in-one,<br>:::: meaning it will be implemented like<br>:::: <br>:::: int hg_log(...) {<br>:::: hg_rawcommand(...);<br>
:::: while(hg_rawread(...)) { ... }<br>:::: return hg_exitcode(...);<br>:::: } <br>:::: <br>:::: [...] which, at this point, confuses me.<br><br>kevin on Aug 5th[5]:<br>:::: <br>:::: It shouldn't be that confusing; <br>
:::: calling hg_log() shouldn't buffer data any more<br>:::: than calling hg_rawcommand("log", ...) should.<br>:::: The difference between the two calls is that hg_log()<br>:::: should return the results in a more structured form for<br>
:::: programmatic manipulation [...]<br><br>martin schröder on Aug 6th[6]:<br>:::: <br>:::: How can you create a list of structured data<br>:::: without buffering it? The only solution<br>:::: I can think of would be kind of like MySQL does:<br>
:::: <br>:::: hg_log_entry_t *le; int return_value;<br>:::: <br>:::: hg_log(...);<br>:::: <br>:::: while((le = hg_fetch_log_entry())) {<br>:::: printf("ID %s, rev %s, user %s, date %s, description %s\n",<br>
:::: le->id, le->rev, le->user, le->date, le->description);<br>:::: }<br>:::: <br>:::: return_value = hg_get_return(); <br><br>mpm on Aug 7th [IRC]:<br>:::: <br>:::: hg_log() can't cache all data.<br>
:::: Probably wants to either<br>:::: do something that looks like an iterator<br>:::: or have callbacks.<br><br>iulians on Aug 7th [IRC]:<br>:::: <br>:::: Let's make an example. Let's say we call hg_log<br>:::: that provides us an iterator<br>
:::: we can call the iterator to get first rev<br>:::: and then we call again for the next rev<br>:::: like Martin Schröder suggests<br><br>mpm & durin42 on Aug 13th [IRC]:<br>:::: <br>:::: <mpm> I think it's fair to allocate one cset worth of data,<br>
:::: however much that is. A cset description can get as large as 2G,<br>:::: which is unlikely to happen in practice.<br>:::: <iulians> so, I could say that it's ok to not worry about the size for a cset?<br>
:::: Probably the user will have enough RAM or space...<br>:::: <durin42> Yeah. If a user can't inflate the changeset object in RAM,<br>:::: they're probably screwed already.<br><br>mg on Aug 19th [7]:<br>
:::: <br>:::: I think it was established that it is okay<br>:::: to return the data for a single changeset as a single value<br>:::: -- the fields in a changeset are not that big.<br><br>[1] <a href="http://markmail.org/message/wdwlrwv6lcacwmrs">http://markmail.org/message/wdwlrwv6lcacwmrs</a><br>
[2] <a href="http://markmail.org/message/tc6hsvl7fofdjqcl">http://markmail.org/message/tc6hsvl7fofdjqcl</a><br>[3] <a href="http://markmail.org/message/3alrorohanhiomqb">http://markmail.org/message/3alrorohanhiomqb</a><br>
[4] <a href="http://markmail.org/message/sfz3qn6scrxwib36">http://markmail.org/message/sfz3qn6scrxwib36</a><br>[5] <a href="http://markmail.org/message/e3stgfheanp4ihxw">http://markmail.org/message/e3stgfheanp4ihxw</a><br>
[6] <a href="http://markmail.org/message/jvk3jfpbsl3bhhs2">http://markmail.org/message/jvk3jfpbsl3bhhs2</a><br>[7] <a href="http://markmail.org/message/ckrhparroxvpibym">http://markmail.org/message/ckrhparroxvpibym</a></div>