<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Aug 6, 2013 at 5:53 PM, Matt Mackall <span dir="ltr"><<a href="mailto:mpm@selenic.com" target="_blank">mpm@selenic.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On Tue, 2013-08-06 at 15:04 -0500, Steve Borho wrote:<br>
><br>
><br>
><br>
> On Tue, Aug 6, 2013 at 2:47 PM, Matt Mackall <<a href="mailto:mpm@selenic.com">mpm@selenic.com</a>> wrote:<br>
> On Tue, 2013-08-06 at 14:15 -0500, Steve Borho wrote:<br>
> ><br>
> ><br>
> ><br>
> > On Tue, Aug 6, 2013 at 2:01 PM, Augie Fackler<br>
> <<a href="mailto:raf@durin42.com">raf@durin42.com</a>> wrote:<br>
> > On Tue, Aug 06, 2013 at 01:53:34PM -0500, Matt<br>
> Mackall wrote:<br>
> > > [this discussion needs to be cross-posted]<br>
> > ><br>
> > > We've got a few long-standing problems with our<br>
> current<br>
> > packaging:<br>
> > ><br>
> > > - lag official release<br>
> > > - not automated<br>
> > > - not reproduceable by third parties<br>
> > > - not signed<br>
> > > - lack nightly builds<br>
> > > - notable missing platforms (RHEL)<br>
> > > - not uniform in configuration<br>
> > ><br>
> > > Our attitude to date has been "well, we're not<br>
> really in<br>
> > charge of that,<br>
> > > that's done unofficially by volunteers". I think<br>
> we've<br>
> > reached a point<br>
> > > where this needs to start changing.<br>
> > ><br>
> > > In an ideal world, a release would work like this:<br>
> > ><br>
> > > - I tag the release<br>
> > > - I push it<br>
> > > - Automated builders kick off on a bunch of VMs or<br>
> hosts<br>
> > > - Builders run stock makefile targets or scripts<br>
> in tarball<br>
> > > - Packages get signed<br>
> > > - Packages get uploaded to one or more locations,<br>
> including<br>
> > > <a href="http://mercurial.selenic.com" target="_blank">mercurial.selenic.com</a><br>
> > > - Download links get updated<br>
> > ><br>
> > > Importantly, this same process would also be used<br>
> to produce<br>
> > automated<br>
> > > nightly builds for testing and bug reporting and<br>
> discovering<br>
> > packaging<br>
> > > problems before release dates.<br>
> > ><br>
> > > The job of packagers would thus shift from<br>
> "(manually?)<br>
> > building and<br>
> > > uploading binaries" to "making sure the committed<br>
> build<br>
> > rules are<br>
> > > correct and up-to-date".<br>
> > ><br>
> > > I think this also needs to be inclusive of builds<br>
> for<br>
> > packaging systems<br>
> > > (Debian, Ubuntu, RHEL, Centos, Solaris, Pypi,<br>
> etc.). These<br>
> > actually<br>
> > > suffer from more or less the same problems even<br>
> though the<br>
> > distribution<br>
> > > model is different. So we should be building<br>
> nightlies and<br>
> > release<br>
> > > builds for these systems and be more directly<br>
> responsible<br>
> > for quality<br>
> > > and timeliness here as well.<br>
> ><br>
> ><br>
> > This honestly sounds like a great improvement, given<br>
> how stale<br>
> > the<br>
> > packages for Ubuntu tend to be, and how some<br>
> packagers took it<br>
> > upon<br>
> > themselves in the past to enable all the extensions.<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > On the Windows side, everything is automated except for two<br>
> steps:<br>
> ><br>
> > 1 - signing packages requires a passphrase to be entered<br>
> > 2 - uploading is currently manually done through BB's clunky<br>
> web<br>
> > interface<br>
> ><br>
> ><br>
> > If we had an SCP or FTP URL to upload to, and a sane policy<br>
> for<br>
> > deleting old nightlies after a short while, it could be<br>
> automated.<br>
><br>
><br>
> Well, you already have SCP access, no?<br>
><br>
><br>
> Yep, it's just a question of whether you want nightlies in the same<br>
> location as releases, or in a different folder that was size-managed.<br>
><br>
> In addition to having a fully automated process that builds<br>
> releases and<br>
> nightlies, I think its also important to get to the build<br>
> script into<br>
> the tree so that people who need to build their own copies for<br>
> various<br>
> reasons (need bugfix, need custom deployment, St. Louis<br>
> trampled by<br>
> kaiju) can do so.<br>
><br>
><br>
> So.. building windows frozen packages is fairly convoluted (especially<br>
> for thg but the simple hg MSI is no cakewalk) and has a boatload of<br>
> prereqs to be installed.<br>
<br>
</div></div>Yep, I'm well aware. And this is actually MORE reason to insist on full<br>
automation. Let me back up and explain some of my motivation here:<br>
<br>
While I think you do a great job with packaging, it's lately become<br>
clear that package maintainers are a significant single point of<br>
failure. And we've also not coincidentally discovered that we don't<br>
actually know how some of our packages are actually built (like OS X)<br>
because the instructions don't work any more. So when the packager<br>
disappears, we've got several problems: find someone with the hardware,<br>
install the OS(es), install the deps, reverse-engineer the build<br>
process, hit unknown number of gotchas..<br>
<br>
The OS X situation is not as complex as Windows, but it's still bad<br>
enough that the current approach is clearly not good enough.<br>
<br>
And that's not the only problem in this area. For instance, we also have<br>
real uses for continuous builds that are not being met by monthly<br>
releases. I really want the BTS robot to point people at builds and say<br>
"try this and report back, please", but can't.<br>
<br>
(And it's also a perennial confusion that we announce releases before<br>
there are any packages built.)<br>
<br>
Fortunately, we know how to fix these sorts of things:<br>
<br>
- manifest the instructions and dependencies as build scripts<br>
- version control the build scripts in the canonical repo<br>
- use the build scripts, continuously and non-interactively<br>
- get people to use those builds and discover problems before releases<br>
- fix the problems found via check-in<br>
<br>
<br>
With my 'hackable' Windows builds, I've gone as far as scripting the<br>
download and install of the right versions of Python and gcc and all the<br>
other deps if not present to make sure its reproduceable, so I know this<br>
sort of thing can be done, even on Windows. I've obviously not yet gone<br>
as far as putting this knowledge in the repo, but I ought to.<br>
<br>
<br>
All that said, let's focus on phase 1: publishing nightly builds,<br>
somehow, somewhere.</blockquote><div><br></div><div>Angel has offered to build nightlies before (using an automated framework like Jenkins), he just needed a place to upload them. We just need to circle around and get that closed. The auto-builds might be unsigned at first.</div>
<div><br></div><div>As far as getting this into the tree and scripted.. I agree this would be astonishingly useful for the long run (it mostly frees me up as well) but my hg hacking time is extremely limited these days and I don't know when I'll get a chance to work on it.</div>
<div><br></div><div>One suggestion I have, if we try to automate this process in the hg/contrib folder, is that we make an attempt to drop the requirement on Microsoft's C compilers and try to use recent MinGW GCCs to compile the Explorer extension and the Mercurial (and other hg extension) Python extensions. We had problems with GCC some 5-6 years ago but perhaps those are all resolved now (that said, the Explorer extension hasn't changed in 19 months, we could probably check in pre-compiled versions in its repo).</div>
</div><div><br></div>-- <br>Steve Borho
</div></div>