API to help writing Mercurial servers

Apr 26, 2011 at 8:44 PM

This may be outside the scope of this project, but I'm curious how hard to would be to support APIs that help writing Mercurial servers. For git, GitSharp has classes like GitSharp.Core.Transport.UploadPack and ReceivePack that support this. e.g. see the GitDotAspx project for an example that uses it to write a little MVC based git server with not much effort.

I'm guessing the main difficulty is that Mercurial.Net in just a wrapper for hg.exe, and hg.exe doesn't directly offer this functionality. Well, it does have the 'hg serve' command but that creates its own all-in-one HTTP listener which would not easily integrate into an existing web app.

Apr 26, 2011 at 9:04 PM

I have been considering the same thoughts.

Not that I have any immediate plans to create a new Mercurial web server, I'm quite happy with the servers I already use (CodePlex, BitBucket, and Kiln), but I have been considering what it would take.

This is a rambling post, please bear with me.

Unfortunately, there are two things that I think make this particular project unsuitable for this purpose:

  1. Overhead
  2. It's a client wrapper, as you've already surmised

The overhead of calling down to the client isn't insubstantial. It might be OK for a single client program, and in particular since I built in full support for doing things asynchronously, so that you can for instance kick off a push in the background and be notified when it has completed. However, I'm not sure it would make for a good experience on a server system.

Secondly, as you say, the client executable doesn't really have any provisions of providing pull/push as a server, except for the "hg serve" part, and, again, as you said, it wouldn't be easy to integrate that with the rest of the server.

Or would it?

Now, OK, let's assume that the overhead can be mitigated. You could ensure that the Mercurial executable was always cached in memory, and though there is some overhead in spinning up the Python interpreter part for it, let's assume we can live with that. You build a fast multi-core server, and it might be more than enough. I'm still pretty sure that a pure Python front-end would have less overhead, but let's be serious, who of us wants to build *that*? :)

What *could* be done to the server part is that you host the web server interface, and the Mercurial push/pull interface separate. I don't know if the "hg serve" server will reload the configuration files in flight, but if it could do that, I could see spinning up a hg serve on a custom port on the same server being done easily.

Though, if you were to do that, why not just host hgweb directly. CodePlex already does this, they have a very thin layer on top of Mercurial, in fact, in my opinion they have a fairly inferior layer on top of Mercurial, since they seem to be branch agnostic. Everything is laid out as a sequential history, even if the changesets are on different branches, named and anonymous alike.

So, perhaps the idea is not so ludicrous after all. *if* the overhead is acceptable. Also, since the overhead would be localized to the end-user experience, and you could very well cache a lot of that, it might actually be more than enough.

And then you would use hgweb to do everything related to the Mercurial protocol. You would automate building configuration files for hgweb and spinning up websites/sub-domains for that. There wouldn't be much more overhead with that than CodePlex already has, and I gotta say, that part works flawlessly on CodePlex.

I know that's not what you asked, but I think that would make for the best experience. Then you would focus on the part where hgweb doesn't shine, providing a nice end-user experience, and let hgweb focus on the things where it *does* shine, which is talking to the client.

It wouldn't, however, be a simple MVC project any more, setting up such a server would be a far larger ordeal.

But, in my opinion, it might just be worth it.

It is certainly worth a try.

Don't you think?

Apr 27, 2011 at 5:41 AM

About implementing a solution based on 'hg serve': one concern is that the feature was (probably) built with the goal to make it easy for developers to pull/push from each other, and may not be scalable enough to run a high traffic Mercurial server.

So yes, maybe running hgweb is in fact the best approach, though it is indeed different from the GitSharp based solution. Jonathan Wanagel (dev manager on Codeplex) has some good steps on setting that up on this thread.

The part that I'm less clear on is the 'layer on top of Mercurial' that they have: what technology are they using for that? Presumably, they are just using hg.exe over the same repositories managed by hgweb. And of course, whenever you use hg.exe you can potentially use Mercurial.net instead :)

So all in all, is what you're suggesting basically the same thing as what Codeplex does, or something a bit different?

Apr 27, 2011 at 6:56 AM

It was in fact something like CodePlex, but just a full open source project you could host on your own servers.

So you could set up your own, internal, project hosting site for your company.

One that would make it easy to configure new projects, have RSS feeds, set up access rights, etc. Basically automate as much grunt-work as possible.

Apr 27, 2011 at 10:16 PM

Yep, I can see that making sense. Maybe the first step is to just have the front end that doesn't deal with the hgserve functionality, and that would be fully built on mercurial.net. Basically, it would be an MVC app like Bonobo is for git that implements some bitbucket/gihub kind of clone. But it would not in itself deal with the server piece.

Then eventually you could envision an all-in-one set up that installs Python, hgserver, etc... and makes it really easy to set up the end to end thing.

Jul 8, 2011 at 6:30 PM

There are some good news as of lately.

Version 1.9 of Mercurial was released with a "command server" mode, which means you can spawn a hg.exe instance that can be communicated with over the standard input/output. The instance will not shut down or exit after executing a command, which means you can issue many commands against it.

This is all relatively new, for all I know there could be bugs in that implementation as well, and I definitely have not implemented anything in Mercurial.Net to support this yet, but the good news is *that I will* implement this support.

This should reduce the overhead when dealing with local repositories, and might be beneficial if a server-type application ever becomes a real project.

Jul 8, 2011 at 6:34 PM

Interesting! Here is the link to more info for reference: http://mercurial.selenic.com/wiki/CommandServer

Jul 8, 2011 at 6:48 PM

Yep, I have already set up a simple LINQPad program that executes Mercurial with that server and communicates with it. Not only does it look completely doable, it also looks like the overhead is zilch, nada, none. There is the same overhead as before, and probably slightly more, when spinning up the initial instance, but if you keep the instance running, there should be next to none overhead compared to, say, executing your code as a Python program instead, calling the methods directly.

I need to refactor the current code, it uses a static class to spawn off hg.exe right now, so here's my current plan:

  1. Convert that static class to an instance-class, and rewrite all the code so that it works with it
  2. Wrap it in an interface, and again rewrite all the code that depends on it (not that much, just a few methods)
  3. Create a separate class that implements this interface, that uses the new command server mode
  4. Make Repository IDisposable (this is the biggest breaking thing in this change)
  5. Add selective code that determines if the new command server mode can be used, there's some rules:
    1. Mercurial 1.9 or higher must be the one being used (obviously)
    2. You need to have an actual repository. The Repository class can be constructed on top of a regular folder, in case you want to execute "hg init", the command server mode requires the repository already present. I will make the necessary changes so that the instance is spawned after you execute "hg init" though, if the rest of the requirements are present
    3. You're executing repository-related commands, some commands such as "hg version" etc. can't be executed through this mode apparently, but I will test all the commands to determine which works and which doesn't

All of these changes are slated for 2.0-release of Mercurial.Net, hopefully this will be at or right after the time of Mercurial 1.9.1 (ie. around August 1st.)

Jul 8, 2011 at 7:17 PM

Just making sure I'm understanding all this correctly: what you describe is a more efficient way of implementing the Mercurial.Net client API by keeping a single long running hg.exe instance for each Repository object. This makes a lot of sense!

But it is not something directly related to implementing a Mercurial server, for which the old 'hg serve' does the job.


Jul 8, 2011 at 7:20 PM

You understood it correctly.

For an internal server, it might be feasible to spin up hg.exe instances for repositories, for instance to spin them up when people are using a repository, and spin them down after some idle-time, but for a public server it would kill the server.

In other words, I'm just leaving all doors open, but for client-access to a local repository from an application, the new command server mode is really going to become handy.

I will add enough overloads or methods to be able to selectively choose whether to use the command server mode, but first I need to add support for it.

I will keep you posted.

Aug 30, 2011 at 5:48 AM

Sounds cool !

Would I be able to write two programs that communicate peer-to-peer, working on a shared / synced set of files using mercurialnet using the change you describe ?


Aug 30, 2011 at 6:54 AM

Well, not exactly. Those two programs would only talk to their local mercurial instance, handling a local repository. While you could ask the clients to push and pull between themselves, you could easily build a peer-to-peer communication platform, but since you would need to handle conflicts etc. it would be quite a lot of work to build such a program.

Note that you will not get the two mercurial instances to *talk to each other*.

Aug 31, 2011 at 12:30 PM


Okay, thanks a lot !