Monday, December 31, 2007

Python Versus Erlang for an MMOG

As I mentioned in my previous post, I'm in the process of evaluating the core infrastructure for a massively multiplayer on-line game (MMOG).

The current technology contenders for the server are Stackless Python (and it's eventual successor PyPy) or Erlang.

In both cases, they provide for concurrent processing using the Actor Model, which we have already determined is our preferred development model (remember, there are no correct models, just more and less useful ones, and this is our useful model). Stackless and Erlang both provide native support for this model, using the tasklet in Stackless and the process in Erlang.

In terms of real-world implementations, Stackless is being successfully used by Eve Online for both the server and the client. Eve Online shares one important philosophy that we want to emulate as well: no sharding of the user population. Python is an incredibly efficient language to program in, sharing strong points of both object oriented and functional languages. Our programmers (me and two others) are also much more familiar with procedural, object oriented programming. All things being equal, we would certainly choose Stackless Python.

But all things aren't equal. Python suffers from a significant barrier in its scalability: the Global Interpreter Lock (the "GIL") prevents Python from scaling to multiple CPUs efficiently. Even without the GIL, multiple processors yield a significant complexity challenge as you deal with locks and concurrent threads.

In fact, that can be seen in Eve Online, as they struggle to deal with very large population centers overwhelming the ability to Stackless to scale horizontally. Stackless Python scales horizontally for multiple processes that don't share state (such as multiple zones in a game), but within a single state construct, you are limited to a single CPU.

This is where Erlang comes in. Built from the ground-up to support concurrency, both on a single machine and on multiple machines, it doesn't suffer from the state construct problems. Erlang is a functional language, and one of the implications of that is there is no state to be shared between processes. Processes can be placed "anywhere", and all state communication between processes is done via messages.

In terms of the real-world, which is an important consideration for me, Erlang came out of the Swedish telecom giant Ericsson as a solution for building telecom switches that required huge scalability and Nine "9's" of reliability. For myself, coming from a Java background, seeing what it takes to hit five "9's", that is a huge motivator. Another large example is Amazon's SimpleDB being written in Erlang.

In terms of gaming, Open Poker was recently re-written using Erlang and Vendetta Online rewrote their AI functionality using Erlang.

So, the downsides...
  • There isn't an established body of work to follow in the game

  • We don't have any experience writing systems using functional

  • There are few to no existing libraries to simplify our lives.

On the other hand, the upsides are significant...
  • Highly scalable across multiple processors and multiple
    systems, for free.

  • Ability to hot-swap code, to patch the system without shutting it

  • A database
    (Mnesia) that
    supports the highly concurrent, distributed environment that Erlang

In short, the benefits are all around the concurrency and
scalability. Personally, I want to see an environment where the
number of actors (people, NPCs, etc) is not arbitrarily limited.
Right now, I'm heavily leaning towards Erlang.


Anonymous said...

Good work. I have often looked at Erlang in building web services. I am going to look at it once again. I think you have found a role that Erlang can fill nicely, though.

Anonymous said...

It's Ericsson not Erickson.

njharman said...

Um, I really don't think Eve online is running on one cpu. Their game is also semi-sharded in that real-time play only exists between players in the same solar system of which there are thousands. Inter-system is limited to chat, market transactions, statistic reportings etc. 4-500 players in same system really, really lag it. Visit Jita sometime, Eve has a free two week trial

Eve regularly breaks their record of concurrent users. And from various dev posts it sounds like the bottlenecks are db related.

To answer your question you should use both. Erlang for the server / networking / "process" managment and Python for the logic / "missions" / whatever your game uses.

Also see

Travis Jensen said...

To be sure, Eve Online is not running on a single CPU. The point is that a single process in Stackless can only run a single CPU. This provides significant challenges in areas where large numbers of people congregate, since the process can't grow horizontally and the process can't be made any more fine-grain.

See, for instance, this discussion.

Similarly, the client suffers from the same challenges of single-threading.

Randito said...

Use the right tool for the right job.

Use Python (or some other scripting language) for your game logic. It's easy to develop in, and easy to understand. Your game logic will change a lot in the course of development; use a language that will support this.

For your server, for your thousand core giganormous box running your hundreds of virtual worlds, use Erlang.

It's not a true/false, if/else, black/white discussion. Use the right tool for the job.

asmodai said...

You are also aware of 'greenlet'?

Anonymous said...
This comment has been removed by a blog administrator.
momerath42 said...

I'm the principle Erlang dev for Vendetta Online. I did some prototyping in stackless years ago for another company (right around the time EVE started using it) and I've done a pretty full-scale actor-pattern game-mech system in common lisp. Erlang beats them both hands down, IMO. Mostly because of the good impedance match between that paradigm and Erlang (especially per-proc gc), but I'm coming to really love the language all around.

Contrary to your mention of our game, and probably some of our own news posts, Erlang is handling a lot more than AI on our server side. It's basically the new game-mech backbone, and over time we'll be moving everything but the most performance sensitive parts of our server side into it. Email me if you want to chat about your design; I might even be able to send you some useful code.

BTW, I'd recommend looking into QuickCheck; I wish I'd known about it *before* writing tens of thousands of lines of Erlang with no test coverage to speak of.

Richard Tew said...

It should be noted that when EVE started using Stackless, Stackless was very different. At the time it used continuations and actually aimed to be stack-less, through eschewing the stack for the heap as you would expect. However, time passed and it was rewritten. As such, opinions based experience when EVE started using it should be taken with a grain of salt. Which isn't to say they are necessarily wrong, but rather baseless as a blind measure of the current state or utility of Stackless Python.

Anonymous said...

I've created a threading framework for Python that implements erlang-style semantics using heavyweight python-processes, avoiding issues with the GIL: