Monday, December 31, 2007

Python Versus Erlang for an MMOG

As I mentioned in my previous post, I'm in the process of evaluating the core infrastructure for a massively multiplayer on-line game (MMOG).

The current technology contenders for the server are Stackless Python (and it's eventual successor PyPy) or Erlang.

In both cases, they provide for concurrent processing using the Actor Model, which we have already determined is our preferred development model (remember, there are no correct models, just more and less useful ones, and this is our useful model). Stackless and Erlang both provide native support for this model, using the tasklet in Stackless and the process in Erlang.

In terms of real-world implementations, Stackless is being successfully used by Eve Online for both the server and the client. Eve Online shares one important philosophy that we want to emulate as well: no sharding of the user population. Python is an incredibly efficient language to program in, sharing strong points of both object oriented and functional languages. Our programmers (me and two others) are also much more familiar with procedural, object oriented programming. All things being equal, we would certainly choose Stackless Python.

But all things aren't equal. Python suffers from a significant barrier in its scalability: the Global Interpreter Lock (the "GIL") prevents Python from scaling to multiple CPUs efficiently. Even without the GIL, multiple processors yield a significant complexity challenge as you deal with locks and concurrent threads.

In fact, that can be seen in Eve Online, as they struggle to deal with very large population centers overwhelming the ability to Stackless to scale horizontally. Stackless Python scales horizontally for multiple processes that don't share state (such as multiple zones in a game), but within a single state construct, you are limited to a single CPU.

This is where Erlang comes in. Built from the ground-up to support concurrency, both on a single machine and on multiple machines, it doesn't suffer from the state construct problems. Erlang is a functional language, and one of the implications of that is there is no state to be shared between processes. Processes can be placed "anywhere", and all state communication between processes is done via messages.

In terms of the real-world, which is an important consideration for me, Erlang came out of the Swedish telecom giant Ericsson as a solution for building telecom switches that required huge scalability and Nine "9's" of reliability. For myself, coming from a Java background, seeing what it takes to hit five "9's", that is a huge motivator. Another large example is Amazon's SimpleDB being written in Erlang.

In terms of gaming, Open Poker was recently re-written using Erlang and Vendetta Online rewrote their AI functionality using Erlang.

So, the downsides...
  • There isn't an established body of work to follow in the game

  • We don't have any experience writing systems using functional

  • There are few to no existing libraries to simplify our lives.

On the other hand, the upsides are significant...
  • Highly scalable across multiple processors and multiple
    systems, for free.

  • Ability to hot-swap code, to patch the system without shutting it

  • A database
    (Mnesia) that
    supports the highly concurrent, distributed environment that Erlang

In short, the benefits are all around the concurrency and
scalability. Personally, I want to see an environment where the
number of actors (people, NPCs, etc) is not arbitrarily limited.
Right now, I'm heavily leaning towards Erlang.

Wednesday, December 26, 2007

Building a Massively Multiplayer Online Game

I've been interested in the game industry for years, both in terms of playing as well as building. The challenges associated with building a game are significantly different than those associated with the normal day job, and the opportunity to provide something that others enjoy certainly appeals to my creative side.

In the early 1990s, I was introduced to the MUD and spent far too much time playing there and far too little time on my computer science studies. Since that time, bringing together a large group of people into a single virtual community has been the only logical direction for games to go (In my not-so-humble opinion :), and that has been realized over the last few years with many exceedingly good massively multiplayer online games (MMOGs) being published.

Of course, the developer in me has always thought that I could do it better. So I am now putting my time (since I'm short on the money) where my mouth is. A group of four of us has started the process of incubating a MMOG, starting with base ideas and features we'd like to see.

Being the type of person who likes to here myself wax prolific, I will be blogging about the experience here. As this venture is intended to be commercial in nature, I won't be giving details about anything that would fall into the "competitive advantage" category, but as there are many different aspect to building a highly scalable game that have nothing to do with the competitive advantage aspect, there should be lots to discuss.

As the guy who is architecting this, there are a lot of decisions that need to be made early on: what technologies do we use, how do we segregate processes, how do we enable virtually unlimited scaling, etc. This is a case where the naive solution will absolutely not work.

So, over the next year or more, expect to see a lot of entries as I discuss trade-offs and decisions. It's going to be a lot of fun for me; hopefully it will be useful for you.

Monday, December 17, 2007

Festering Pustuals

As I ponder more and more about the issues faces with Software Development, I can't help but agree with this quote. I think more damaging is that there is only a small percentage of people who even recognize that sickness abounds...

We know about as much about software quality problems as they knew about the Black Plague in the 1600s. We've seen the victims' agonies and helped burn the corpses. We don't know what causes it; we don't really know if there is only one disease. We just suffer -- and keep pouring our sewage into our water supply.

-- Tom Van Vleck

Thursday, December 6, 2007

Being a Craftsperson or a Laborer

I'm coming to realize that the world of software development is very similar to the world of construction. Given that particular comparison, it is obvious that it probably isn't a very compelling comparison, either.

There are a lot of vocational programmers out there, keeping the wheels of business information running. Business needs those legions of framers, plumbers, electricians, and drywallers to keep systems talking to each other. And lest I come off even more arrogant, some of them are amazingly good at what they do. But, just like in construction, these people are not, generally, paid to innovate and their working environment is never optimal.

Joel Spolsky said it well:
...if you’re not very, very careful when you graduate, you might find yourself working on in-house software, by accident, and let me tell you, it can drain the life out of you.

The next level comes in software companies around the world. Life is usually better here, though the developers may or may not be. The work is usually more interesting because, to a greater or lesser degree, innovation is what allows that business to stay in business. Unfortunately, most companies, over a period of time, lose the need and ability to innovate: The need because the rely on it being too expensive for other businesses to change from their software and the ability because process and formality overwhelms spontaneity and creativity.

These are your commercial builders. Their scale is usually larger, the complexity of the result is much greater. There are a few, very good people leading things up with a mass got it, framers, plumbers, electricians, and drywallers.

Then there is the top tier. These are the best companies to work for. Innovation isn't something that is talked about in meetings as a "goal", then promptly followed up with enough crap to stifle any desire to do anything; innovation is the norm. Management exists to ensure that technologists get what they need to do their job. Customers look at the results and say "Wow." Businesses choose to change to them, because not changing will cost them more.

They are places where you can write about your job like this:
And with that, I'm going to get back to my "day job" project(s), which are sadly confidential. But they're so cool that if I did tell you about them, you'd be so overwhelmed that you'd have to go sit in one of our $5000-ish massage chairs, like the dude sitting next to me right now.

Not surprisingly, the percentage of really good people who work in the top tier is much higher. The top-tier survives at the top because of that talent (not just in development, mind you). These are the master craftsmen, brought in to do the finish work on one-of-a-kind structures around the world.

I love crafting software. I love software development. I am, however, getting tired of needing to unplug the blocked sewage because the company won't invest in the business it is in. I've heard and tried to buy into the argument "if you cut investment, you'll make more profit", but those companies with $5000-ish massage chairs are proving otherwise.

Monday, December 3, 2007

Laptops for Developers

All developers should have laptops and all conference rooms should have projectors. Honestly, I'm surprised there are software companies out there that think otherwise, but I just found that out to be the case today.

  1. Given that work areas are so open, it is really obnoxious for a few people to gather around one person's monitors for anything. It is distracting to everybody else around them.
  2. Code reviews are much easier with a projector and a white-board. Switching between code modules while doing chalk-talk is inefficient using paper. Code review notes belong in the code, not on a piece of paper that will get lost.
  3. Your good developers will spend time outside of work doing "geek" stuff. You encourage that to be work related if you provide the environment.
  4. At some point, almost every developer will find that they are traveling, be it for a conference, an emergency customer issue, or to train a development group in Indiana. You've made them 100% inefficient by no providing a laptop.
  5. Needing to be 100% at the office to "work" is anachronistic, especially for development. Providing a mobile development environment means a developer can work productively for 3 hours at home before his dentist appointment instead of taking 1/2 day off of work. That means better productivity and happier employees, since they don't have to waste leave for life-maintenance.
The days of "the laptop is too slow" are gone. So are the days of "the laptop is too expensive." The only thing preventing laptops from being used in place of desktops is attitude, and that attitude costs real dollars.