WoW’s launch day approaches and EQ’s has passed. There’s endless debate about both products, and a lot of talk about how ready the two projects are, and how those launch days reflect their companies. I don’t buy it. Both are first-class studios with talent I respect greatly. Both will ship products that are quality, but will have launch issues. Saying there will be launch issues is kind of like saying that there will be a new car smell when you buy a car. The question is not whether there are issues, but how a team reacts to those issues.

All that being said, I thought I’d lift the hood and show what’s going on – from a developer’s viewpoint. I personally advocate that any design team for an MMO write a design doc for their launch day, because launch day has a set of concerns that are unusual.Here are some lessons I would impart about product launches, having been present for launches of both Meridian and Shadowbane.
1) Expect infrastructure to be overwhelmed. This mostly refers to the support systems – account creation/billing websites, account verification servers, patchers, even web forums. Two things make this happen – the first is that you just don’t have enough people in a beta to really hammer on them the way they are hammered in live, and the second is that these systems are typically designed for ‘normal’ traffic patterns, and launch day is the most abnormal day possible. Why does this always happen? Because it’s really hard to convince the guys with the money to buy lots of extra hardware and bandwidth for a one-day problem. It’s also hard to plan to buy extra hardware when there is always the risk your game will underperform.

2) Expect initial gameplay to be fairly lag-free – and degrade a few days later. Players always think that the game is working pretty well on launch day – lag is low, the mood is optimistic. Dev teams hang around all day, waiting for the other shoe to drop, but it usually doesn’t. Why? Because players can’t log in due to your infrastructure being overwhelmed. =)

3) Expect newbie content to be swamped. Again, this is a problem of abnormal traffic patterns – everybody is killing rats in newbieville, and every rat starts to be contested. Since everyone is in a low number of places, your server performance will seem extremely degraded – don’t worry, that will pass as players level up and spread out. It’s really hard to devote coder time to solving a one-week issue. In Shadowbane, we countered this with personnel – by derailing our planned events and having our Events staff spawn low level critters to kill.

4) Your plan to patch in that one remaining major system a month later won’t happen. Even if it’s almost done, and just needs more testing, your team will be far too busy fighting fires – that is, when they’re not drinking themselves in a stupor.

5) Expect hacks, crashes and severe server slowdowns – after about a week. There WILL be hacks, there WILL be crashes, there WILL be unexpected server bottlenecks. It’s not that we’re stupid, it’s that these things are insanely complicated to build. On the hacker front, no matter how smart your programmers are, they’re outnumbered by a couple of orders of magnitude, and forces of evil have found their best hacks in beta and are saving them for live. On the slowdown front, it’s just that no amount of load testing in beta can really approximate your full population of players playing the way the game was meant to be played. On the crash front, it’s that no amount of load testing in beta can really approximate your full population of players playing the game it WASN’T meant to be played.

If you’re really lucky, then nVidia and ATI will make your lives worse by releasing new drivers that your QA team doesn’t have time to test right before ship.

So the real question is not “Will there be problems” – there will be. The question is ‘How well will you deal with those problems?’. What’s your escalation path? How long does it take for a CSR to get a severe issue in front of a programmer’s eyes? Once you’ve got a fix lined up, how quickly and confidently can you code the fix, build it, QA it and deploy it? How are you managing the community in these times? And is your CSR staff prepared to ban the people that need to be banned from day one?

This is all complicated by the fact that our players are playing in elapsed time. An hour of play time feels like an eternity if you’re in the game, but it could take the dev team that long just to find a reproducible case of the bug in question. This is usually when producers tell their coders to not read the boards anymore – let the producers and community managers do it – for fear that the morale – which is already in the toilet – will flush itself. This is also about the time that first-time teams discover that perhaps it was unwise to plan vacations right after ship.