Single points of failure, leaving time for iterations and learnings from experts in the industry. Catch up on what we chatted about at Devcom.
Late last year, our CEO Sebastiaan Heijne hosted a fireside chat at Devcom. He was joined by Brian Jordan from Doborog and Stephen Clayburn from Bungie. The three discussed their experience developing multiplayer games and the pitfalls new developers can fall into.
Check out the full video. But here are the key takeaways and advice from their conversation.
It’s incredibly difficult to take a single-player game and try and make it work for multiplayer. Brian knows this from experience – as Doborog’s title, Clone in the Danger Zone, was originally just meant to be single player. Moving it over to multiplayer proved much more work than they originally thought it would be. But that doesn’t mean you shouldn’t try to do it at all.
“Looking back, it would have been much easier to start out building for multiplayer,” Brain said. “But it was a lot faster to prototype, figure out what was fun, and test it with players by starting out single-player. It’s not clear to say, had we started with multiplayer, if we would’ve ended up building out the game to the extent we did.”
Saying that, this approach can be quite expensive, Brian admitted. And considering multiplayer from the start is the approach most developers would want to take.
“Real multiplayer is a much larger investment in every single aspect. It’s harder to design, it’s harder to implement, it’s harder to iterate on, it takes more time,” Stephen said. “I’ve done it before on teams, [where we’ve said] we’re just going do single-player because it’s the fastest thing. And then we came to regret that later.”
One reason that it can be so difficult to switch from single-player to multiplayer is that not everybody will have a great internet connection. For example, your combat might feel great with zero lag, but with a few milliseconds, it suddenly breaks.
The first version of your game will never be perfect. This is especially true for multiplayer games, where there’s so much more to consider. Not only do you need to think about the game’s overall design and loop, but the backend systems.
“The thing that I’ve consistently found is making sure that you build in time for service iteration, because I’ve never successfully shipped the first version of a service,” Stephen said. “It was always the third iteration that stood the test of time and that we actually kept for a long time.”
This might be your matchmaker, your chat system or any other one of the moving cogs that make up a multiplayer game. These parts make up the experience, and it’s easy to forget to allow time for them – not just the game itself.
“We ended up having to take certain multiplayer-supporting web services in-house. At one point, we had to urgently write our own matchmaker and transition server hosting to a new service provider,” Brian added, explaining the types of surprises that can slow things down. “We didn’t expect that there would be so much future work cropping up post-launch. You’ve really got to budget in a lot of future time to maintain a multiplayer backend system.”
Nowadays, players almost take for granted being able to join a match with someone on a different console. But for the developer, it can be a hassle.
“Release cycles can become way less predictable – and longer,” Brian said, while talking about adding consoles into the mix. “You have to think very carefully: Okay, can we add this to every platform? Is it okay if a player is running an old client? And if not, how do we time the release so that it really upgrades all platforms at once, or at least makes it so clients can’t match with the wrong server version?”
And if you have a problem, some platforms might take quite some time to go back through the approval process. This can lead to a lot of versions, especially if you’re making a mobile version of your game, too.
“You have even less control when new versions of your product go out globally,” Stephen said. “You really have to be explicit because it’s very possible that you’ll have two or three or four versions out in production at the same time, because you can’t really control when either people upgrade or when the new version goes live.”
The solution? Use a routing service. Have a central version of the client that gets the requests from the players, which you can use to force players to upgrade to the newest version.
The biggest killer of a game is downtime. Downtime leads to bad reviews, and players can all too easily switch to another game. So it’s important to make sure that you’re aware of what services are critical to your business.
“What’s mission critical to your game?” Brian said. “So that players can sign in, so that players can pay you, so your players can play the game. There are a few mission-critical pieces that need to stay online.”
It’s important to make sure that you have good control over these parts. Other elements, like your chat, aren’t as important. It’s not a disaster if people can use voice chat for a day. But if they can’t buy packs or join a match? That’s going to hurt your bottom line. So decide what you own yourself. And which parts you can safely outsource.
This is called a single point of failure. And it’s extremely important to avoid having that single part that could completely topple your game.
“We definitely ended up getting bit by not owning our services end-to-end,” Brian said. “When there’s downtime that’s outside of your control, it’s really painful to sit around waiting for a fix, and harder to communicate the incident status to players.”
It’s much more frustrating to rely on a third party to fix a bug than it is to go into the code and make the changes yourself.
One way to make sure you have that control is to make sure you can easily change settings, without needing to release a new version.
“I remember when I was working on the Halo Wars backend,” Stephen said. “From the server config, we could actually change the variables that the Halo Wars client used when talking with the Xbox matchmaker. And it was super great – we could just go in, and within five or ten minutes, we would have changed how our matchmaking buckets within the Xbox matchmaker were actually matching people together.”
This meant that Halo Wars didn’t need to rely on title updates. Doborog has a similar philosophy – making sure that if the cloud provider for their matchmaker went down, they could point to a new one, using the same API.
“When writing code that’s connecting to web services, make as much configurable as possible or give yourself as many dials that’ll let you recover from issues, without requiring client updates,” Brian said.
Not everything needs to be outsourced, of course. If you have the tools in-house, you can fix problems without needing to rely on a third party. But to do that, you’ll need to sit down and discuss what you can practically do yourself.
“A lot of that core competencies conversation will come down to the expertise you have in-house,” Stephen said. “If you don’t have server developers, and you don’t want to hire server developers, you probably don’t want to make that a core competency of the studio. If you have people who are super excited about that, and you have that skill set in-house, that’s probably a good indication that you want to invest.”
So it all comes down to your people, their skills and what they’re interested in. But with a smaller team, you might have to learn and build these tools yourself.
“Initially, it was just a couple of us doing the development,” Brian said. “So we ended up having to build some multiplayer services ourselves due to provider circumstances. And I learned to love it.”
The gaming industry is seeing more and more services being turned into packaged products. For example, most people don’t bother building their own game engine – because Unity or Unreal’s are going to be far better than what you could create yourself.
“We’re getting to a point where with multiplayer games, game developers can go focus on the fun of the game, rather than having to focus on the really hard job of building tech,” Stephen explained. “We’re going to see even better games, because the tooling is going to be even better.”
As long as you have ways to configure your vital settings or can easily switch between providers, using third-parties can free up your time.
“It’s never been easier and less painful to create multiplayer games than right now,” Brian said. “And it’s only going to get better over the coming years.”
A single point of failure that developers often overlook is their server provider. Thankfully, with Gameye, our platform is completely agnostic. That means that you can hook up multiple server providers to our orchestrator. If one goes down, you can easily switch to another. So if you’d like to learn more, get in touch or join our Discord.
