Bots that play your server

Every new MU server has the same first evening. Somebody sets it up, posts the address, a handful of players log in, and each of them sees a Lorencia with two people standing in it. Most of them log out again, and the ones who stay are looking at an empty world - which is the one thing an MMORPG cannot be, no matter how correct its damage formulas are.

OpenMU can now populate itself. Since this summer the server can run bots: persistent characters which hunt, level up, spend their points, keep their buffs up, wear the gear they find, restock in town, group up, and log in and out over the day. The feature was built by nolt and is off by default - switching it on is a deliberate decision by the server admin.

I want to write about it because the design turned out more interesting than "fake players walking around", and because most of the hard parts had nothing to do with making a character move.

No client involved

Years ago I wrote about a demo console client - a program which connected to the server and spoke the protocol like a real client would. That is the obvious way to build a bot, and it's the wrong one for this purpose: a thousand of them means a thousand connections, a thousand encryption pipelines, and a thousand copies of a client's state, to simulate players who are already living in the server's own memory.

So a bot here isn't a client at all. It's an OfflinePlayer - the same class that keeps a character playing after its owner logs out, which is what MU Helper's offline mode does - with a navigator on top. No connection, no packets, nothing to encrypt.

Two ticks make up its mind. The offline helper AI runs twice a second and does the close-up work: attack, heal, buff, pick things up. The navigator runs once a second and decides the things an offline session never had to answer: where to hunt, when to travel, when to go shopping, whom to follow.

And bots are ordinary accounts. Each one is a real Account with an IsBot flag, its characters saved in the database like anyone else's, reloaded on the next start. A bot's progress belongs to the server's data, not to a process.

The rule which holds the whole thing together is this: a bot acts through the regular player actions. Moving an item, talking to a merchant, consuming a jewel, joining a party - all of it goes through the same code path with the same validations a client's packet would trigger. A bot cannot do anything a player couldn't, and when a rule changes, it applies to the bots for free. It's the same principle as the attribute system: don't build a second version of the game next to the game.

The interesting problems

Almost none of them are about walking or hitting things.

Everybody went to the same map. The obvious rule - always hunt the best map you can survive - turns every bot of a level band into the same answer, and leaves the rest of the band deserted. So the map is drawn instead of maximized: the best option still wins about a third of the picks, the runners-up split the rest, and the population spreads out the way players of that band would. Only maps which are actually an improvement take part in the draw.

A map can pass every check and still pay nothing. The monsters a bot is allowed to fight might be rare there, or other hunters get to them first. A bot notices this the way a player does - it hasn't landed a hit in minutes - and steps down one notch at a time until it finds something it can farm. The normal map choice carries it back up when its level and gear recover.

A mastered bot wants the weakest monsters, not the strongest. Master experience is only granted above a monster level threshold, and it barely grows with the monster's level after that. So the cheapest kill above the line is the best one - the opposite of what a bot does for the rest of its life. Those monsters still carry 40.000+ health, which is far beyond what a bot's found gear normally budgets for, so the hit budget is stretched for them. Its survivability isn't: a monster whose hits it can't take is still refused.

A veteran can own a skill it cannot cast. On a reset server, a reset keeps every learned skill but takes back the level which unlocked it. A character back at level 12 still owns Swell Life, which asks for 120. The game refuses such a cast silently - so a bot which kept trying would stand there forever, buffing something that never takes effect and never getting as far as attacking. Skills which can't currently be cast are passed over, in the attack rotation and in the buff rotation alike.

Being broke is a trap. A merchant trip only pays off if the bot can pay for something. A broke bot buys nothing, comes back as poor as it left, and would set out again instead of hunting - which is the only way it could have earned the money in the first place. Restocking therefore needs the means to pay, or loot worth selling once it's there.

A pet is not a stat. Plasma Storm draws its damage from the Fenrir, but the attribute behind it is derived from the character's own stats - nothing except the pet slot distinguishes a mounted character from one riding nothing. So pet skills are excluded unless the pet is really equipped, and the same goes for skills the game only activates during a castle siege.

My favourite of the lot is the PvP rule, because it's defensive in an unusual sense: a bot fights back only as far as the game's own rules allow - inside the self-defense window, or against someone already flagged as a killer. That means it can never be provoked into becoming an outlaw which players could then farm for free. It does remember who hit it, though, and a killed bot walks back to its killer to wait for a legal opening.

A population is a deployment question

The part I didn't expect to be interesting: bots count towards the player count of their game server, exactly like players do. And a server which reached its maximum player count turns new connections away. So a bot population large enough to fill a server would lock out the humans it was supposed to attract.

Hence a Bot capacity % - 60 by default - which is the share of a server's player limit its bots may occupy, with the rest reserved for players. And because OpenMU can run each game server as its own process, the split has to work without the servers negotiating: which accounts a server animates is a pure function of the account index and the set of configured game servers, so every server computes the same answer on its own. What doesn't fit stays offline until the deployment has room.

The cost, measured on a 12-core host: about 0.35 core and 760 MB for 250 bots, about 1.7 cores and 1.2 GiB for 1100. Generating a fresh population costs roughly a second per account - the password hashing dominates - and starting an existing population of 1100 takes some 15 seconds.

The honest part

Yes, this is a feature which can make a server look busier than it is. There's no point pretending otherwise, and it's why it ships disabled: turning it on is the admin's decision, not something OpenMU does behind anyone's back.

What surprised me is the other direction. A thousand characters playing at once is the load test nobody ever bothered to write, and it found real problems in the engine: neither MagicEffectsList nor ComposableAttribute is thread-safe, and a population that size runs into those races far more often than human players do - a few caught exceptions per minute. The bots deal with it pragmatically, by counting failed ticks and restarting themselves after twenty of them, which is what a player would do too. But the fix belongs in the engine, and now there's a reproducible way to hit it.

The limitations are documented rather than hidden. Bots never buy equipment - they wear what drops for them, so a bot at maximum level is weaker than a player of the same level would be. They do no quests and don't trade with players, the latter deliberately, because trading would be an abuse surface. Master skills which cost ten points at once are never learned, because a bot spends every point as it earns it.

Thanks

The bot feature is nolt's work - the initial one in July, and then a series of changes which turned a set of characters into something that behaves like a population. Eduardo added, among other things, the fresh-start option, which lets generated bots begin at level 1 with starter gear like a real new character instead of appearing with a random level and upgraded equipment.

Reviewing a feature of that size is its own kind of work, and most of what's in this post came out of that review going back and forth. The details above - the drawn map choice, the skill a reset character can't cast, the bot that can't afford to go shopping - are the ones which took the longest to get right, and they're the reason a bot is hard to tell apart from a quiet human player.