MU Online

On item duplication exploits and how to prevent them

Sven

• 7 min read

Hello,

first of all, if you're Webzen or a private MU Online server owner or developer, don't take this post as a threat. These item duplication examples which I will describe are known to me since many years. As you might imagine a lot of other people may know this issues already, too. So, if your server still has no fix for them, I'm very sorry.

I'm trying to describe how item duplicate bugs worked at the original MU Online server, how they can prevented and how the OpenMU project tries to do that. As data coming from the game client shouldn't be trusted, all solutions I describe here should be implemented server-side, of course.

Basics

What are item duplicates and how are they used

It's probably the best known problem in the world of MMORPGs. Probably every MMORPG has items which are particularly valuable, so cheaters try to duplicate them to gain an advantage in the game over other players. Either to use them themselves, sell it to other players for in-game currencies or even to sell them for real money on the black market.
In MU Online, these exploits were also used to upgrade or convert items without risk to lose them. For example, if you have some rare ingredients to create new rare and valuable items by using chaos machine combinations, you could create these endlessly.

How are item duplicates created

Basically, every item duplication trick which I know depends on the timing of saving (item) data. Additionally, there are no optimistic or pessimistic lock strategies which could detect if there was data saved for the same account before from another session. So, if you know when a server saves data, you can figure out how to exploit certain game actions.

Item persistence on the original server

On the original server, when a player succeeds of getting an item duped in a non-persistent state, it can be easily turned to be persistent. As items are saved in big binary fields, their "serial numbers" are not automatically indexed or checked for uniqueness by the database. Instead, these checks have to be done at application level... which might have loopholes.
It's also possible to scan the whole database for duplicated items periodically, but that's very time- and resource-consuming. Also this shouldn't run while the server is running, because the state in game servers might not be completely persistent. Otherwise, you may face false-positives, e.g. Player A is still saved with an item which somehow moved to Player B, but only the state of Player B is saved yet after the movement.
Because of this delay, cheaters could have enough time to clean up traces of their duplicates, e.g. selling them to an NPC merchant or washing jewels with the Lahap NPC. Item duplicate bugs are still useful for cheaters even if they can just keep one copy of an item: They can duplicate items as backup before trying to upgrade them (fails might destroy an item) or using them to create rare items in the chaos machine.

Dupe method examples

Logging in at the same time with the same account

In a very early version of Mu Online, even before Atlans map arrived at GMO, you could log into an account twice at the same time if you did that synchronously enough. Because this bug is so old, it's probably fixed in all versions of Mu Online.
It's the bug which brought all the maximized Excellent Brass Sets to Wigle and Maya realms. When logged in multiple times with the same account, you could do different actions to get duped items in the end:

  • Trade the same item to two different characters.
  • Trade just to one different character, then disconnecting the account which traded, first.
Possible Solutions:
  1. Thread-safe login server or session manager. Obviously, a thread-safe data structure or process is needed to flag an account to be logged-in. And at login stage, this state needs to be checked in a safe manner. An account should only be able to be logged in once at the same time and after all other previous session states are saved.
  2. Usage of optimistic or pessimistic lock strategies.
Using Lahap and Trade

This one is an exception - it's not based on timing of item saving, but on a fact about trading which is like an invitation to cheaters: Cancelling a trade resets the state of the whole inventory and it's items to the previous state - as you will see, only half-hearted.
Back then, hackers opened a trade with some other character and at the same time used the Lahaps functions. The Lahap NPC is the guy which is able to pack and unpack jewels to save some inventory space. For example, instead of 30 single Jewel of Bless, you could pack it into one item. And to be able to use them later, it must able to unpack them again.
The server didn't check if you had the Lahap NPC opened when you used these functions. So what the hackers did:

  • First, having 10, 20 or 30 unpacked jewels in inventory
  • Opening a trade with another character
  • Sending a data packet to pack these jewels
  • Sending a trade cancel packet
  • Then you'd still have both, the packed and the unpacked jewels.
  • (... repeat the steps until the inventory is full ...)

The server obviously didn't expect that you gained items during trade and didn't remove them from its memory, when cancelling the trade ;-)
This bug led to an insane amount of jewels on the market of global mu online - regular Items were being sold for tens of thousands of jewels.
You could now argue, Webzen (or K2 Network) could've just deleted these illegally created jewels - but that's not easy. When packing/unpacking, each jewel gets a new item Id assigned. You can't tell (without detailed logs) which ones were created illegally and which not. Another point is, this Lahap functions can be used to "wash" duplicated jewels when they were duplicated by other ways, because each "created" jewel or pack gets a new Id by it.

Possible Solutions:
  1. Allow Lahaps functions only when the Lahap NPC is opened.
  2. Prevent opening NPC dialogs and trades to other players at the same time.
  3. The inventory reset when cancelling a trade must be implemented correctly.
  4. Keep track of the Ids of the packed jewels and reassign them after unpacking - of course that's a bit more complex to do.
Forced automatic server changes

I don't know if this is still working in GMO (I hope not :D), but it did at least until the end of the Season 6 era. And this issue is/was huge, really huge.
When moving between some specific maps, the server urges the client to connect to another game server, e.g. when you move from Lorencia to Loren Market and vice-versa. The server saves the players data to the database after it has been disconnected. When connecting to the other game server, it loads this data from the database as well. Usually, the game client closes the first connection, before connecting to the new game server. But when you keep the first connection open as long as you can, there might be enough time to trade away some items (or do other things...) and to disconnect before the first connection closes which causes the old data to be written to the database. Example:

  • Player A moves from Map Lorencia to Loren Market
    • The server sends a server change request with an authentication token to the players game client over connection 1.
    • The player keeps the connection 1 open (e.g. with firewall rules or a proxy program).
    • The game client connects to the server of Loren Market using the authentication token, we call it connection 2
  • Player A trades an item to Player B
  • Player A leaves the game, so that connection 2 closes.
  • Player A closes connection 1 or waits until the server closes it by itself

Can you spot the issue? Because the server saves the account data after a connection closes, the data saved by closing connection 2 is getting overwritten by the data saved by closing of connection 1. Items which got traded exist now at the account data of Player A and Player B.

Solutions:

Again, you could fix this issue in several ways:

  1. Send out the authentication token AFTER the data got saved on the database by the first game server
  2. Right after sending the authentication token, close the connection
  3. Usage of optimistic or pessimistic lock strategies

Conclusion and general solutions

As you can see, the original Mu Online server is or was pretty vulnerable when it comes to item duplications. I think that's because of its whole design and how it's saving item data. I don't think these were all existing issues here, there might be dozens of others...

The OpenMU project tries to be less vulnerable to these kind of issues. I try to describe, how.

Item table and IDs

We try to prevent persistent duplications by saving items in an own table, where each item has its own row which is identifiable by a GUID as primary key. Therefore, there can only be one item with the same Id and only assigned to one account. So from a persistence point of view, we're safe. Cheaters would have a lot less time to clean up their traces of non-persistent duplications.

Thread-safe Login Server

Of course, this project has a thread-safe login server. One account can only be connected to one server at the same time. Of course, setting an account offline is done after saving the data.

No forced automatic server changes through the client

To minimize the attack vector, we just don't support this feature. Even if we need to scale-out to separate game servers for certain maps, we'll keep the initial connection and manage this all internally. See also https://gameserverarchitecture.com/2015/10/pattern-distributed-network-connections/.

State machines

Each player holds a thread-safe state machine which allows or prevents the transition to a requested state. For example, when opening a Lahap NPC dialog it requests to change the state to PlayerState.NpcDialogOpened. The state machine only allows it if there is no trade open and the current state is PlayerState.EnteredWorld.

To do: Item history log

It would be nice to have some kind of log about each item, so you could find out easily if an item was duplicated or generated in an illegal way.
There could be automatisms which could check the plausibility of an item history, so manual interaction is not required.
For example, when selling an item to a NPC merchant or using it in a chaos combination, the server could check if the item was sold or destroyed before.

To do: Reproducible Ids for generated items

Generated items (e.g. by chaos machine) with the same identifiable valuable ingredients should get the same Id assigned. So when cheaters use the same ingredients over and over to create new items out of them, the resulting items should have the same Id. This way, the database could instantly tell us if something is fishy with a particular item when the session is saved. When multiple valuable ingredients are used, a hash value of all of their Ids could be built, otherwise you could just use the Id of the most valuable ingredients.