C#

Generating message structs by data

In this post I want to explain how I'm planning to generate ref structs in OpenMU and how this fits into the big picture.

Sven

• 4 min read

In my previous post I have presented my new idea of handling network packets using ref structs. In this post I want to explain how I'm planning to use them in OpenMU and how this fits into the big picture.

Packet structures as data

The first thing to do is building up something like a database of message struct definitions. I want to save the following stuff for each packet:

  • Struct Name
  • Descriptions ('sent when?', 'caused reactions?')
  • Header type (C1 etc.)
  • Code and SubCode, if applicable
  • Direction
  • Expected length, if known
  • Fields, for each field:
    • Index of the field in the struct
    • Size
    • Type (Integer, String, Enum etc.)
    • Byte order (endianness)
    • Name
    • Description
  • Enum Types, for each with their possible values, including names and descriptions. If an enum is used by more than one packet, an external definition should be possible.

As you see, I want to define more than just what's required to generate some struct code. I also want to be able to generate a documentation of all message structs. This also means, whenever a message gets extended or needs to be handled by the server, the corresponding data must be extended beforehand.

A similar, but incomplete collection of this data is already contained within the Network Analyzer project in some XML files. These files could be replaced with the new approach, once it's complete. How the new data is stored, isn't clear yet. I have some ideas, but it's too early to talk about it.

Generating and Compiling

The next step is generating the code out of the available information. The generated code could look like this:

/// <summary>
/// Is sent when:
///   The client opened an quest NPC dialog and decided to start an available quests.
/// Causes the following actions on the server side: 
///   The server decides if the character can start the quest. A character can run
///   up to 3 concurrent quests at a time.
/// </summary>
[SentWhen("The client opened an quest NPC dialog and decided to start an available quests.")]
[ReactionsOnServer("The server decides if the character can start the quest. A character can run up to 3 concurrent quests at a time.")]
[Direction(PacketDirection.ToServer)]
[Length(9)]
public ref struct QuestInitializationRequest
{
    public static byte Type => 0xC1;

    public static byte Length => 9;
    
    public static byte Code => 0xF6;
    
    public static byte SubCode => 0x0A;
    
    private static readonly byte QuestNumberIndex = 4;
    
    private static readonly byte QuestGroupIndex = 6;
    
    private static readonly byte UnknownFieldIndex = 8;

    private Span<byte> data;

    private QuestInitializationRequest(Span<byte> data)
    {
        if (data.Length < Length)
        {
            throw new ArgumentException($"Expected a span which is at least {Length} bytes long");
        }
        
        this.data = data;
        
        var header = this.Header;
        if (header.Type != Type)
        {
            throw new ArgumentException($"Wrong header type. Expected: {Type}, Actual: {header.Type});
        }
        if (header.Code != Code)
        {
            throw new ArgumentException($"Wrong header code. Expected: {Code}, Actual: {header.Code});
        }
        
        if (header.SubCode != SubCode)
        {
            throw new ArgumentException($"Wrong header sub code. Expected: {SubCode}, Actual: {header.SubCode});
        }
    }
    
    /// <summary>
    /// Gets or sets the header of this message.
    /// </summary>
    public C1HeaderWithSubCode Header
    {
        get => new C1HeaderWithSubCode(this.data); // this could be another ref struct
    }

    /// <summary>
    /// Gets or sets the number of the quest which should be initialized.
    /// </summary>
    [FieldDescription("The number of the quest which should be initialized.")
    [BigEndian]
    public ushort QuestNumber
    {
        get => this.data.GetWordBigEndian(QuestNumberIndex);
        set => this.data.SetWordBigEndian(QuestNumberIndex, value);
    }

    /// <summary>
    /// Gets or sets the group of the quest which should be initialized.
    /// </summary>
    [FieldDescription("The group of the quest which should be initialized.")
    [BigEndian]
    public ushort QuestGroup
    {
        get => this.data.GetWordBigEndian(QuestGroupIndex);
        set => this.data.SetWordBigEndian(QuestGroupIndex, value);
    }

    /// <summary>
    /// Gets or sets an unknown field.
    /// </summary>
    [FieldDescription("An unknown field")
    public byte UnknownField
    {
        get => this.data[UnknownFieldIndex];
        set => this.data[UnknownFieldIndex] = value;
    }

    /// <summary>
    /// Performs an implicit conversion from a Span of bytes to a <see cref="QuestInitializationRequest" />.
    /// </summary>
    /// <param name="packet">The packet.</param>
    /// <returns>
    /// The result of the conversion.
    /// </returns>
    public static implicit operator QuestInitializationRequest(Span<byte> packet) => new QuestInitializationRequest(packet);

    /// <summary>
    /// Performs an implicit conversion from <see cref="QuestInitializationRequest" /> to a span of bytes.
    /// </summary>
    /// <param name="packet">The packet.</param>
    /// <returns>
    /// The result of the conversion.
    /// </returns>
    public static implicit operator Span<byte>(QuestInitializationRequest packet) => packet.data;
    
    public static ThreadSafeWriter StartSafeWriting(IConnection connection, out QuestInitializationRequest message)
    {
        var writer = new ThreadSafeWriter(connection, Type, Length);
        var span = writer.Span;
        span[2] = Code
        span[3] = SubCode;
        message = span;
        return writer;
    }
}

Once this is compiled, we could push this to a NuGet repository. This could also be done automatically by the continuous integration build.

Workflow

So, when adding a new field or message we first have to get an updated NuGet package. The workflow would be as follows:

  1. Extend the data, push to Git
  2. CI will build and push it to the NuGet package repository
  3. Wait some minutes...
  4. Update the NuGet package reference in the consuming project (e.g. GameServer, Network.Analyzer)
  5. You're ready to use the new message structures

Usage

So, now we have a NuGet package which can be referenced by the GameServer and used in all of the packet handler and view plugins. I want to give a short example about how to use them.

Reading messages

public class QuestInitializationRequestHandler
{
    public void HandlePacket(RemotePlayer player, Span<byte> packet)
    {
        QuestInitializationRequest request = packet;

        // now you can work with the fields of the request:
        Console.WriteLine(request.QuestNumber);
    }
}

Writing messages

public class SomeClass
{
    public void SendRequest(IConnection connection)
    {
        using (var writer = QuestInitializationRequest.StartSafeWriting(connection, out var message))
        {
            message.QuestNumber = 1234;
            message.QuestGroup = 42;
            writer.Commit();
        }
    }
}

Why?

You might ask, why should I do that? Well, I see the following benefits:

  • A precise and consistent documentation is enforced, can be automatically generated and is even available in code.
  • Less errors when parsing and serializing messages, assuming the data is correct
  • We get a reusable message library (e.g. network analyzer, client simulator)
  • Writing code which writes code is always fun ;-)

The downsides are obviously:

  • Additional work. However, if we want a complete documentation about all message types, this kind of work is required anyway.
  • Additional build step. Because we'll reference a NuGet package for the messages, whenever we extend the messages we have to wait for the build to complete and for the published NuGet package update.

All in all, I think it would be worth the effort.

Versioning

As you might know, the OpenMU game server supports network protocols of multiple game client versions since a few weeks. The idea would be to create one struct definition per variant, like we also use different packet handler and view plugins in the game server.