Measure first
How OpenMU’s startup got faster: a measurement tool first, then compiled EF Core models and generated plugin proxies - and one line of SQL which was worth more than all of them.
The plan had five items, ordered by expected benefit. The biggest cost wasn't on it.
The plan
OpenMU did a lot of work at startup which could just as well be done at build time. It compiled proxies for its plugin points with Roslyn, scanned every assembly for plugins, let Entity Framework Core build its models, and loaded the configuration with a JSON converter that discovered every type by reflection. So I wrote an epic: replace runtime code generation and reflection with build-time generators. Faster startup, faster logins, less memory, and errors at compile time instead of at use.
It had five milestones, ordered by what I expected each to bring: plugin proxies and registries, generated JSON converters, compiled EF Core models, a batch loader instead of the big JSON query ("only if the measurements justify it"), and a handful of smaller reflection cleanups.
And above all of them, one rule: before the first change, measure.
The measurement tool
So the first pull request didn't make anything faster. It added a console tool to tests/MUnique.OpenMU.Measurements which measures plugin discovery and the runtime proxy compilation, the EF Core model building of every context, and the loading of the game configuration and the accounts - split into database planning, execution, reading the JSON and deserializing it.
Two details made it trustworthy. Every measurement runs in its own process, so the cold numbers include .NET's JIT and all first-time initialization, like a real server start. And the tool writes canonical JSON snapshots of the loaded configuration and accounts, with all collections sorted, so they are byte-identical across runs. Every later change had to reproduce those snapshots exactly.
The numbers are orders of magnitude, not decimals - the cold plugin registration took 2.2 s in one run and 3.9 s in the next. But orders of magnitude were enough.
The suspect nobody had on the list
The game configuration is loaded with a single JSON query: 46,105 characters of SQL, hundreds of small correlated subqueries, producing 21.6 MB of JSON. Warm, it took about 7.5 seconds to execute.
About 7.1 of those seconds were spent in PostgreSQL's JIT compiler.
PostgreSQL decides whether to compile a query to machine code with LLVM based on its cost estimate, and this query's estimate is enormous. So it compiled 1,038 functions - "Optimization 3818 ms, Emission 3135 ms" - to run a query whose actual execution takes about half a second. With jit = off, that's what it took, and only 0.6 s even cold, right after a PostgreSQL restart.
The obvious remedies didn't work:
- Prepared statements: PostgreSQL doesn't cache JIT-compiled code. Every execution of a prepared statement compiled everything again, about 8 seconds each.
- Table statistics: after data initialization, 58 of 111 tables had never been analyzed.
ANALYZEbrought the cost estimate down from 1.3 billion to 2.9 million - still far above the threshold of 500,000 for full optimization. - JIT without optimization and inlining: 1.1 to 1.7 s. Better, but still slower than no JIT at all.
JIT is a bet that the execution will pay back the compilation. For hundreds of tiny subqueries with little data each, it never does. The fix is one statement in front of the query, in the same command:
SET LOCAL jit = off;
Because the statements of one command run in one implicit transaction, the setting ends with the query - the pooled connection has JIT back for whatever runs next. The account queries were below the threshold all along and aren't affected. Warm, the configuration loading went from 7.8 to 2.1 seconds.
That one line was worth more than any milestone of the plan. None of them would have found it.
Then the plan, with numbers
With the baseline in place, the milestones went in one after another, each pull request carrying its own before-and-after table from the same tool:
| Change | Before | After |
|---|---|---|
| Plugin proxies generated at compile time, first plugin manager | 1.5-2.4 s, 26 MB | 0.16-0.18 s, 4 MB |
| Compiled EF Core models, main context incl. EF startup | 2.8-4.0 s | 0.67 s |
| Compiled EF Core models, all main contexts | ~290 MB allocated | ~4.5 MB |
| Compiled models for the 8 typed contexts of the servers | 3.8-4.5 s, 437 MB | 37-47 ms, 0.8 MB |
| Game configuration in one select per table instead of one JSON document, cold | 3.4-3.6 s, 124 MB | 1.7-1.9 s, 42 MB |
| Generated mapping code instead of Mapster, first conversion | ~990 ms | ~180 ms |
| Generated JSON converters, configuration deserialization cold | ~1,730 ms | ~1,530 ms |
The batch loader, the milestone which should only happen if the measurements justified it, did get justified - though only after the JIT fix had made the JSON query fast enough to compare fairly. The accounts kept their JSON query; at 15-30 ms per login it wasn't worth touching. In the server log, the host now starts in about 1.5 seconds instead of 2.9.
The last row is the humbling one. The generated JSON converters were the second item of the plan, ordered by expected benefit, and they brought about 12 percent on one step. The reflection converter had already been compiling expressions; what remains of the cold cost is mostly .NET compiling the code on first use, which no source generator removes. A plan ordered by gut feeling is a list of suspects, not a list of culprits.
A side effect I didn't plan for
Generating the plugin proxies at compile time made the main test project run in 43 seconds instead of 3 minutes and 34. Its tests create plugin managers all the time, and every one of them used to compile its proxies with Roslyn again.
Startup costs are not only paid by the server. They're paid by every test run, every debug session, every CI build - and those happen far more often than server starts.
Generated first, the old way as fallback
None of the generators replaced the old code outright. Each one is tried first, and the reflection path stays as the fallback:
- A plugin point without a compile-time proxy - from an assembly that wasn't built with the generator - still gets one generated with Roslyn at runtime.
- The JSON converter factory uses the generated converters and falls back to the reflection converter for any type they don't cover.
- A test registers the plugins once through the generated registries and once by reflection, and the two plugin managers have to end up identical.
That has a price which I want to be honest about: Roslyn is still a runtime dependency. The epic hoped to drop it, and it didn't - it's just not on the normal path anymore. I'd rather have a fallback that is slow than a plugin that silently stops working.
How it was built
Sixteen pull requests, in two days, stacked on top of each other - written with an AI coding agent, one branch per step. That pace is exactly why the "measure first" rule mattered more than usual. When changes arrive faster than you can form an intuition about them, the measurement tool and the byte-identical snapshots are the referee: every pull request had to show its numbers, and every one had to load the same configuration as before.
What I'd tell anyone with a slow startup
Write the measurement before the fix, and make it measure the cold path the way production sees it - separate processes, real database, default settings. Keep a snapshot of the result so a faster version can prove it's also a correct one. And expect the biggest cost to be somewhere you didn't look: in this case not in my code at all, but in a database deciding that a query was worth seven seconds of compilation.