Mastodon

Working with Adastral

Something fast has been brewing...

January 26, 2025


to my employer reading this blog post, all of my work relating to this blog post was not done on company time and hardware

For the past 6-12 months I've been working more with Adastral Group on a backend management system for the Adastral Launcher. Recently, I have reached the point where said management system is stable enough for a public1 release. With that being said, I have the opportunity to discuss some cool stuff that myself (and the team) have been working on in the shadows.

Under the hood, Adastral mainly depends on butler2, which is the patching software made by the developers at itch.io. The founder of Adastral, intcoms, has already mentioned some optimizations that he has made to butler in his own blog post3, but I've been working on something that will make Adastral much better than any other sourcemod launcher in the long-term (and software distribution methods in general).

Intcoms found multiple issues with butler that make it unnecessarily slow for our use case, which is for large-ish game updated (e.g; Open Fortress v19 to v21). This is frustrating when we are testing our software, and when the end-user is installing or updating their game. For context, most of our Open Fortress players usually have relatively low-spec computers when compared to players of most AAA games, since that is what usually attracts people to playing some Source Engine mods.

There are numerous issues with our current usage of butler, which are:

  • Slow patch generation time
  • Slow patch apply time (compared to Steam)
  • Large patch size
  • Previous versions need to be fully extracted into a directory4

Intcoms has fixed some of these issues by replacing the hashing algorithm that's currently used (md5) with highwayhash, which has significantly increased the patch time, and generation time. But this isn't enough for me; I want it to be faster, I want the files to be smaller, I want it to be the best.

(re)Discovery

Around a year and a bit ago, I was going through a manic episode due to some mental health issues, and I decided that I would write a competitor to Steam and Microsoft's Software Delivery/Deployment feature in SCCM. I wrote up a patching system that was (at the time) the fastest solution I was aware of, including a whole-ass backend to manage it, and I believe I lost it, or so I thought. After digging through my computer for a few hours, I found a couple of screenshots and notes about how my implementation worked.

It took about 4 weeks of tinkering and trying my best to remember what I did to create a proof of concept program that was able to generate, apply, and verify updates with my own format.

How it works

This patching library/method/system, called Northam (because of Adastral's naming scheme5), is about 12x-17x faster than Steam when I benchmarked it recently with the new Factorio update. (From build 153976926 to 161818327)

Steam took 3min 35s for the "Updating", "Patching", and "Installing" stages (as reported by the status text below the "Manage Downloads" button). Northam took 2.59s to verify the game files, and 9.86s to apply said patch. The download size for Northam was ~1.3gb (which included deltas, new files, and files to delete), and Steam's reported download size was ~1.1gb (with a ~200mb .delta file).

Northam uses a modified version of the rdiff algorithm that is used in rsync and depends heavily on C#'s Parallel.ForEach method to ensure that multithreaded systems can benefit greatly8 by using Northam instead of a different patching library. The chunk size that is being used for diffing files is 256kb and the patch/signature files are binary formatted to ensure that it can be read as fast as possible (which takes ~800ms for a 164mb file).

Not everything will be revealed on how Northam works under the hood since it's currently faster than anything we're currently using (and Steam)

Benchmarks

System Specifications for Benchmarks

Factorio

Build 153976926 to 161818327

Files changed: 12,370

Size Difference: +200mb

Steam Butler Northam (v3.2)
Download Size 1.1GB 1.2GB 1.3GB
Generate Patch 37.2s 28.62s
Verify (before) 1-2s 5.19s 2.213s
Verify (after) 1-2s 5.39s 2.59s
Apply 3min 35s 17.18s 9.86s
Signature Size (before) 1.28MB 26.6MB
Signature Size (after) 1.31MB 28.5MB
Reproducing Test Results

Terminal Recordings;

Downloads

Counter-Strike 2

Build 163338619 1643226510

Files changed: 136

Size Difference: -3.65GB

Steam Butler Northam (v3.2)
Download Size 13.75GB 9.19GB 10.1GB
Generate Patch 27min 15s 3min 31s
Verify (before) 2min 3s 1min 32s 36.432s
Verify (after) 2min 9s 1min 36s 32.998s
Apply 11min 55s 2min 58s 44.388s
Signature Size (before) 20.8MB 545.5MB
Signature Size (after) 21.9MB 573.4MB
Reproducing Test Results

Downloads

Terminal Recordings

Open Fortress

Alpha v19 to v21

Files changed: 2,260

Size Difference: -12.3MB

Butler (Modified) Butler Northam (v3.2)
Download Size 174mb 174mb 161.7MB
Generate Patch 30s 1min 12s 13.864s
Verify (before) 4.7s
Verify (after) 4.98s
Apply 2s 4s 1.297s

Note: Unable to test w/ Steam since OF doesn't have it's own AppID & I don't want to break my partner agreement.

Reproducing Test Results

Downloads

Terminal Recordings

Notes

  • This blog post was first created on the 14th of November 2024.
  • Unable to provide the "Generate Patch" metrics for Steam, since recreating that could be slightly different to the actual time it took for the developers.
  • Steam applies game patches to a staging directory while the update is downloading. Due to this feature, the "Apply" metric may be unreliable. This mainly impacts games that don't use a custom "bundle" file format (like VPKs).
  • The script that was used in the Northam v3.2 recordings can be found here. Requires PowerShell.

Remarks

If you want to know details about Adastral-related releases, you can join the discord server which is available on the Adastral Website, or you can follow the GitHub Organization and it's respective repositories.