Baking with Swarm


One of the challenges in game development is pre-computing static lighting. Emulating photons bouncing around in a virtual environment can require legions of compute cycles. This process is called baking. The end result is a lightmap, a compact data structure that represents the way static objects are lit.

The Unreal Engine suite provides software that makes baking lightmaps easy and fast. There are three components involved:

To build lighting, anybody running the Unreal Editor can kick off a bake by selecting Build Lighting in the editor. This results in a number of Swarm Agents across the local network participating in the build. The number is a function of the size and quality of the bake, and also depends on which Agents have free CPU cycles. Each selected Swarm Agents receives Lightmass along with enough data to run its portion of the bake. The combined results are ultimately assembled back into a complete lightmap.

By running Swarm Agents on many computers on your local network, lighting builds can be performed very quickly. What might take hours on a single machine takes only a few minutes with Swarm.

The documentation on Swarm and Lightmass is sparse. The best reference is a single Unreal AnswerHub post.

Where I work at HBO, only a subset of employees use Unreal. However, we want every machine on our network to participate in lighting builds — without the hassle of installing Unreal. That goal turned into an interesting adventure. Although Epic recommends copying the Engine\Binaries\DotNET folder to install Swarm, that alone is insufficient.

Without rehashing what the existing documentation already covers, I’m including everything I’ve learned about Swarm as of Unreal version 4.9. This information is useful to anybody setting up Swarm on your network or wanting to understand how to best configure Swarm.

Swarm Coordinator

The Coordinator keeps track of all of the Swarm Agents on the network and the status of each. You can also use the Coordinator to restart Swarm Agent instances.

Unreal Lightmass

Lightmass is the workhorse application that bakes lighting. It is automatically distributed to Swarm Agents on a per bake basis.

Swarm Agent

Swarm Agent is designed to run on multiple machines across your network, building lighting on demand.

This diagram shows a scenario where the PC in the bottom center has kicked off a Unreal lighting bake. Three PCs have Unreal installed, but all the PCs are running Swarm Agents. The top center PC is running the Swarm Coordinator. Five PCs were chosen to participate in the bake. The other PC was busy (compiling code, etc.). Communication between the local agent and the remote agents, represented by the arrows, happens in a peer-to-peer fashion.

Swarm Parameters

Many elements of how Swarm operates can be modified. The Settings tab in Swarm Agent allows you to change settings. Configuration information is stored in SwarmAgent.Options.xml and SwarmCoordinator.exe.config. Here are some of the most interesting elements, in order of importance.

Swarm Installer

To make life easier for our extended team, I wrote a Windows installer that handles the complexity described above. The installer uses the WiX Toolset to generate a standard Windows MSI file. Notes about the installer:

The Swarm requirements we publish to our team:

We recommend that everybody on the team with machines that meet the spec — including people that already have Unreal installed — install Swarm Agent.