Tell Me About Yourself

I’ve been interviewing a lot of software engineers lately. I typically start my screen with a question like “Give me a two minute overview of your career.” The goal is to give the candidate some time to settle their nerves and learn something interesting about them at the same time.

I’m surprised how rare it is for candidates to stick to that “two minute overview” instruction. Five minutes isn’t uncommon, at which point I cut them off and move on to the next question, while writing down something in my notes about “doesn’t follow directions.”

I’m also surprised the way candidates interpret the question. Maybe it’s just nerves, but even experienced candidates don’t take advantage of the open-ended nature of the question. The typical answer is a historical recap. “I graduated with a degree in CS. My first job was at Amazing, Inc, where I worked on algorithms for efficient cargo ship loading. Then I moved on to ….”

I saw all of this on your resume already. This is your opportunity to go deeper. Here’s what I really want to know:

  • What 3-4 skills do you have that I most care about?
  • What are you really, really good at?
  • What gets you up in the morning excited to come to work?
  • What makes you special or even unique among candidates?

The best candidates will talk about their gaps, too. Examples:

  • “I’ve always wanted to work on machine learning, but I haven’t had the chance yet. I’ve been reading all the research.”
  • “I led a team, but what I learned is that I’m better at writing code.”
  • “I listed C# on my resume, but I haven’t used it three years. My best programming language is C.”

The next time you’re interviewing and you hear anything like “tell me about yourself” or “what’s your background” or “give me an overview of your career,” the interviewer is giving you chance to go much deeper than surface level. Take advantage.

Posted in C++, Computers and Internet, Management | Leave a comment

Quantum Management

At the recommendation of my current manager, Laura, I read this article from James Everingham, the head of engineering at Instagram. It’s an accurate description of my management philosophy. The gist is that managing by interjecting your views into the conversation typically results in non-optimal solutions. In the framework of quantum physics, you kill the cat. A better way to manage is to frame problems and define ambitious endpoints. Then let the team figure out the best way to get to the endpoint. Assuming you’ve followed another one of my principles — hire people smarter than you — you end up with an optimal solution.


Posted in Computers and Internet, Management | Leave a comment

References For The Win

I love single character code changes that generate massive performance improvements.

Unreal has had static code analysis passes not too long ago. It’s surprising that SA didn’t identify this issue. It’s also unlikely that programmer code reviews would have detected something as tiny as a missing ampersand.

Just another reminder of how difficult it is to identify performance problems.

Posted in C++, Computers and Internet, Game Programming, Unreal | Leave a comment

July 4, 1977

It was the summer before 4th grade. Jimmy Carter was president. Apple Computer had just incorporated, and the Atari 2600 video game console had been released, but personal computers were still a novelty. I wouldn’t use a PC for many more years.

I was a voracious reader who loved anything involving knights or pirates. I spent hours building castles and sailing ships with Legos. I watched Star Trek episodes, of course, but science fiction wasn’t really that compelling. The world of Star Trek was too obviously fake.

We were visiting my aunt and uncle in Eugene, Oregon. On the 4th of July, we all went to the movie theater for a matinee. It was a new movie everybody was talking about. The theater was packed. We split up to find seats. I sat next to my mom on the left hand side near the front.


Ten words in blue appeared: “A long time ago in a galaxy far, far away….”

Then that opening brass fanfare and the now famous title crawl began. I was transfixed. Mesmerized. Spellbound.

Fantastic images filled my impressionable brain. The unbelievably massive Imperial Star Destroyer chasing Princess Leia’s tiny Rebel Cruiser. Then robots. Now rebel troops. Stormtroopers. Blasters. Incredible sound effects and soundtrack.

Then the gong and ominous musical pause. Darth Vader. Hell, yeah!

In those five minutes, my life changed. I had never seen anything like this. Even watching that opening sequence 38 years later, I’m still hypnotized. I challenge you to find a better movie opening.

To a nine year old in 1977, the world of Star Wars was absolutely real. Watch the droids in that opening scene on the Rebel Cruiser. C3PO’s shiny skin is scuffed and marred. R2D2 is downright dirty. All those little details convinced me — and millions of others — that Luke and Leia and Ben and Han actually existed in some galaxy far away.

That summer, I started reading science fiction. I started building X-Wings and Star Destroyers and TIE Fighters. My bedroom was plastered with photos and drawings and newspaper clippings of anything and everything Star Wars. And when we finally got our first PC, I started programming (text-based) Star Wars simulations.

Which ultimately led me to programming jobs, then video game jobs, then Xbox, and now virtual reality.

George Lucas gets a bad rap these days, but I for one thank him and the entire original Star Wars production team. You transported this fan into a life and career he never would have foreseen on July 3, 1977.


Posted in Computers and Internet, Game Programming, VR | Tagged | Leave a comment

Baking with Swarm

One of the challenges in game development is pre-computing static lighting. Emulating photons bouncing around in a virtual environment can require legions of compute cycles. This process is called baking. The end result is a lightmap, a compact data structure that represents the way static objects are lit.

The Unreal Engine suite provides software that makes baking lightmaps easy and fast. There are three components involved:

  • Unreal Lightmass: the app that does the actual lighting computations. It’s designed to run on many different PCs simultaneously.
  • Swarm Agent: a background app that runs on any PC that has been designated to bake lighting. The Swarm Agent invokes Lightmass to do bakes.
  • Swarm Coordinator: an app that runs on a single always-on PC that receives requests for lighting builds and determines which Swarm Agents are available to participate in the bake.

To build lighting, anybody running the Unreal Editor can kick off a bake by selecting Build Lighting in the editor. This results in a number of Swarm Agents across the local network participating in the build. The number is a function of the size and quality of the bake, and also depends on which Agents have free CPU cycles. Each selected Swarm Agents receives Lightmass along with enough data to run its portion of the bake. The combined results are ultimately assembled back into a complete lightmap.

By running Swarm Agents on many computers on your local network, lighting builds can be performed very quickly. What might take hours on a single machine takes only a few minutes with Swarm.

The documentation on Swarm and Lightmass is sparse. The best reference is a single Unreal AnswerHub post.

Where I work at HBO, only a subset of employees use Unreal. However, we want every machine on our network to participate in lighting builds — without the hassle of installing Unreal. That goal turned into an interesting adventure. Although Epic recommends copying the Engine\Binaries\DotNET folder to install Swarm, that alone is insufficient.

Without rehashing what the existing documentation already covers, I’m including everything I’ve learned about Swarm as of Unreal version 4.9. This information is useful to anybody setting up Swarm on your network or wanting to understand how to best configure Swarm.

Swarm Coordinator

The Coordinator keeps track of all of the Swarm Agents on the network and the status of each. You can also use the Coordinator to restart Swarm Agent instances.

  • SwarmCoordinator.exe is designed to run on a single machine on your network. We run it on a low-end always-on Windows box. We also run Swarm Agent on the same machine.
  • Swarm Coordinator does not require that Unreal Engine be installed. I didn’t do the original setup of the Coordinator on our network, but based on empirical evidence, it requires the same files as the Swarm Agent (see below), plus the SwarmCoordinator.exe.config file.
  • We place a shortcut to SwarmCoordinator.exe in the Windows Startup folder so that the Coordinator always begins at system start.
  • In theory you could have multiple Coordinators on the network, each managing an independent set of machines/Agents. We haven’t tested this theory.

Unreal Lightmass

Lightmass is the workhorse application that bakes lighting. It is automatically distributed to Swarm Agents on a per bake basis.

  • Do not install UnrealLightmass.exe directly; it’s automatically copied and run whenever a bake is kicked off.
  • Because Lightmass is completely independent of Swarm, Swarm Agents can run different versions of Lightmass. That means you can have teams running different versions of Unreal and using different versions of Swarm Agents to bake lighting. Nice.
  • Lightmass will utilize multiple CPU cores (you can configure how many) and peg the CPU on most machines. However, because (by default) it is run at below normal priority, it doesn’t normally impact the productivity of anybody actually using the machine.
  • Lightmass is RAM intensive. It is not uncommon for it to use 1GB or more RAM depending on the bake size/quality.
  • Lightmass does not utilize the GPU (as of 4.9).
  • Depending on the size/quality of your bakes, Lightmass can require significant hard drive space. For instance, for our current small project, Lightmass uses many hundreds of MB per lighting build. See notes on configuring the Swarm cache for more details.
  • Lightmass source code is located at Engine\Source\Editor\UnrealEd\Private\Lightmass and Engine\Source\Programs\UnrealLightmass
  • The version of UnrealLightmass.exe that an Agent receives is dependent on processor architecture. There are 32 and 64-bit versions. Epic recommends only using Swarm/Lightmass on 64-bit systems for best performance, and we follow this advice.
  • Lightmass requires the following be installed in order to run. If you’re installing Swarm Agents on machines, you’ll need to be sure these components are also installed.
    • DirectX v9 extensions, specifically D3DX9_43.dll, DComp.dll and XInput1_3.dll
    • Microsoft C runtime for Visual Studio 2013, specifically MSVCP120.dll and MSVCR120.dll

Swarm Agent

Swarm Agent is designed to run on multiple machines across your network, building lighting on demand.

  • SwarmAgent.exe runs on Windows XP and up. It does not run on MacOS or Linux (as of 4.9).
  • Swarm is built with C# using the .NET 4.0 framework
  • SwarmAgent.exe and its associated DLLs are 32-bit images
  • We place a shortcut to SwarmAgent.exe in the Windows Startup folder so that the Agent always begins at system start.
  • Swarm can technically run a variety of distributed tasks, but as of 4.9 it only runs lighting calculations by distributing and spawning Lightmass
  • There are two types of logical Swarm Agents. The same SwarmAgent.exe client supports both modes.
    • The Local agent is the agent that kicked off the lighting bake
    • Remote agents running on other network PCs are assigned to participate in a given lighting bake
  • After initial communication with the Coordinator, Swarm Agents appear to perform all Agent-to-Agent communication in a peer-to-peer fashion.
  • The Local agent distributes UnrealLightmass.exe and lighting information to each assigned Remote agent. The Local agent also participates in the bake (unless otherwise configured), collects the work from remote agents and merges the results.
  • Swarm will not execute bakes on machines where it detects that the CPU usage has been, on average, above 20% for the past 10 seconds (see CPUBusyThreshold in Agent.cs)
  • Swarm spawns the Lightmass process at BELOW_NORMAL_PRIORITY_CLASS on the Local agent and IDLE_PRIORITY_CLASS on Remote agents to reduce the CPU impact on busy machines. These values are configurable.
  • Swarm will not necessarily use all available non-busy agents. The number of agents appears to depend on the size and complexity of the lighting bake.
  • Swarm does not require that Unreal Engine be installed.
  • Swarm requires the following files be present in the same folder as SwarmAgent.exe. These files are found in the Engine/Binaries/DotNET folder.
    • SwarmAgent.exe
    • AgentInterface.dll
    • SwarmCommonUtils.dll
    • SwarmCoordinatorInterface.dll
    • UnrealControls.dll
    • SwarmAgent.Options.xml
  • Swarm requires that the .NET 4.0 framework (or greater) be installed. If you’re running on Windows 8 or better, it’s already present. On Windows 7, it’s not preloaded, but comes automatically with Windows Update.
  • Although Swarm itself doesn’t require DirectX v9 extensions or the C runtime, Lightmass does need these, so if you’re installing Swarm, be sure to install those components, too.
  • Swarm Agent runs as a Notification (or System Tray) app. It is designed to run continuously in the background.
  • To view the Swarm Agent interface, locate the app in the system tray (yellow and black S icon) and double click the icon.
  • SwarmAgent.exe does not respond to WM_CLOSE. To close it manually, you must choose File->Exit from the main menu. To close it programatically, you can run taskkill.exe SwarmAgent.exe.
  • Empirically (using Microsoft Message Analyzer) SwarmAgent and SwarmCoordinator use the following protocols and ports:
    • ICMP protocol for ping requests between agents
    • TCP protocol for all other communication, using ports: 8008, 8009, 54430, 56574, 56587 and 56589
  • Some ports are configurable. For instance, SwarmCoordinator.exe.config allow you to remap port 8009.
  • Swarm will not activate on sleeping PCs. I recommend that if you always want your Swarm Agents to be available, those PCs should not enter power-saving sleep states. Alternatively, you can use Windows Task Scheduler to wake machines at the appropriate times to run your bakes. See powercfg command line tool for details. We’ve found that not all machines have BIOS configurations that support powercfg, so your mileage may vary.
  • Until we explored the issue in detail, the most common Swarm problem was blocked communication between different Swarm agents. Firewalls are the culprits. Windows Firewall will block Swarm by default, even if the user chooses to allow Swarm to do outbound communications. Because Swarm communicates in a peer-to-peer fashion, it must also be configured to allow inbound communication from other agents.
  • Programmatically, the Windows Firewall can be configured to permit Swarm usage with the following commands. We also do IP filtering to make sure these firewall exceptions only happen on our local network — replace x.y with your actual leading IP octets.
    • Allow Swarm ICMP pings: netsh advfirewall firewall add rule name=”Unreal Swarm” dir=in action=allow enable=yes remoteip=x.y.0.0-x.y.255.255 protocol=icmpv4 interfacetype=lan
    • Allow Swarm TCP commands: netsh advfirewall firewall add rule name=”Unreal Swarm” dir=in action=allow enable=yes remoteip=x.y.0.0-x.y.255.255 protocol=tcp interfacetype=lan localport=8008,8009,54430,56574,56587 remoteport=8008,8009,54430,56574,56587
    • Uninstall firewall exceptions: netsh advfirewall firewall delete rule name=”Unreal Swarm”

Swarm DiagramThis diagram shows a scenario where the PC in the bottom center has kicked off a Unreal lighting bake. Three PCs have Unreal installed, but all the PCs are running Swarm Agents. The top center PC is running the Swarm Coordinator. Five PCs were chosen to participate in the bake. The other PC was busy (compiling code, etc.). Communication between the local agent and the remote agents, represented by the arrows, happens in a peer-to-peer fashion.

Swarm Parameters

Many elements of how Swarm operates can be modified. The Settings tab in Swarm Agent allows you to change settings. Configuration information is stored in SwarmAgent.Options.xml and SwarmCoordinator.exe.config. Here are some of the most interesting elements, in order of importance.

  • CoordinatorRemotingHost: must match the Swarm Coordinator Computer name from Control Panel->System
  • CacheFolder: where Swarm caches job information. At install time, we set this to the equivalent of %temp%/SwarmCache
  • MaximumCacheSize: the maximum amount of HDD space that Swarm will consume in GB. We use 10 (the default).
  • MaximumJobsToKeep: the maximum number of cached jobs (and logs) to keep around. We use 5 (the default), but it’s extremely rare that we need to look at logs of old jobs, so I think 1 or 2 would be fine.
  • AllowedRemoteAgentNames: allows you to select a subset of Remote agents that you want to use. We always want to use all available, so we set this to *
  • AllowedRemoteAgentGroup: allows different PCs to participate in different Swarm groups. We currently have a single group with our team name.
  • AgentGroupName: this is the name of the group that the local agent belongs to. We use our team name.
  • LocalJobsDefaultProcessorCount and RemoteJobsDefaultProcessorCount (developer settings): the number of CPU cores used by Lightmass for local and remote bakes. These are set automatically to reasonable values, but can be reduced to lessen CPU impact at the cost of bake speed.

Swarm Installer

To make life easier for our extended team, I wrote a Windows installer that handles the complexity described above. The installer uses the WiX Toolset to generate a standard Windows MSI file. Notes about the installer:

  • The installer is a 64-bit package. This addresses: 1) Unreal’s recommendation that Lightmass run on 64-bit systems only, 2) a limitation of WiX that doesn’t allow 32-bit packages to include/install 64-bit binaries, plus 3) the fact that everybody in our office is running 64-bit Windows anyway 🙂
  • The following DirectX v9 files must be installed: dxsetup.exe,, dsetup.dll, dsetup32.dll, and You can grab these files from the DirectX SDK. The WiX toolset page has good instructions for installing DX extensions.
  • In addition to the normal installer-y stuff (copying files and updating the registry), the installer:
  • For uninstalls, in addition to nuking the right files, the installer:

The Swarm requirements we publish to our team:

  • 64-bit Windows 7 or above
  • 4+ GB RAM
  • 10+ GB free hard drive space

We recommend that everybody on the team with machines that meet the spec — including people that already have Unreal installed — install Swarm Agent.


Posted in Game Programming, Unreal | Tagged , , | 4 Comments

Windows Installer: Removing Folders

Recently I put together a Windows Installer package (MSI file) for our team to use internally.

Without you having to do any extra work, the Windows Uninstaller automatically takes care of deleting any folders where it installed content. However, if an app writes to other folders that the installer doesn’t know about, it’s your responsibility to handle the removal of those folders.

In my case, the app that I was installing writes temporary files to a cache folder. The app could cache gigabytes of data, so it was key that the uninstall remove all those temporary files. Fortunately, WiX makes this easy. Here are the steps. First, we search the registry for the location of the cache folder and store it in a Property. We’ll use this if we’re uninstalling. Replace CompanyName and AppName with your company and app names.

<?define AppRegKey = "Software\CompanyName\AppName" ?>
<Property Id="CACHEFOLDER">
 <RegistrySearch Key="$(var.AppRegKey)" Root="HKCU" Type="raw"
 Id="CacheFolderRegSearch" Name="CachePath" />

Second, in the case we’re installing, we need to save the location of the cache folder in the registry, so that we can invoke the “RemoveFolderEx” command on that folder during a future uninstall.

<?define AppCacheFolder = "AppCache" ?>
<Component Id="CacheCleanup" Guid="*">
 <RegistryValue Root="HKCU" Key="$(var.AppRegKey)"
 Name="CachePath" Type="string" Value="[TempFolder]$(var.AppCacheFolder)" KeyPath="yes" />
 <util:RemoveFolderEx On="uninstall" Property="CACHEFOLDER"/>

We need to reference the component we’ve registered in a Feature block, along with other relevant components.

<Feature Id="MainApplication" Title="App Name" Level="1">
 <ComponentRef Id="OtherComponents" />
 <ComponentRef Id="CacheCleanup" />

To make this all work correctly, there are two more steps. First, ensure your WSX source references the utility extension where util:RemoveFolderEx is located.

<?xml version="1.0" encoding="UTF-8"?>
<Wix xmlns = ""
     xmlns:util = "">

And second, reference the extension when you compile and link:

candle source.wxs -ext WixUtilExtension.dll
light source.wixobj -ext WixUtilExtension.dll

That’s it. Now the installation will correctly store a registry reference to the cache folder and clean up the cache folder on uninstall.

Posted in Computers and Internet, Installers, Microsoft, WiX | Leave a comment

Windows Installer: Firewall Settings

Recently I put together a Windows Installer package (MSI file) for our team to use internally.

The app I was installing communicates with other apps on our internal network, and it requires certain network ports be open for ICMP and TCP communication.

My first thought on solving this issue was to figure out where Windows Firewall Manager stores its firewall settings and then tell the firewall to read the new settings. Indeed, I found the location in the registry and I was able to successfully create the new settings. I was also able to kick the firewall using the commands “net stop “windows firewall”” followed immediately by “net start “windows firewall””. Everything seemed great.

But there were issues. For instance, during install, a little message showed up in the taskbar that the firewall was stopping. That’s confusing to the end user, especially because there’s no message that shows up saying the firewall is starting back up. Modifying the registry directly felt dirty. This was not a maintainable solution. The whole approach was a hack.

So I found a better way: the “netsh advfirewall” command. It’s available on Windows 7 and up.

Say you wanted to open ports 8008 and 8009 for inbound TCP traffic on the local network. Here’s the full command to add that rule:

netsh advfirewall firewall add rule name="My Firewall Rule" dir=in action=allow enable=yes protocol=tcp interfacetype=lan localport=8008,8009 remoteport=8008,8009

Even better, the firewall automatically starts applying this rule without a restart. There are many other advfirewall parameters, but you can explore those on your own.

Running these commands in the context of the installer was a little more challenging. I needed to add multiple rules, and I needed to handle removing the rules on uninstall. Let’s start with the variables I used to simplify things:

<?define Quote = """ ?>
<?define IpRange = "w.x.y.z-w.x.y.z" ?>
<?define OpenPorts = "8008,8009" ?>
<?define FirewallRuleName = "$(var.Quote)Open 8008 and 8009 for MyProject$(var.Quote)"?>
<?define FirewallAddRule = "advfirewall firewall add rule name=$(var.FirewallRuleName) dir=in action=allow enable=yes" ?>
<?define FirewallRuleICMP = "remoteip=$(var.IpRange) protocol=icmpv4 interfacetype=lan" ?>
<?define FirewallRuleTCP = "remoteip=$(var.IpRange) protocol=tcp interfacetype=lan localport=$(var.OpenPorts) remoteport=$(var.OpenPorts)" ?>
<?define FirewallRemoveRule = "advfirewall firewall delete rule name=$(var.FirewallRuleName)" ?>

The Quote variable handles the cases where we need to embed quotes in the command string. The IpRange variable locks this rule down to a specific set of IPs. Fill in your own values for w.x.y.z. Like many other advfirewall commands, it’s optional. FirewallAddRule has the meat of the command string, and then there are individual suffixes for the ICMP and TCP rules. The ICMP rule doesn’t accept ports. The FirewallRemoveRule ensures that all rules matching the given name will be removed on uninstall.

Here’s the code for executing these rules. First, for install:

<CustomAction Id="SetUpdateFirewallICMPCmd" Property="UpdateFirewallICMP" Execute="immediate"
 Value="&quot;netsh&quot; $(var.FirewallAddRule) $(var.FirewallRuleICMP)" />
<CustomAction Id="UpdateFirewallICMP" BinaryKey="WixCA" DllEntry="CAQuietExec" Execute="deferred"
 Return="ignore" Impersonate="no" />
<CustomAction Id="SetUpdateFirewallTCPCmd" Property="UpdateFirewallTCP" Execute="immediate"
 Value="&quot;netsh&quot; $(var.FirewallAddRule) $(var.FirewallRuleTCP)" />
<CustomAction Id="UpdateFirewallTCP" BinaryKey="WixCA" DllEntry="CAQuietExec" Execute="deferred"
 Return="ignore" Impersonate="no" />
<Custom Action="SetUpdateFirewallICMPCmd" Before="InstallFinalize"><![CDATA[NOT REMOVE]]></Custom>
<Custom Action="UpdateFirewallICMP" After="SetUpdateFirewallICMPCmd"><![CDATA[NOT REMOVE]]></Custom>
<Custom Action="SetUpdateFirewallTCPCmd" After="UpdateFirewallICMP"><![CDATA[NOT REMOVE]]></Custom>
<Custom Action="UpdateFirewallTCP" After="SetUpdateFirewallTCPCmd"><![CDATA[NOT REMOVE]]><Custom>

There are four custom actions that execute in sequence after the InstallFinalize stage. The first action establishes the command line for adding the ICMP rule using the variables defined above. CAQuietExec requires that the actual app command (netsh in this case) be quoted. The second action executes the command line established in the first command. Using multiple CAQuietExecs requires configuring the command line as a property with a matching ID in the following action. Changing firewall settings requires Windows elevated privilege, so we use deferred execution with no impersonation. The final two commands set the TCP rule in the same manner.

The funky notation ![CDATA[NOT REMOVE]] tells the installer that these commands should run whenever we’re not actually uninstalling the product (e.g. installs, reinstalls, upgrades, repairs).

Here’s uninstall:

<CustomAction Id="SetUpdateFirewallRemoveCmd" Property="UpdateFirewallRemove" Execute="immediate"
 Value="&quot;netsh&quot; $(var.FirewallRemoveRule)" />
<CustomAction Id="UpdateFirewallRemove" BinaryKey="WixCA" DllEntry="CAQuietExec" Execute="deferred"
 Return="ignore" Impersonate="no" />
<Custom Action="SetUpdateFirewallRemoveCmd" Before="InstallFinalize">
<Custom Action="UpdateFirewallRemove" After="SetUpdateFirewallRemoveCmd">

We set up the command line to remove the firewall settings, then invoke that command line with elevated privileges. We only remove the firewall settings when the product is being uninstalled (not installed, reinstalled, or upgraded)

One additional note. I really wanted to lock down the firewall rule to a specific app, so I was excited that advfirewall has just the option I was looking for:

allowedprogram program=C:\MyApp\MyApp.exe name="My Application"

But unfortunately in my particular case the app I was installing is not actually the app that does all the LAN communication. My app spawns other apps that do the communication, and the locations and names of those apps is not necessarily known at install time. Hence we chose to lock down the firewall rules to specific (small) range of IPs, so that if this install package gets out in the wild it’s not unnecessarily introducing security holes.


Posted in Computers and Internet, Installers, Microsoft, WiX | Leave a comment