A Tale of two Apps

by
Photo: Dulcey Lima via Unsplash

In my favorite ECS, Bevy you usually have one App with potentially many SubApps that run your systems. Bevy will make sure to spread your systems across all your cores for best parallelization via a thread pool. For building a game (or other software) in which two things happen at very different and independent pacing this may not be enough and you may actually need two Apps. A prime example of this is a simulation in game in which the simulation runs at some framerate that is slower than that of your monitor and a frontend that should still react snappy to button presses and play smooth animations.

In this post we’ll first discuss the problem and why (as of 0.14.2 which admittedly is somewhat dated by now) this cannot be solved with SubApps and what unholy, unsafe and otherwise actually relatively unproblematic approach can resolve this.

Our requirements are these:

  • The simulation app should run in parallel to the frontend/main app (not serialized)
  • We expect simulation app to sometimes do heavy computation, this should not affect the frame rate of the frontend app
  • Some synchronization of entities needs to happen between the two apps, for this, occasionally both need to be stopped briefly.

This article was written by a human. Humans can make mistakes, check important info.
AI helped with extracting reduced code snippets from my code.

SubApps don’t work for this

At frist glance, SubApps are whats meant for this: Each app can run their own systems in parallel and once per frame the scheduler will run an extract system that transfers data from one app to the other. Most famously, Bevy uses this for the RenderApp so the actual rendering can happen in parellel to the actual game computations.

So why does this not work for us? The reason is, that the App and its subapps always run in lockstep: One frame on the app means exactly one frame on the subapp. Now if our simulation does some heavy computation that takes much longer than a 60th of a second (or whatever your refresh rate is), this means it will bring down the framerate. Not cool.

Schedules don’t work for this

A Schedule is a synchronous, non-interruptile unit of execution within one App/World. Systems run completely inside that schedule execution. So same story as with SubApps, we cannot havy any system in the schedule that runs for longer than one frame. One positive thing to mention here is that it would allow sharing a single World (which is where all the data lives), so synchronization would be trivial.

AsyncComputeTaskPool is not what we need

AsyncComputeTaskPool is built for spawning independent futures/tasks you can poll for a result. The big benefit over subapps/schedules is that it runs truly asynchronously but it runs a single function only. A complex simulation with many different systems and plugins would need to hand-roll their own ECS inside that async task pool, so we`d be duplicating the work of a bevy app.

Two Apps, one thread pool

The one thing I found (in Bevy 0.14.2) that fulfills all our requirements is this: Just build a second app. It will get its separate world, separate schedule, everything. There is absolutely no shared ownership of anything in this model and no lockstepping (or any form of foreseen communication for that matter). One noteworthy thing: These apps will automatically share the default thread pool, so without intervention they can content over compute resources. In practice I did not find this to be a problem but depending on your application could be worth noting.

Having a second app in on itself is suprisingly easy. We just need to be a bit more mindful about the simulation apps set of plugins, as this can and should not be controlled by the screen refresh rate. Instead, we want it to compute in some comparitively low, independent framerate, which in most games is referred to as the tick rate. Changing this at runtime alas is nontrivial in 0.14.2, so instead for features like “fast forward” mode, pause, etc… we can just decide in each tick whether we want to compute or not (eg computing every tick is “fast forward”, computing every 3rd step is normal speed):

// The "main" app: rendering, input, everything the player sees.
fn build_main_app() -> App {
    let mut app = App::new();
    app.add_plugins(DefaultPlugins);
    app
}

// The "simulation" app: no rendering, ticks at its own fixed rate.
fn build_simulation_app() -> App {
    let mut app = App::new();
    app.add_plugins(MinimalPlugins.set(
        ScheduleRunnerPlugin::run_loop(Duration::from_secs_f32(1.0 / 10.0)), // max simulation speed is 10Hz
    ));
    app.add_systems(FixedUpdate, tick_simulation); // tick_simulation can decide whether we actually want to do something in this tick
    app
}

Communication

Now on their own, these apps are as much separated as they can be without running two actually separate processes. But ultimately they need to communicate: User input from the main up needs to have some effect on the simulation and the simulation results must be visualized by the main app. The simplest way for this would perhaps be to clone the entire world of one of these into the other on every sync but this may be prohibitely expensive: Consider the case of having large maps or other structures, in addition both the main and the sim app may have entities and resources that are just no concern of the other, so you may want more isolation here already for architectural concerns.

In our model most data flows from simulation to main app: Visualize the current state of the world, whereas the communication from main app to simulation can be mostly thought of as individual events (abstracted user input).

The way we implement this is the following: We first set up some crossbeam channels between the two apps. When the simulation is done with its work, it asks the main app to share its World pointer and waits for that (we want this waiting to be on sim side, where it cannot affect frame rate). Once we have that, we copy over everything we need and signal we are done, so the frontend app can continue. Where we can, we prefer swapping out things rather than copying or other more efficient mechanisms but it depends heavily on your data structures and simulation logic. Bevys change tracking is also very useful here to keep things snappy.

As a rustacean the word “pointer” maybe gave you a little scare. Yes this is an actual unsafe part, we have to make sure the simulation is not doing *anything* to the world when doing this. Note this is very easy in Bevy: We just implement a system that gets mut World access and then does nothing with that world other than sending a pointer to it to the main app and waiting for confirmation that the syncing is done. Bevy ensures no other systems can access the world in this time.

Here is the setup:

use crossbeam_channel::{Receiver, Sender, unbounded};

/// Cheap, ordinary messages — safe to send any time, no handshake needed.
enum SimCommand { Pause, Resume }
enum SimEvent { TickCompleted(u64) }

/// The dangerous one: a raw pointer into the other side's World.
/// Only ever sent while both apps are provably blocked waiting on it.
enum WorldHandoff {
    Pointer(*mut World),
}
// SAFETY: sending a pointer across threads is fine on its own — see
// https://internals.rust-lang.org/t/shouldnt-pointers-be-send-sync-or/8818
// The actual safety property we rely on is that only one side ever
// dereferences it, and both sides are stopped for the duration.
unsafe impl Send for WorldHandoff {}

enum SyncDone {
    Confirmed,
}

struct MainChannels {
    commands_out: Sender<SimCommand>,
    events_in: Receiver<SimEvent>,
    world_out: Sender<WorldHandoff>,
    confirm_in: Receiver<SyncDone>,
}

struct SimChannels {
    commands_in: Receiver<SimCommand>,
    events_out: Sender<SimEvent>,
    world_in: Receiver<WorldHandoff>,
    confirm_out: Sender<SyncDone>,
}

fn make_channels() -> (MainChannels, SimChannels) {
    let (cmd_tx, cmd_rx) = unbounded();
    let (evt_tx, evt_rx) = unbounded();
    let (world_tx, world_rx) = unbounded();
    let (confirm_tx, confirm_rx) = unbounded();

    (
        MainChannels { commands_out: cmd_tx, events_in: evt_rx, world_out: world_tx, confirm_in: confirm_rx },
        SimChannels { commands_in: cmd_rx, events_out: evt_tx, world_in: world_rx, confirm_out: confirm_tx },
    )
}

fn spawn_simulation(sim_channels: SimChannels) {
    std::thread::spawn(move || {
        let mut app = build_simulation_app();
        app.insert_resource(sim_channels);
        app.run(); // blocks this thread forever, ticking at its own pace
    });
}

fn main() {
    let mut main_app = build_main_app();
    let (main_channels, sim_channels) = make_channels();

    main_app.insert_resource(main_channels);
    main_app.add_systems(PostUpdate, main_lend_world_if_requested);

    spawn_simulation(sim_channels);

    main_app.run();
}

And this is how to actually synchronize:

// This runs as an exclusive system (&mut World) in the sim app, after a tick completes.

fn sim_sync_after_tick(sim_world: &mut World) {
    sim_world.resource_scope(|sim_world, channels: Mut<SimChannels>| {
        // Ask main for its world and wait for the pointer.
        channels.events_out.send(SimEvent::RequestSync).unwrap();
        let WorldHandoff::Pointer(main_world_ptr) = channels.world_in.recv().unwrap();

        // SAFETY: main is blocked in `main_lend_world_if_requested` right
        // now, waiting on `confirm_in` — it will not touch its World again
        // until we send SyncDone::Confirmed below.
        let main_world: &mut World = unsafe { &mut *main_world_ptr };

        // Copy whatever the frontend needs to render, e.g.:
        copy_positions(sim_world, main_world);
        copy_health(sim_world, main_world);

        channels.confirm_out.send(SyncDone::Confirmed).unwrap();
        // Do not touch `main_world` after this point.
    });
}

… and thats all there is to it!

Additional things to care about & Loose ends

Note that as soon as Entitys are involved in synchronization (or referenced from anything that is), you need to worry about entity mapping (entity IDs will generally be different in the two apps), implementing MapEntities for affected resources/components and maintaining a BiMap<EntityID, EntityID> in your copy_ functions is advised.

You may also want to reduce synchronization operations where they are not necessary: Eg when the game is in pause state and no new inputs happened, there might be no reason to even check for things to synchronize, so you may want to optimize these cases a little.

As it should be obvious by now of course this is not completely parallelized: There is a phase in which we stop both worlds and synchronously copy over everything we need. This buys us good night sleep as we don’t need to worry about how asynchronous updates may affect the correctness of our game state: At each synchronization we get to see a complete and consistent simulation state and can sync that to our main world and even may apply checks at that point and discard it if it is not to our liking. The price for this is some performance/latency: The frontend will stall for a moment if you really have *a lot* of data to transfer. So far I found that as long as I’m disciplined with only syncing what actually changed I find it hard to get the copy time to be more than 1ms (my in-game performance stats dont show sub-ms resolution), so in my case definitely good enough, but your milage may vary.

Conclusion

Bevy 0.14.2 gives you schedules and SubApps for structuring the frame of a single simulation, it does not give you a primitive for two independently clocked simulations that need to occasionally synchronize. This may be much different in more recent versions of Bevy (at time of this writing 0.20rc-1 is out), I have not checked.