In the first post spine-gpu drew 8,000 animated Spine characters in 6.7 ms per frame, 16 to 111 times faster than the other Spine renderers for Bevy. The one idea behind it: the meshes stay on the GPU, and each frame only the bones move. This post follows the data. What do the other renderers actually spend their time on? What does spine-gpu upload, how is it laid out, and how did the per-character upload shrink from 2,160 to 680 bytes?
All numbers are from one machine (Ryzen 9 9950X3D, RTX 5080, Vulkan) and the scene from post 1: Ella, 27 bones and 18 slots, playing a random animation at a random phase. spine-gpu runs on Bevy 0.14, the others on Bevy 0.19. The code is in bevy-spine-gpu, the benchmark in bevy-spine-gpu-benchmark.
What the other renderers spend the GPU on
Before rebuilding anything, it’s worth knowing where the time goes in the usual approach. PresentMon reports how long the GPU works for a process each frame, and for the other renderers that number was 3 to 50 times larger than Bevy’s own timing of its render passes. So the GPU was busy with something other than drawing. My first guess was vertex uploads. To check, I ran all four renderers under Nsight Systems at 2,000 characters, with the Vulkan calls traced and GPU metrics sampled at 10 kHz.
The guess was half right. Lots of bytes do arrive at the GPU every frame:
But they cost almost no GPU time. These renderers put their buffers in VRAM that the CPU can write to directly. The CPU writes the vertices over PCIe, and the GPU doesn’t need to do anything for that. In the trace, 97 to 100% of the incoming PCIe traffic happens while no GPU work is running.
The GPU time goes somewhere else:
Drawing costs the same everywhere, about 0.6 ms for 2,000 characters. The difference is one long block of work on the GPU’s copy engine, every frame. spine2d creates 4,007 new GPU buffers per frame at 2,000 characters, bevy_spine 38,006. With wgpu, a buffer created with initial contents normally gets a staging buffer and a copy command. During that block, the copy engine is 99% busy, but it moves little data. Divide the block by the number of buffers and you get about 1 µs per buffer, whatever its size. The copy engine is busy with the commands, not with the bytes.
Here is one frame of spine2d next to the same 31 ms of spine-gpu, redrawn from the trace:
The GPU is not the bottleneck for the other renderers, though. At 8,000 characters their main thread needs 95 to 173 ms per frame and their render thread 65 to 574 ms, while the GPU is busy for 13 to 109 ms. Most of the cost is producing those bytes on the CPU: skinning every vertex, building meshes, creating buffers. Fewer bytes per frame help on the CPU first.
A caveat: tracing tens of thousands of Vulkan calls per frame slows the renderers down (bevy_spine 208 ms per frame under Nsight vs 168 ms without). I use the traces to see where the time goes, not as headline numbers.
What has to move every frame
What is the least that has to cross the bus each frame? Almost everything in a Spine skeleton is fixed once it’s loaded: the mesh vertices in setup pose, which bones each vertex follows and with what weights, the texture coordinates, the triangles. What changes per character and frame is the pose, i.e. one transform per bone, and which attachments are visible.
So spine-gpu uploads per character and frame:
- a 32-byte header: visible attachments as a 96-bit mask, the index of the character’s first bone, and a color tint,
- one 24-byte transform per bone: a 2×2 matrix and a translation.
For Ella that’s 32 + 27 × 24 = 680 bytes, or 5.4 MB per frame at 8,000 characters. At 144 fps that’s about 0.8 GB/s, around 1% of what PCIe 5.0 x16 can carry. The other renderers send 20 to 44 KB per character per frame.
Nsight agrees with the budget: at 2,000 characters it measured 1.6 MB per frame arriving at the GPU, against 2,000 × 680 B = 1.36 MB of instance data plus a few small uploads from Bevy itself.
The static side: vertex pulling
The static data is uploaded once per skeleton type, when it loads. It’s a Bevy RenderAsset (render/gpu_skeleton.rs) with three parts: an index buffer with all slots’ triangles in draw order, the atlas texture, and a storage buffer of vertices.
There are no vertex buffers.
The vertex shader looks its vertex up itself, by vertex_index (this is called vertex pulling). Each vertex is 80 bytes (WGSL):
struct VertexInfo {
bone_indices: vec4<u32>, // up to 4 bones
bone_weights: vec4<f32>,
bind_positions_x: vec4<f32>, // the vertex in each bone's space
bind_positions_y: vec4<f32>,
uv: vec2<f32>,
attachment_index: u32, // which attachment the vertex belongs to
color: u32, // slot color × attachment color, unorm8x4
}Spine stores a weighted vertex as one position per bone it follows, so the shader gets four bind positions, not one. Skinning is then a short loop (WGSL):
let instance = &instance_infos[input.instance_index];
let vert = &vertices[input.vertex_index];
var skinned_position = vec2<f32>(0.0);
for (var i = 0u; i < 4u; i += 1u) {
let bone = &bones[(*instance).bone_offset + (*vert).bone_indices[i]];
let p = vec2((*vert).bind_positions_x[i], (*vert).bind_positions_y[i]);
skinned_position += ((*bone).scale * p + (*bone).translation) * (*vert).bone_weights[i];
}All characters of one skeleton type share the same vertices and indices, so they are a single instanced draw call: instance_index selects the character, vertex_index the vertex. Why that single draw still gets the overlaps right is the topic of the next post.
The “&” matter. They make instance and vert pointers into the storage buffers, so the shader reads only the fields it uses. An earlier version said var instance = instance_infos[input.instance_index];, which asks for a local copy of the whole record. Back then that record held 64 bones (> 2 KB data) per vertex.
The per-frame side: two flat buffers
Every frame, the render thread packs all characters, in draw order, into two global buffers: one with the headers, one with all bones (render/instance_infos.rs). The header of character i stores where its bones start, so characters with different bone counts can share the buffer.
The packing writes directly into wgpu’s staging memory. write_buffer_with hands out a mapped slice of a staging buffer, and the packing copies into it:
fn pack_type(t: &TypeInstances, first_bone: u32, headers_out: &mut [u8], bones_out: &mut [u8]) {
headers_out.copy_from_slice(cast_slice(&t.headers));
bones_out.copy_from_slice(cast_slice(&t.bones));
for i in 0..t.len() {
let offset = first_bone + (i * t.stride) as u32;
let at = i * HEADER_SIZE + BONE_OFFSET_FIELD;
headers_out[at..at + 4].copy_from_slice(&offset.to_ne_bytes());
}
}The main world already keeps headers and bones per skeleton type in flat vectors, so this is two memcpys per skeleton type and one 4-byte patch per character for the bone offset. A few more details keep the per-frame cost flat:
- The same buffers every frame. The two buffers grow to the next power of two when they’re too small and shrink only when less than a quarter is used (Vec-style). The bind group is recreated only when a buffer is. In the Nsight trace, spine-gpu creates 8 buffers per frame in total, staging buffers and Bevy’s own included, against thousands for the other renderers.
- No allocations during extraction. Bevy copies data from the main world to the render world every frame. spine-gpu keeps the render-world copy around and updates it with
clone_from, which reuses the vectors’ allocations.
From 2,160 to 680 bytes
Initially, each character was one fixed record with room for 64 bones (Ella has 27), at 32 bytes each, plus a 112-byte header: 2,160 bytes per character, 17 MB per frame at 8,000. Each frame, that data was cloned whole into the render world, together with its lookup tables, and then copied again into a staging buffer by write_buffer. It took three steps to get to 680 bytes:
1. Size the records to the rig: 2,160 → 976 bytes. Bones sized to the skeleton, both buffers packed in draw order, staging writes and clone_from as described above. This came as one larger rewrite, I didn’t measure the effect of these changes individually. Together, at 8,000 characters, they took the render thread from 1.90 to 0.87 ms per frame.
2. Shrink the header: 112 → 32 bytes. The header used to carry a 4×4 matrix and four flags for effects of our game, which I removed for the public crate. Render thread −4.5%, GPU −1 to −2% at 8,000.
3. Shrink the bones: 32 → 24 bytes. This one is an interesting layout trap. On the WGSL side, a bone is mat2x2<f32> (16 byte) plus vec2<f32> (8 byte). Both have an alignment of 8, so the struct is 24 bytes and so is the array stride. On rust side however we needed to pad to 32 bytes:
#[repr(C)]
struct GlobalBoneTransform {
matrix2: Mat2, // glam: 16-byte aligned SIMD type
translation: Vec2,
_pad: Vec2, // 24 B rounded up to 32
}glam’s Mat2 is a SIMD type with an alignment of 16 bytes, so the struct is rounded up to 32. bytemuck doesn’t allow implicit padding in a Pod type, so the padding had to be spelled out, and the WGSL struct got a matching pad field, wasting 8 bytes per bone and frame, totalling 1.7 MB per frame at 8,000 characters. The fix is to store the matrix as its two columns to get around the alignment requiremnet on the Rust side:
#[repr(C)]
struct GlobalBoneTransform {
x_axis: Vec2,
y_axis: Vec2,
translation: Vec2, // 24 B, alignment 8
}The bytes are the same as before, minus the padding, and the shader reads them as the same mat2x2<f32>. Render thread −10%, GPU −2.7% at 8,000 characters.
For both cleanups, screenshots of frozen scenes before and after are byte-identical, so they changed nothing that’s visible.
So the upload shrank by 30% in the last two steps, and the render thread by 14%. The frame time however improved only by 2-3%. spine-gpu’s frame time is limited by the main thread, which samples the animations, and none of these changes touched it. Render thread and GPU work in parallel with the main thread, so making them faster doesn’t help until they’re the slowest (we’ll talk more about this in an upcoming post).
Keeping Rust and WGSL in sync
Two definitions of the same struct, one in Rust and one in WGSL, could drift apart without any loud error. The layout rules in Rust & WGSL are different which makes this easy to overlook: in WGSL a vec3<u32> is 12 bytes but 16-byte aligned, so the header’s bone_offset fits into the 4 bytes after the visibility mask, and the header stays 32 bytes. A small change to either side can shift every field after it, and the shader then silently reads the wrong bytes.
Two checks catch that:
- A compile-time assert on the size of every GPU struct ensures we’re not accidentally changing its size:
const _: () = assert!(size_of::<InstanceInfo>() == 32); - A test per struct that compares the Rust size with the size WGSL gives it, through
encase‘sShaderTypederive (InstanceInfo::min_size()), plusoffset_of!checks for fields the code patches by byte offset.
Colors are packed as unorm8x4 into one u32 per vertex and unpacked in the shader with unpack4x8unorm.
Hidden attachments: collapse, don’t discard
Spine characters swap images all the time: eyes open or closed, a hand in one of several poses. All of them are in the mesh, and the header’s visibility mask says which are shown. Each vertex knows its attachment, and the vertex shader checks the bit:
if uvec3_get_bit((*instance).attachments, (*vert).attachment_index) == 0u {
output.clip_position = vec4(2.0, 2.0, 2.0, 1.0); // outside the clip volume
return output;
}All vertices of a hidden attachment land on the same point outside the clip volume. Its triangles have no area and are clipped before rasterization, so no fragment shader runs for them. Initially, I checked the visibility in the fragment shader, which then called discard: the hidden triangles were still rasterized, and the fragment shader had to run for every pixel they covered.
This went in with a few other shader changes (pointer access instead of struct copies, flat varyings). Together, GPU time at 8,000 characters went from 3.4 to 2.7 ms.
What’s still left
Characters outside the screen don’t get their pose evaluated, but they are still packed and drawn with a stale pose. In the benchmark scene with 10,000 extra characters offscreen on top of the 8,000, the render thread needs about 0.3 ms more per frame, and so does the GPU. Skipping them in the packing would be the next step for the data path if we care about these 0.3ms. Since the render thread and the GPU aren’t the bottleneck, it isn’t the most important one.
Summary
- The other renderers don’t lose GPU time with drawing. They lose it on the copy engine, about 1 µs for each of the thousands of buffers they create per frame. Their real cost is on the CPU: producing 20 to 44 KB of vertices per character per frame.
- spine-gpu uploads 680 bytes per character per frame: a 32-byte header and 24 bytes per bone, written straight into staging memory, into two buffers that are reused across frames.
- Shrinking that from 2,160 bytes halved the render thread, but barely moved the frame time. That’s the subject of an upcoming post
Next post in this series will be about how to avoid extra draw commands and still get the overlaps right.