Web export — resource channel

This document describes the resource channel: the out-of-band upload stream that carries textures, meshes, and future resource kinds from the Python runtime (in Pyodide) to the browser-side WebGPU renderer. It complements the per-frame scene binary produced by DrawSerializer / Scene3DSerializer.

The wire format, kind registry, and Python/JS entry points in this document are the contract between the two sides. Adding a new resource kind means claiming a reserved byte — it does not require a version bump or a format negotiation.

Why a separate channel

Frame binaries carry per-frame state — viewports, lights, materials, draw groups, post-process parameters. They regenerate every tick.

Resources (texture pixels, mesh vertex / index buffers) are upload-once, reuse forever lifecycle data. Embedding them in every frame binary (pre-M2) wasted bandwidth, forced the JS side to re-detect “is this texture new?” on each frame, and made the protocol noisier than it needed to be.

Separating the two means:

• Python hands the JS side a tiny TLV blob only when something new exists.

• JS knows every byte in the blob is a fresh upload and needs handler dispatch.

• The scene-frame header stays small (20 bytes + viewports + materials + lights

  • draw groups + optional post-process — no resources section).

Wire format

Little-endian, no version field. Format changes happen in place and both sides ship in the same HTML bundle, so mismatch is impossible.

count(u32)
  [ kind(u8) + id(u32) + length(u32) + payload(length bytes) ] × count

An empty drain is encoded as b"" (zero bytes) — not as count=0. The JS router treats any zero-length ArrayBuffer as a no-op.

Kind registry

Kind	Hex	Meaning	Status	Consumer
0x01	KIND_OVERLAY_TEXTURE	2D overlay texture (Sprite2D / NinePatchRect / AnimatedSprite2D)	used	Renderer2D
0x02	KIND_MESH	3D mesh (vertex + index buffers)	used	Renderer3D
0x03	KIND_MATERIAL	Material SSBO entry	reserved	—
0x04	KIND_MSDF_ATLAS	MSDF font atlas	reserved	—
0x05	KIND_MESH_TEXTURE	3D PBR texture (albedo / normal / MR / emissive / AO)	used	Renderer3D
0x06	KIND_CUBEMAP	Environment cubemap (6 RGBA8 faces, +X -X +Y -Y +Z -Z)	used	Renderer3D
0x07	KIND_AUDIO_PCM	Decoded float32 interleaved PCM	used	AudioBridge
0x08	KIND_AUDIO_FILE	Raw audio file bytes for AudioContext.decodeAudioData	used	AudioBridge

Why 0x03 and 0x04 are reserved but unused — see What is not on the channel.

Per-kind payloads

KIND_OVERLAY_TEXTURE and KIND_MESH_TEXTURE

width(u32) + height(u32) + rgba(width × height × 4 bytes)

Both kinds share the same payload layout; they only differ in which GPU cache the handler populates. The texture is rgba8unorm, one mip level, no sampler state (samplers are owned by the renderer).

KIND_MESH

vertex_count(u32) + index_count(u32)
  + vertex_bytes(vertex_count × 32)
  + index_bytes(index_count × 4)

Vertex layout matches VERTEX_DTYPE (position vec3 + normal vec3 + uv vec2 = 32 bytes). Indices are uint32.

KIND_AUDIO_PCM

sample_rate(u32) + channels(u8) + frame_count(u32)
  + pcm(frame_count × channels × 4 bytes)

PCM is float32 interleaved in the range -1..+1. Spaceinvaders-style procedurally synthesised numpy buffers ship through this kind without a file-decode round trip. JS calls AudioContext.createBuffer(channels, frame_count, sample_rate) and writes deinterleaved channel data.

KIND_AUDIO_FILE

mime_hint(u8) + file_bytes(...)

mime_hint: 0 auto / 1 wav / 2 ogg / 3 mp3 / 4 flac. The hint is advisory — browsers sniff the format from the bytes themselves. JS feeds the payload to AudioContext.decodeAudioData, which produces the same AudioBuffer as the PCM path. File-backed AudioStreams use this kind so the Python runtime never has to ship a decoder.

Python API

Helpers live in simvx.web.runtime.resource_kinds:

from simvx.web.runtime.resource_kinds import (
    KIND_OVERLAY_TEXTURE, KIND_MESH, KIND_MESH_TEXTURE,
    pack_overlay_texture, pack_mesh_texture, pack_mesh,
    encode_tlv, decode_tlv,
)

# Build a single texture entry
entry = pack_overlay_texture(tex_id=7, w=16, h=16, pixels=rgba_array)
# entry is (kind, id, payload_bytes)

# Build a TLV blob from a queue of entries
blob = encode_tlv([entry, another_entry, ...])

# Round-trip / debug
decoded = decode_tlv(blob)  # [(kind, id, payload_bytes), ...]

WebApp.pending_resources()

payload = webapp.pending_resources()
# payload is b"" when nothing pending, else a TLV blob.

Drains the unified pending queue — both the 2D overlay textures gathered by prepare_2d_overlays and the mesh / texture entries accumulated by WebRenderer during submit_scene. Each (kind, id) pair is emitted exactly once per app lifetime.

For Python-side tests / diagnostics:

payload, summary = webapp.pending_resources(debug_log=True)
# summary: [(kind, id, payload_len), ...] — cheap to inspect, no payload copy.

JavaScript API

Implemented in packages/graphics/src/simvx/graphics/web/resource_router.js and consumed by both Renderer2D and Renderer3D.

import { ResourceRouter } from './resource_router.js';

const router = new ResourceRouter();
router.label = 'Renderer2D';
router.register(ResourceRouter.KIND_OVERLAY_TEXTURE, (id, payload) => {
    // payload is Uint8Array viewing the TLV entry's payload bytes (zero-copy).
});

router.drain(arrayBuffer);  // returns number of entries delivered

Renderer2D registers KIND_OVERLAY_TEXTURE. Renderer3D registers KIND_MESH and KIND_MESH_TEXTURE. CombinedRenderer.drainResources(buffer) hands the same buffer to each sub-renderer; the router silently skips kinds with no registered handler, so a combined 2D+3D export with a 3D texture and a 2D overlay in the same drain blob needs no special routing logic — each sub-router picks up what it cares about.

Debug logging

Set window.__simvxDebug = true (or append ?debug to the URL) to enable debug logging at boot. This turns on:

• Per-entry router logs: [Renderer3D] kind=MESH_TEXTURE id=5 len=4104 delivered

• Per-frame renderer stats (throttled to once per second): [Renderer3D] viewports=1 materials=12 lights=2 drawGroups=38 textures=7 meshes=4 bloom=on fog=mode=1 useLDR=false

Flip at runtime from devtools:

renderer.debug = true;           // renderer stats + router logs on both subs
renderer._router.debug = false;  // router logs only (Renderer2D standalone)

What is not on the channel

Deliberately scoped out, even though it could technically fit:

• MSDF atlas — delivered via the dedicated WebApp.load_atlas(...) boot call, because JS needs the GPU texture and Python needs the glyph metrics for text measurement. Routing through the drain channel would require either a two-stage handoff or a virtual-FS round trip. The current path runs once at boot and is optimal as-is.

• Materials — per-frame state, not lifecycle data. Material SSBO content is rebuilt every tick; bundling with draw groups matches Three.js / Babylon / Bevy conventions. Lives in Scene3DSerializer’s per-frame binary.

• Shaders — compiled at export time and embedded as WGSL strings in the HTML. No runtime upload.

• Streaming audio chunks — AudioStreamPlayer.stream_mode = "streaming" feeds PCM in small (~50 ms) bursts to an AudioWorkletNode. The chunks are not lifecycle data and would balloon the TLV blob if shipped through this channel; instead WebApp.drain_audio_feeds() returns a packed binary feed payload alongside the resource drain (see Audio).

• Per-frame audio updates — (channel_id, gain, pan, pitch) quads from update_audio_2d/3d ride a separate Float32Array channel via WebApp.drain_audio_updates(). They change every frame for every active voice, so they don’t fit the upload-once shape of this channel.

Adding a new kind

1. Reserve a byte in resource_kinds.py and in this document’s kind table.

2. Write a pack_XXX(id, ...) -> (kind, id, payload_bytes) helper alongside the existing helpers so call sites stay declarative.

3. Push entries onto WebApp._pending_resources (2D / 2D-overlay path) or a renderer-local queue drained by WebApp.tick (3D path).

4. JS side: register a handler on the appropriate renderer’s ResourceRouter.

5. Add a round-trip test to test_resource_channel.py.

That’s it — no version bump, no format change.