Design: mod datapack registries

Provisional contract for #811, not an implemented API. Ari's side posts the decision on the issue. This note files no issues or PRs.

Decision: decoder and generation ownership

The declaring mod owns the typed codec and its schema. The game owns decoding orchestration, candidate storage, reference resolution, validation, generation assignment and atomic publication. A decoder never installs values into live state. The runtime retains its callable only while the declaring mod is loaded; unloading requires closing its worlds first. No raw native pointer or wasm guest address is a registry value in host storage.

Propose one load-time registrar, corresponding to DataPackRegistryEvent.NewRegistry, with the appropriate Host and a scoped registrar. Illustrative declaration: example:spell, type Spell, data codec {power: integer 0..100}, optional network codec and explicit client requirement. Names and signatures await API review. Require namespaced registry keys owned by the declaring mod; reject a second declaration even with an identical codec, reporting both owners. Freeze declarations after dependency-ordered loading. Other mods and packs may contribute entry names in their own namespaces. Entry path for example:spell, entry addon:fire, is data/addon/example/spell/fire.json; pin this proposed mapping in tests before implementation. The namespaced registry folder is NeoForge's rule for datapack registries outside minecraft (its Registries guide, section "Datapack Registries"); vanilla's own registries use the registry's path only, and registration refuses minecraft keys, so the vanilla form never applies to a mod's registry.

Codecs consume bounded immutable data plus a candidate registry lookup, returning a typed immutable value or structured error. No world mutation, I/O, retained Host, scheduling or active-generation lookup during decoding. Resolve references against the complete candidate graph, including forward references; reject unresolved references and unsupported cycles. Use one generated codec/value representation for native and wasm, with typed reads through registry handles. Erasure is internal and checked against the declaration's type token; no unchecked downcasts or guest memory retained after calls.

Evidence and limits

Vanilla 26.3 was inspected only through cd /workspace/horizon && python3 tools/rea_vanilla.py method net.minecraft.resources.RegistryDataLoader load --overload 0. It accepts context lookups, registry descriptors and an executor, creates resource-manager load tasks per descriptor, and delegates to the common loader returning a future of frozen registry access. This supports descriptor-driven candidate loading; it does not prove arbitrary mod registration, reload scope or failure atomicity. Those are Horizon proposals here. Vanilla caller coverage and the distinction between world-generation registries and reloadable registries still need oracle fixtures before implementation.

The public registry guide was reachable but labels itself 26.1. It is background only, not proof of 26.3 behavior. The 26.3 NewRegistry Javadoc was requested and inaccessible through the reader. Exact overloads, network codec semantics, absent-mod behavior, reload guarantees and failure policy are unverified. The issue names the counterpart and asks for declarations and entry decoders. No NeoForge source, patches or other project's source was read. Do not advertise exact counterpart equivalence until public documentation verifies it.

The public NeoForge 26.3 changelog (maven.neoforged.net, neoforge-26.3.0.102-changelog.txt) and the pull requests it names (GitHub descriptions only; no source read) add what the 26.1 guide could not: 26.3.0.20-beta "[26.3] Add support for registering custom reloadable datapack registries" (#3527, with a cleanup of the registration event and hooks), 26.3.0.44-beta "[26.3] Add dedicated builder for datapack registries" (#3572, whose first option is a RegistryValidator for the registry), 26.3.0.58-beta "[26.3] Gather Datapack Registry Entries Before Data Providers" (#3587) and 26.3.0.88 "[26.3] Improve datapack registry definition storage" (#3659: world, reloadable and synced registry definitions are all kept in lists mods can read). So NeoForge 26.3 distinguishes world registries (read at world load) from reloadable ones, which the earlier open question about which categories reload anticipated. Horizon's custom registries reload with /reload and are validated by their schema and bounds, so they correspond to the reloadable kind with a built-in validator; Horizon has no world-load-only kind, and adding one would be a separate decision. The Javadoc for the builder's exact methods was still unreachable, so no signature is claimed.

Decode, commit and retained identity

Decode after pack selection and overlay precedence have selected one winning file per entry, before any live table or network publication. Reuse existing pack conditions and overrides; never invent another pack scanner. Include prepared base data even when there are no overlay files: the early return in build_world_tables must not skip declared registries. Report mod, registry, entry, pack, file and field path for invalid data. Duplicate declarations are registration errors; multiple pack definitions are ordinary priority overrides. Invalid winning data must fail, never fall back to a lower-priority value. Initial failure prevents world admission. Reload failure retains all old tables, registry values, generations and client snapshots.

Use a world/session identity plus monotonically increasing u64 registry-set generation. Initial successful publication is generation 1, every successful reload increments once, including identical content. Refuse exhaustion instead of wrapping. Failed reload does not increment. A handle contains session, registry type, generation and entry identity. All access and use checks them against the active set: generation 1 handles and held entry wrappers return StaleGeneration after generation 2, even if names and slot numbers match. Deleted entry lookup returns MissingEntry. Closing and reopening a world invalidates session identity. A retained entry wrapper must validate on every read; do not expose an Arc or reference that bypasses validation. Explicit copied plain data is a snapshot, carries no live entry authority, and cannot be passed as a current entry. Persist names, never session slots; re-resolve explicitly after load.

Prepare codecs, references, derived tables and wire payloads before commit. Publish the registry set with WorldTables and dependent data under one coordinator at a safe server boundary. Consumers cannot observe mixed generations. Existing reload-listener apply callbacks run before world preparation today; implementation must move fallible preparation before effects or constrain callbacks to candidate-local effects, rather than claiming existing ordering provides rollback. Listener panic or guest trap aborts the candidate; no partial publication. Catch native unwind at the boundary, disable/report the failing callable under runtime policy; process-aborting panics cannot be recovered.

Verified Horizon route owners

Paths and lines are inspected at base 8b736c3cbc; these identify integration owners, not implementation claims.

This is the inspected production route inventory; tests also call the shared functions directly. Exact receive-side configuration commit and all vanilla callers remain implementation trace prerequisites; packet tests alone are not proof of client publication coverage.

Clients and absence negotiation

Propose three explicit modes: server-only (no network codec), synchronized-required, synchronized-optional. A network codec defines a bounded client projection, which may omit server secrets. Negotiate registry key, declaring mod, schema fingerprint and mode before any payload. Required missing mod or incompatible codec refuses join with a precise reason before entry decoding. Optional absent client mod receives no payload and gains no fabricated empty registry; server gameplay may not require that client's interpretation. Server-only registries do not require the client mod merely because they exist. Reconcile this with current mod-list comparison so blanket mismatch refusal does not silently defeat optionality; preserve unrelated dependency requirements.

Server is authoritative. Send a full sorted snapshot initially and after reload with session, generation, entry keys and bounded projected values. Client decodes and validates off to the side, acknowledges readiness, then activates at the agreed boundary; generation-tagged traffic is held until activation. Failed required-client decode disconnects that client without exposing a partial set; successful server reload need not roll back for a disconnected peer. Never send generation-2 references to a peer still using generation 1. Late joins receive only the current complete generation. Network numeric slots are scoped to this snapshot and never persisted. Transport and configuration re-entry details require protocol review; do not create a parallel registry packet store.

Existing registrars, runtime, and cost

Existing register_* hooks declare built-in content or codec types at mod load; they do not become per-reload callbacks. The new registrar declares the registry schema once; loading its JSON happens each candidate build. add_server_reload_listeners remains for independent resource consumers, not a second way to declare this registry or publish entries. Data maps annotate existing entries and remain a separate feature.

crates/horizon-mod-runtime/src/registry_keys.rs:6 owns load-time Requests and line 18 owns Names, including requested slots and resolved maps. These tables are name/slot machinery, not typed datapack value storage or a reload generation authority. Reuse their validated key resolution where appropriate; place immutable per-world generation sets under the world coordinator. Keep mod-runtime engine-independent. Generate native and wasm registration, codec calls, typed reads and refusal errors together; both use the same dependency ordering and semantics. Native exact-build checks and wasm interface checks remain their existing admission rules.

Treat files, decoder results and network bytes as untrusted. Proposed hard defaults: 1 MiB per encoded entry, depth 64, 65,536 entries per registry, 64 MiB aggregate encoded candidate data, and 256 registries per session. Check lengths before allocation, decoded collection counts as well as encoded sizes, identifiers, finite floats, schema ranges, reference count and total decoded memory budget. Bound decoded memory to 128 MiB per candidate and callback recursion to 64; wasm additionally consumes its existing fuel/memory limits. Native callbacks require trusted executable admission and cannot be forcibly time-limited safely; document this limit. Over-budget candidates fail wholly, never truncate. Refuse only custom registry keys owned by this feature; unrelated unknown-key behavior remains with its existing owner.

Resolve declarations once. With none, skip custom scanning, codec invocation, candidate allocations and synchronization extensions; no per-tick mod scans. Reads validate session/generation in constant time and perform indexed lookup. Reload work scales with selected entries and references, not ticks. Measure the empty declaration path and native/wasm bounds independently.

Acceptance matrix for implementation

Expected values below are proposals, not tests reported passing.

Proposed stages

File none. Stage 1: obtain exact public 26.3 contract and vanilla load/reload caller fixtures; settle the decisions below. Stage 2: generated registrar and bounded codec representation for native and wasm, duplicate/type validation and test-mod fixtures. Stage 3: candidate ownership, generation-checked reads and common atomic coordinator across every listed route, including listener failure. Stage 4: negotiation, snapshot transport, receive-side activation and absent-mod fixtures. Stage 5: acceptance matrix, route parity, adversarial limits and empty-path allocation measurements. No stage authorizes Rust work in this documentation task.

Stage 3 progress

Built in horizon-mod-vocab (world/datapack_registry_set.rs), engine-neutral and tested without a world: DatapackCandidate::build decodes the winning file of every entry of every declared registry and resolves references against the whole candidate (forward references work, an absent target refuses everything, a reference to an undeclared registry is left to its owner); DatapackRegistrySet publishes it as generation 1 of a session, reload gives the next generation (once, even for identical content, never wrapping), and DatapackHandle reads check session then generation, answering WrongSession, StaleGeneration, UnknownRegistry, MissingEntry or, with no world open, Unavailable. The game builds the candidate in build_world_tables and publishes it at world open and on /reload, and a mod reads it on the server through level_datapack_entries, level_datapack_handle and level_datapack_read. Every route in the table above ends in install_world_tables, so each publishes once: explicit /reload and datapack selection, a content inventory change (reload_datapacks(true)) and the additional content refresh all prepare their tables with prepare_world_reload and install them there, and a world open publishes through open_world_tables. A remote client builds no tables, so it publishes nothing. Not built yet: the listener ordering (candidate-local writes until publication).

Stage 4 progress

Built in horizon-mod-vocab (world/datapack_snapshot.rs), engine-neutral and tested without a connection: DatapackRegistrySet::snapshot takes the synchronized registries (never server-only ones) that a peer's negotiation lets through, as a DatapackSnapshot of session, generation and, per registry, its key, the schema it was decoded with and its entries sorted by id. It encodes with the wire format, bounded at 64 MiB, 256 registries and 65,536 entries per registry. On the client, DatapackSnapshot::candidate checks the snapshot against the client's own declarations and builds nothing unless everything passes: an unknown registry, a schema that differs from the local declaration, a required registry that is missing, a bad or repeated id, a value outside its field's bounds, or a reference to an entry the snapshot lacks each refuse the whole snapshot. A declared registry the snapshot lacks (server-only, or optional and not negotiated) stays empty, never a fabricated registry. DatapackRegistrySet::received publishes the candidate at the server's generation under the client's own session, so a held handle fails across a reload and across a reconnect as on a server.

The join carries it (horizon-client/src/datapack_net.rs). A Horizon client that declared synchronized registries sends horizon:datapack_offer (key and schema of each) after the mod list. Before any registry data the server answers on horizon:datapack_registries with the snapshot of the registries both sides have, built by snapshot_for: a Required registry the client did not offer with the same schema ends the join with a reason naming the mod and the registry (a vanilla client, or a Horizon one that sent no offer, is refused the same way), and an Optional one is left out with nothing sent. The client checks the snapshot with candidate while configuring; a snapshot that fails ends the join and nothing is kept. The received set is what a client mod reads through level_datapack_entries, level_datapack_handle and level_datapack_read, under a session of the client's own.

A reload that publishes the next generation sends every joined Horizon client a new snapshot on the same channel, built from the offer it made at the join (owned by that live connection and released when its senders are dropped). The client checks it as at the join and swaps its set in one step, keeping its session, so a handle from the generation before fails as StaleGeneration. A server refusal or encoding failure disconnects that client with a named reason instead of silently retaining its previous generation. A snapshot that fails the check, or whose generation does not come after the one held, ends the connection with the old set unchanged until then; the server's committed generation does not roll back for a disconnected peer.

What was left open in the first draft of this stage is settled by how the code is ordered, not by new code. (1) Activation versus play traffic: install_world_tables publishes the set and sends each joined Horizon client its snapshot first, through the connection sender, which preserves packet order; what a mod sends to players is queued in native_net.to_players and leaves at the tick's end (native_net.rs). So on one connection the snapshot always precedes every mod message queued in the reload's tick, and a message queued earlier in that tick (naming the old generation) arrives after the swap and reads as StaleGeneration, which a mod already handles. A client never sees a message that names a generation it has not yet received, so no holding queue is needed. (2) Reconfiguration: the declarations (registries, schemas, required or optional) come from the loaded mods and are fixed for the process; a reload changes entries only, so the negotiated set of registries cannot change under a joined client, and the snapshot a reload sends is built from the offer made at the join. A change of the mod list needs a new join, as for every other mod content. (3) Wasm: level_datapack_entries, level_datapack_handle and level_datapack_read are methods of LevelsHost, and their types (DatapackHandle, DatapackValue, DatapackReadError) derive Wire, so the generated wasm host exposes them exactly as it does level_biome_at; the client's implementation reads the received set. The generated-API checks of CI cover the projection.

Snapshot transport uses ordered fragments on the same custom-payload channel, for configuration and play alike. Each fragment starts with DRS1, a little-endian 32-bit total length and byte offset, followed by at most 512 KiB of data. Totals above 64 MiB, changed totals, wrong offsets, duplicate starts and truncated fragments are refused with a named reason. The client keeps its previous set until the complete snapshot decodes and validates; configuration cannot finish with an incomplete transfer. Reconnect clears both the old set and any partial transfer.

Vanilla's Registry Data packets accumulate one registry per packet and validate at Finish Configuration; Known Packs omits data already held by the client. Neither mechanism specifies arbitrary fragmentation of a single large custom payload. This channel therefore uses the small explicit total/offset scheme above, rather than pretending vanilla fragments custom payloads. Its fragments fit below the vanilla unknown clientbound custom-payload limit (1 MiB), the protocol's 21-bit frame bound and its 8 MiB decoded-packet bound, even for incompressible data. The 64 MiB bound applies to the entire decoded snapshot, not each packet.

Open questions

• What does the exact 26.3 counterpart guarantee? Recommendation: require reachable public Javadoc before implementation; retain explicit Horizon proposals where public documentation is silent.
• Which registry categories reload? Recommendation: custom registries here always replace on reload; do not infer that vanilla world-generation registries also reload. Verify their distinction with oracle fixtures.
• Shared codec API with #813? Recommendation: use one generated bounded value/codec foundation, with separate registry lifecycle ownership; avoid a second transport representation.
• How should callback ordering become atomic? Recommendation: complete all fallible world and registry preparation before effectful listener apply, and require candidate-local writes until publication; review existing callback semantics before changing them.
• Are optional synchronized registries safe for all consumers? Recommendation: permit omission only through explicit declaration and negotiation; required gameplay interpretation must select required mode.
• Are proposed budgets appropriate? Recommendation: retain the concrete limits above as provisional defaults, measure representative packs, and approve changes before freezing the API; never accept unbounded values.
• Exact client receive and reload activation owner? Recommendation: trace configuration collector through world admission and reconnect/reconfiguration before coding transport, then add verified file:line owners and tests to this note.