TensorFold v1.3.3 Release Notes: TensorFold 1.3.3
1.3.3 makes GLM faster. On CUDA, GLM-5.3-Flash drafted decode after a 1,040,384-token prompt is 13.8% faster at TP=4. On Macs without tensor units, GLM cold prompts take 5.17% to 5.68% less time. It adds picture generation for MOSS-Image on CUDA through POST /v1/images/generations, GLM-OCR on HIP, DFlash2 drafting for GLM-5.3-Flash on Metal, EXL3 mcg experts at 1 to 8 bits for GLM-5.3-Flash on CUDA, structured output for DeepSeek-V4.1 on CUDA, and Idol's image encoder on Metal. The server gains an opt-in execution witness, a prompt-cache flush route, per-request deadlines (deadline_s), structural tags, deferred and namespace tools, and TENSORFOLD_PARALLEL. By default, long prompts on CUDA now share their steps with running replies, each image takes a fixed share of the visual-token budget, and LFM2-MoE, LFM3, MiniMax-M3 and Qwen3.5-2B on Metal fill new prompts between other streams' decode rounds. The speed changes below keep the same tokens, bits or bytes where their bullets say so.
Speed
- GLM-5.3-Flash on CUDA (long-context decode): drafted decode after a 1,040,384-token prompt takes its DSA index scores from
tf_glm_index_scores. It is 13.8% faster at TP=4 (172.39 against 151.51 tok/s median), with byte-identical scores and the same tokens. Shorter windows keep tf_glm_scores (#3689).
- GLM-5.3-Flash on CUDA (TP=3, 5, 6 or 7): at these widths a rank's fused KDA input is not a multiple of eight columns. The stack is now padded at load to its own width rounded up to eight columns and stays row-major bf16, so prompt windows take the staged B16 tiles with every byte as before.
TF_GLM_KDA_PAD=1 still tiles the stack for glm_mm (#3697, for #3516).
- GLM-5.3-Flash on CUDA (split drafter):
TF_GLM_DRAFT_SPLIT=1 splits the DFlash2 drafter's MLP across three ranks or more where each share is 256-aligned, without TF_ROCE=1. The ranks must set it alike, two ranks split as before, and unset changes nothing. On three GB10s whose gathers stay on NCCL, decode with --drafter is about 5% faster at one stream (four streams within their start-to-start spread), with the same replies and the same tokens a round on the measured prompts. As with TF_ROCE=1, the split drafter adds its down projection's rank shares in another order, so its proposals can differ at near-ties (#3616, for #3581).
- GLM on Metal before M5 (prompts): vectorized prompt, KDA and indexer kernels on Macs without tensor units cut cold prompt time 5.17% to 5.68% (2K to 64K tokens), with 0/64 token differences. (#3921, refs #3152).
- GLM on Metal (DFlash2): decoding skips inactive draft rows and their tap capture, and the tap mean writes into its ring in place. This cuts sampling overhead without changing verified tokens (#4257).
- DeepSeek-V4.1 on Metal (multi-stream decode): a decode round over several streams runs each per-stream stage (window key, compressed rows and index keys, ratio-2 pooling, decode attention, index scores, candidate blocks and the top 512) as one dispatch over a table of the streams' state. There is one barrier between stages instead of one a stream, and the bits are the same (#2954, refs #2984).
- DeepSeek-V4.1 on Metal (sampling and drafts): DSpark drafts are selected over the whole vocabulary on the GPU with an adaptive block size (#3405). Sampling keeps its scratch between picks and scans greedy logits in vectors, choosing the same IDs (#3288).
- DeepSeek-V4.1 on CUDA (copy drafter): opt-in
TF_DSV41_COPY_DRAFTER=1 turns on the lookup planner with a copy drafter that ranks matches within a length band by how much continuation they offer. It offers 8 drafts, widens to 15 after a round that keeps every copy, and offers a copy only when it is expected to keep at least as many tokens as the other drafter. It is refused when TF_DSV41_LOOKUP is off or blank, and unset leaves TF_DSV41_LOOKUP_PLAN=1 as it was (#2638).
- Flash Next on CUDA (draft head): a bf16 language head's draft rows load as 4-bit affine groups of 32 by default.
TF_FLASHNEXT_DRAFT_HEAD=bf16 keeps them as stored, and a malformed value stops the start by name. Replies are unchanged because drafts are verified (#3477, refs #1870).
- Flash Next on CUDA (
--tp 2 control link): a rank's control-link wait polls with a 50 µs timeout for its first 50 ms and then blocks as before, so drafted rounds pick up the next header sooner. Replies are unchanged (#2971).
- Flash Next on HIP (prompts and load): on one GPU, a prompt's first token goes out before its last chunk's kept prompt states are saved, and they are saved before any later step writes the stream's caches. Kept-state copies pass through one pinned 32 MiB buffer, and on Linux each kept state sits on small pages of its own (#3647). With the prompt cache on, a prompt chunk runs on through the cache marks its side buffers can hold, writing each marked row's state aside, instead of ending at the first mark (#3646). The first stream's caches are made at load, so the first prompt no longer waits on allocating them. Each layer's weights are read past the page cache by threads, with the next layer's read under way, and the n-gram gather spreads over all of the pool's parts instead of at most 32 (#3644).
- Flash Next on XPU: prompt and decode passes run on Xe2 DPAS tile kernels, with prompt chunks sized to the solo stream and MTP chains that stop when draft confidence falls. Replies stay bit-exact across chunk sizes (#3603, refs #3583).
- Qwen3.8-27B on Intel GPUs (attention): decode and verify attention (
attn_decode) runs on a (rows, heads, slices) grid, so the tree rows of one key slice run back to back. Each work group's arithmetic is unchanged, so replies keep their tokens. The XPU kernel contract holds attn_decode and attn_merge to the previous kernels' bytes, with a planted control for the launch order (#2595, refs #2543).
- Qwen3.8-27B on Metal (sampled drafts): a lone sampled stream's DFlash2 draft tree ranks its candidates with the stream's own keyed draw, so the drafts follow what the target samples. The target still decides every token, and greedy and multi-stream rounds are unchanged (#3978).
- Qwen 3.5/3.6/3.8 on CUDA (prompt chunks): a prompt chunk before the last that nothing reads (no last-layer tap, learning, MLP or pocket reader, prompt int8 path or image) stops its last layer after its keys, values and RoPE. It skips that layer's query projection, attention, output projection, MLP and head, with the same reply tokens. The final chunk is unchanged (#3060, fixes #2941).
- CUDA (images on resume): a resumed prompt encodes only the images its kept state does not hold (Gemma 4, Qwen3.5 and Flash Next on one rank), so a follow-up turn skips the vision tower for images the restored prefix already holds. A resume never cuts an image, and Flash Next at
--tp 2 keeps encoding every image as before (#3164).
- CUDA (long prompts beside replies): a long prompt now shares its steps with running replies by default. A filling prompt with more than 32,768 rows left takes a decode share of 0.25 and the 512-row chunk hint, as
--decode-share 0.25 does, while a shorter prompt fills whole as before. --decode-share and TENSORFOLD_PREFILL_SHARE keep their meanings (#3486, refs #3397).
- Nemotron and LFM2 MoE on CUDA (message starts): the prompt cache keeps a state at each message start by default. The next request in a conversation resumes from where its last message began instead of reading the conversation again, within the existing prompt-cache budget and slots (#3696, part of #3617).
- Nemotron on HIP (MTP drafts): at
--tp 1, --mtp-drafts takes up to 15, widening a lone stream's verify window up to 16 rows. Unset or at 3 or fewer, the window stays at four rows. Rounds shared by several streams chain at most three drafts a stream, and above --tp 1 more than three is refused by name (#3613, refs #3394).
- Metal (new prompts beside replies): LFM2-MoE (#3058), LFM3 (#3088, part of #3021), MiniMax-M3 (#3089, part of #3021) and Qwen3.5-2B (#3090, refs #3021) fill a new prompt a chunk at a time between other streams' decode rounds, instead of stalling those replies for the whole prefill. On LFM3, MiniMax-M3 and Qwen3.5-2B, chunks are cut as before, so replies are unchanged, and a cancel keeps the state at the last chunk's end. On MiniMax-M3, a stepping drafter absorbs each chunk before the next. LFM2-MoE declares
prefill_step and cancel_prefill beside reports_fill.
- MiniMax-M3 on Metal (drafters): the DFlash2 and DSpark drafters hold up to eight streams' blocks in one pass and one wait, instead of a pass per stream, and absorb no longer waits for the GPU. Drafted replies are unchanged (#3121).
- LFM2-MoE and LFM3 on Metal (rounds): a keep that drops no rows skips the conv-state and logits copy in
Cache.keep, since commit and run already wrote those bytes; a partial keep still restores through a new take() (#3078). On LFM3, every stream's qk in a shared round runs in one dispatch instead of once per stream per attention layer, with the same arithmetic per row, so replies are unchanged (#3079).
- Spark 2.5 on CUDA: gate and up load as one stacked weight and decode in one launch on every CUDA chip instead of two launches, with the bits unchanged. Only 4-bit checkpoints take this path (#3130).
- HIP (routed prompts): a routed prompt plan keeps the stacked shared expert whole up to the GEMM tile instead of cutting it into 8-row items. On gfx1151, a prompt pass of 33 rows or more takes the GEMM tile for every expert. Decode, lane rounds and the MTP draft head keep the 8-row tile, and outputs keep their bytes (#3521, fixes #3161).
- Lane host (identical prompts): concurrent identical prompts join a leader that is filling a prompt state it will keep, so they reuse its whole fill and return the same tokens they would alone. A prompt whose state would not be kept, a zero budget, the leader's cancellation or a timeout still runs on its own (#3693, refs #3397, #3487).
- Tokenizer: a request gap with no normalized added spelling reuses the tokenizer's memo instead of re-encoding cold. Long multi-message requests on tokenizers with normalized added tokens (DeepSeek-style ones) cost less per call. Gaps carrying a spelling keep the mask, and every id is unchanged (#2844).
New
- MOSS-Image on CUDA: the checkpoint's Qwen3.5 prompt encoder and VAE decoder run beside the transformer, so the native server serves pictures through
POST /v1/images/generations. The same request twice gives the same bytes (#2281, part of #1573).
- OCR on HIP: GLM-OCR on native HIP serves
POST /v1/ocr. It returns normalized page Markdown with pipe or spanned-HTML tables and $/$$ math, blocks: [] because it has no geometry, and named refusals for header and footer extraction (#2572).
- GLM-5.3-Flash on Metal (DFlash2): it can draft with a published DFlash2 checkpoint behind
--drafter DIR (and --drafter-bits). Drafted replies equal plain ones, and the default stays MTP (#3920).
- EXL3
mcg checkpoints on CUDA: GLM-5.3-Flash loads EXL3 mcg routed experts at any whole-bit width from 1 to 8, mixed by projection within a layer, where only 4-bit was admitted, and its prompt expert kernels gain a 3-bit form; the 4-bit path is unchanged (#2094, refs #2093). Qwen3.5 admits EXL3 checkpoints with the mcg codebook marker alongside mul1 ones, and unknown markers are still refused as an unsupported quantization (#3412).
- DeepSeek-V4.1 on CUDA (structured output): in the
dsv41-cuda source build, structured output runs on the lane core's token grammars with drafts on. This covers response_format, the guided_* fields, tool_choice: "required" or a named function, and strict tools under tool_choice: "auto". A grammar's stop token ends the reply with finish_reason stop, even with ignore_eos (#3026, closes #1664, from #1665).
- Qwen3.5/3.8 and Flash Next on CUDA (quantized vision towers):
--vision loads a tower whose linears ModelOpt stores as MXFP8 or NVFP4, as nvidia/Qwen3.8-Flash-Next-FP4-Local does, where it stopped at the first quantized linear. They are decoded to bf16 on the host at load, one linear at a time, so the tower runs in bf16 as before. A floating-point tower is still read as stored (#3727).
- Drama 3 on CUDA (NVFP4): compact NVFP4 checkpoints load as stored, held experts are staged in slot order, and routing is masked to their original ids (#4144).
- Nemotron 3.5 Lightning on HIP (MTP head): the MTP head file is read at the width its own metadata states, so it drafts beside an oQ checkpoint of any width. A head at a width the launches do not read is refused by name (#3251, part of #3061).
- Idol on CUDA (decoders): Idol's octree and Gaussian decoders run on CUDA beside the Metal decoders, driven by the same
tf-idol-decode case driver and gates. The stem, to_kv and the head norms run in fp32, input_layer and the head in true fp32 (no TF32), and the blocks through the denoiser's CUDA linears and norms. Attention keeps fp32 scores and sums and rounds P to nearest fp16 as torch's CUDA kernels do, and runs fp32 throughout at 8 query rows or fewer. The six kernels and 13 controls are registered with the kernel contract (tf-cuda-test idol-decoder-contract). It admits sm_121 (GB10) and sm_120 and refuses other GPUs. It compiles for both and its host tests pass, but it has not yet run on a GPU (#1858, after #1773 and #1767, for #1757).
- Idol on Metal (image encoder): Idol's FLUX.2 VAE image encoder runs natively in fp32, from one 1,024 x 1,024 picture to the denoiser's 4,101 condition rows, with the posterior's noise as an input. It takes 0.185 s an encode on an M5 Ultra, against 0.095 s for the SDK's bf16 encoder on torch MPS, and lies far closer to a float64 run: worst row 5.1e-5 against the SDK's 2.7e-2.
tf-idol-vae gates it from a case that tools/zig/idol_case.py vae records, with the ungated FLUX.2-klein-4B weights (#1774, after #1766, for #1757).
- MiniMax-H3-Image on Metal:
tf-h3-image compares the Metal transformer steps with exported picture cases (#2766). It now runs the transformer (MiniMax H3's, without the audio layers, on M5 tensor units) from a picture case through its six distilled steps, each with its own output head, beside the engine. The VAE's rules (config, tiles and blend, latent statistics, pixels) run on the host, core/picture gains Pillow's Lanczos-3 resampling, and clip cases run as before (#2775). tf-h3-image-encoder turns an edit's reference pictures into their clean rows with the VAE's encoder in fp32 on any Apple GPU, with a selftest of each kernel against a float64 host reference; the reference rules (sizes, crops, grids, encoder layers, clean and anchor rows) run on the host (#2778, on #2775).
- Ling-3.0-tiny on Metal (diagnostic): a bf16 forward diagnostic runs all 24 layers of the published checkpoint on kernels that already exist, with no new GPU kernel source.
zig build test-ling-forward, given TF_LING_CHECKPOINT and a Python with torch and safetensors (TF_LING_PYTHON), checks whole, chunked and resumed runs byte for byte and reports the distance from float64. Nothing is served yet (#3919, part of #1247).
Server and API
- Execution witness:
serve --execution-witness (off by default) adds GET /v1/execution/witness under the read key scope. It returns a snapshot naming the running binary, the checkpoint's text files, the serve defaults and TF_* settings, the engine's epoch and a per-process instance, or unknown with its reason. It needs an API key or --unix-socket and hashes only after the server is ready (#2901, refs #2741). With the flag on, "execution_witness": true on a chat completion or completion adds an execution_witness object to the reply (on a stream, its finish chunk). It holds the witness read when the engine takes the request and again at the end, the SHA-256 of the prompt ids, the resolved request and the reply tokens, the finish reason and a binding over them, or unknown with a reason (#2958, refs #2741). Native Qwen3.5-2B on CUDA (one rank, MLX affine 4-bit, text only) attests its path: once the server is ready it reads every checkpoint shard, after stamping the files before the load, and binds them with the kernel images, GPU, driver, family facts and resolved profile values. Other engines still read unsupported (#2956, refs #2741).
- Image token budget: every image takes a fixed share of the visual-token budget,
--vision-image-tokens / --vision-max-images (1,024 tokens at the defaults), whatever the request's image count. A turn that adds an image no longer re-sizes earlier ones, so the prompt cache can resume from the first image. A lone image gets that share, not the whole budget, and a budget that leaves no share is refused by name, naming the image and both flags (#2717, fixes #2510).
- Prompt cache flush:
POST /v1/prompt_cache/flush drops the prompt states the server keeps for resuming conversations, in RAM and on disk when it keeps them there (--prompt-cache-disk, --learn-conversations or --spill-gib), once no request is running, so replies in flight are unchanged. It needs a full or [admin] key when API keys are on (a loopback caller without keys). It answers 503 if requests have not ended within 10 minutes and 501 on an engine without the prompt cache, and it leaves --learn's shared-prefix states on disk (#3210, fixes #3183).
- Request deadlines: chat and legacy completions requests take
deadline_s (seconds, more than 0 and at most 1,000,000; any other value is refused before the engine). At that time the server cancels the engine itself, and the reply ends with finish_reason: "deadline", keeping every token the engine gave and its usage. A reply the engine ended first, or a cancel that came first, keeps its own ending, and without the field nothing changes (#3438).
- Structural tags:
response_format: {"type": "structural_tag", ...} on the native server keeps the reply free text, while each structure's begin opens a region that must be JSON matching its schema and close with its end, with optional triggers. Past 64 structures or triggers, 256-byte tags or the 512 KiB grammar limit, the request is refused with HTTP 400 before any stream starts (#3208).
- Deferred tools on
/v1/messages: /v1/messages and its count_tokens accept deferred tools. A tool with defer_loading: true stays out of the prompt until a tool_reference block in a tool_result loads it. It is rendered by a template that reads both fields (GLM-5.3's), or else as the tool's name, description and parameters in text. A reference to a tool the request does not offer, a non-boolean defer_loading or all tools deferred is refused, and requests without these fields are unchanged (#3212, fixes #3115).
- Namespace tools on
/v1/responses: one level of namespace tools is accepted. Each function is offered to the model as <namespace>__, and its calls come back with name and namespace, whole, streamed, stored and over WebSocket. Requests without namespace tools are unchanged (#3610, fixes #3266).
- Thinking before tool calls: opt-in
--tool-think-close bounds thinking for required or named tool calls, so the call has room in max_tokens. The flag defaults off and keeps smaller thinking budgets (#4157).
- Default lanes:
TENSORFOLD_PARALLEL sets the default lanes when serve has no --parallel: a whole number 1 to 4096 or auto. The flag wins when both are given, a blank value keeps the default, and any other value is refused by name at startup (#3685, for #3684).
- Slow stream readers: a streamed reply whose client falls 1,024 tokens behind sits out rounds until the reader is within 512 again. One whose client reads nothing for 60 seconds is cancelled like a disconnect, so a stalled reader stops holding its lane. The text a reader receives is unchanged (#3234).
- Qwen3.8 Flash Next on Metal (cancelled prompts): a request cancelled while its prompt is still filling keeps the prompt state it had already landed at a resumable cut, so a follow-up with the same prefix resumes from there instead of starting over (#3657, refs #3618).
Desktop and Web
- App on Linux (sandbox): with
TENSORFOLD_LINUX_SANDBOX=1 and bubblewrap installed, the app's commands run in a bubblewrap sandbox. It keeps every repository's config, config.worktree, hooks and info read-only, nested repositories included, and freezes worktree .git pointers. A workspace with no repository, a worktree whose gitdir or shared directory lies outside the workspace, a path the sandbox cannot inspect, or a walk past 32 levels or 50,000 directories is refused with code 126 before anything runs. Without the setting, Linux commands stay refused as before (#3364, fixes #1693).
- App (hardening): a packaged build ignores
VITE_DEV_SERVER_URL (#3049). Explore and Oracle child agents refuse a tool outside the list they were given, so a read-only run stays read-only (#3040). Only the owner token resolves a tool approval on the remote-control server (#3034).
- Mobile: a pairing link opened from outside the app asks first and names its host (#3036).
- Web: Qwen3.6-35B-A3B is listed for CUDA only, and starting it on a machine without CUDA is refused before anything launches, so a Mac is no longer pointed at the Python engine for it (#3738, refs #3710).
Platforms and settings
- Flash Next on HIP (integrated GPUs): on an integrated GPU that shares the host's pages (no firmware carve-out), the default kept-prompt cache takes no more than the memory the streams leave, at most 16 GiB and none when nothing is left. An explicit
--prompt-cache-gib still sets it (#3645).
- CUDA (RoCE with
TF_ROCE=1): the RoCE all-gather pairs each peer's stripes with the HCA on that peer's IPv4 /24 (the rule fabric/tp uses, now one copy for both) when every rank's plan holds. Three GB10s cabled as a switchless ConnectX-7 triangle now take the RDMA path instead of falling back to NCCL. A switch or two ranks keep the same-numbered pairs, and a TF_ROCE_HCA of more than two names is refused by name instead of cut to two (#3518, fixes #3411).
- Qwen dense and MoE hybrids on CUDA (KV in host memory): opt-in
TF_QWEN_KV_HOST=1 keeps the KV pages in mapped pinned host memory, with the state and verify runs on the device, so a long window can fit on a card with little room left beside the weights. It needs TF_QWEN_KV_PAGES, TF_QWEN_KV_PAGES_ALL=1, --parallel 1 and a discrete GPU, and takes an optional TF_QWEN_KV_HOST_GIB cap. It refuses spill, disk snapshots and TF_QWEN_KV_SHARE by name. It trades speed for room, since attention reads the pages over the device interconnect; unset, the plan and every path stay as they were (#2637).
- Qwen3.8-27B on Metal (
TF_QWEN27_METAL_KV_PAGES=1): a stream that opens a slot is promised the KV pages of its prompt and one widest pass, instead of every slot being promised its whole window at load. A pass past that promises the rest before it takes a page, and fails with LaneMemory before any page is taken when the pool cannot meet it. A stream that leaves, a failed or requeued prompt pass, a reset and a move to slot 0 give back the slot's pages and promise (#3377, refs #1293). Macs with tensor units (M5) now take the flag instead of refusing it, and a paged prompt pass finds each 512-key chunk's two pages once instead of reading the page table at every 32-key step, with every token and cache row unchanged (#3180, part of #1293). Without the flag nothing changes.
- Qwen3.8-27B and 9B on Metal (M1 to M4): the default prompt frame is sized to the 128-row passes these chips run, instead of 512 rows, so the frame, attention and GDN scratch no longer hold room that is never used (#2797, refs #2089).
- Qwen3.8-27B on Metal (rollback bank): the GDN rollback bank is leased from the shared recurrent workspace (#4178).
- Gemma 4 on Metal (memory): each linear's bytes are held once instead of three times (the shard, the lane or row pack and a prompt-chunk copy). The linears are staged from their files during load and the files close after it, with the same bytes and the default prompt-cache budget unchanged (#3350).
- DeepSeek-V4.1 on CUDA (float linears): opt-in
TF_DSV41_FLOAT_LINEAR runs backbone projections kept as float matrices on the plain path. With it unset, such a pack is still refused (#4076).
Docs
- Optimization ledger: rows 34 to 37 are added (FP8 KV prefill waits, Qwen normalization and head, text metadata uploads and greedy readback), each
failed on one GB10 against the 3% served-speed bar with equal tokens, and the SM120 follow-up is recorded as inconclusive (#4054, fixes #3741).
- README build section: a Linux
zig build native compiles for the build machine's CPU, and -Dcpu=baseline or -Dcpu=x86_64_v3 builds a binary for another machine (#3179, #3301).
- Desktop release notes: Desktop's packaged release notes move to
app/CHANGELOG.md, apart from the engine's, with a link each way (#4284).
Fixes
- Flash Next on CUDA (
--tp 2 start-up): a rank now waits for its peer to finish loading before the first collective, up to 10 minutes, instead of failing at the collective's 30 s bound. A peer that fails or closes its link ends the wait with RankDisconnected (#2972, fixes #2970).
- Qwen3.8-27B (DSpine drafter):
tencent/Qwen3.8-27B-DSpine-b16, whose current revision names its architecture DSpineDraftModel, loads again on Metal and CUDA instead of stopping with UnsupportedDraftConfig. Any other name is still refused (#3959, fixes #3819).
- CUDA (capabilities):
capabilities --json now advertises modelopt-mixed and modelopt-nvfp4 for the dense Qwen3.5 family, matching the ModelOpt exports its loader serves (#4192). The dense Qwen3.8-27B row advertises them too, modelopt-mixed (nvidia/Qwen3.8-27B-NVFP4) and modelopt-nvfp4, beside mlx-q4g64 and exl3, so a client that filters on capabilities --json no longer hides a checkpoint the engine serves (#4150).
- Prompt cache (expiry): states displaced by backend expiry are saved to the eviction-only disk tier, avoiding stale resumptions (fixes #2669).
- MOSS-Image and Step-5 (RoPE tables): the vision encoders take their RoPE frequencies from the shared exact helper. MOSS-Image's bf16 cos and sin tables change only from row 565, so shorter prompts read the same tables, and Step-5's bf16 tables are unchanged.
zig build test-moss-image-rotary-host checks the MOSS-Image tables without a device (#1998, part of #1852).
- Cluster bring-up: bring-up reports the leader, epoch and view it agreed on and counts only acknowledgments for that view, so a rank that finishes after a peer says goodbye no longer reports a smaller view on slow runners (#4135, fixes #4123).
- Windows (CUDA builds): the native CUDA product compiles again. Muse Spark, which needs two TP ranks over the POSIX-only TP fabric, is refused by name on Windows (TP across ranks runs on Linux only) instead of breaking the build, and CI now type-checks the Windows CUDA product (
zig build compile-native-cuda-windows) (#4171, fixes #4105). The Windows Qwen3.5 diagnostic build compiles again too, with named refusals where a POSIX stage or pipe is needed; 5/5 Windows GNU cross-build steps pass (#4112, refs #4105).
Tools and tests
- Kernel contract (coverage): the HIP relu² stream tile is checked at 5, 6 and 8 bits in groups of 64 as well as 4, with gates 1 to 4, a planted control for each wider width, and gate-5 records on the four RDNA chips (#3163, part of #3061). The CUDA contract's NVFP4_NZ5 stress fill holds +5s among its 4s and 6s, its float64 reference reads code 0b1000 as 5, and a second planted control goes red at the sums gate, so that gate covers the +5 code; no kernels change (#3543, refs #2894).
- Kernel contract (nvcc 13.3 records): resource records for 121 launches clear 408 stale lines on five-SM fatbins (#3953). The GLM record for
df_sumn under nvcc 13.3 (40 registers) is the one checked, and a stale generic 33-register entry no longer overrides it (#4291).
- Code: Drama 3's codec weight map, decoder stage plan and decode and encode schedules live once in
codec_net.zig, and the CUDA and Metal backends each bind only their own kernels. A new check-drama-codec-bindings build step, run by zig build test, compiles both bindings on any host, and the waveform and codes are byte-identical to before on the author's CUDA and Metal runs (#2893, refs #2118). The HIP gfx12 WMMA step pieces move from tiles/wmma12_gemm.hpp into a shared header, tiles/wmma12_step.hpp, for the coming gfx12 affine tile; it is only a move, so the kernels' bytes do not change (#3862, refs #2401). Lanes gain an optional normalized-row view of the prompt features, whose null default keeps every existing drafter path as it is (#4132, refs #2485).
- Tools:
tools/comfyui_runtime installs the ComfyUI MOSS sampler on a GB10 to run without a network. It checks the release CLI and the pinned native source, builds only moss-image-cuda, and switches to a new runtime only after integrity and run checks, keeping the active one when setup fails. A separate tool writes fused BF16 checkpoint metadata without changing tensor payloads (#2737).
- Tests (timing): the last real-clock multi-rank bring-up tests run on the lockstep clock, so a busy CI runner cannot change their outcome (#4193). The served-evidence concurrency test pairs its two generated streams in the fake server, so a busy runner can no longer decode them one after the other and turn it red; other conformance fixtures keep their pacing. The Web loopback startup test sets its fixture ready before probing, so its health, model-list and relay checks no longer depend on warm-up timing (#3753, fixes #3752).
- Tests (host): DeepSeek's checkpoint reader in load prefetch makes the same number of allocations on every run, so the allocation-failure host test passes on macOS (#4344). The serve-config fixtures no longer depend on a group-friendly umask, while the permission refusals stay; 39/39 host tests pass under umask 002 and 022 on Linux and macOS (#4115, fixes #4113). The three bounded top-k selection tests run in the fast
test-lanes-topk-host step, with a planted failure confirming they can fail by name (#4095, refs #2460).
- CI and release tooling: the Zig format workflow runs again on pushes and pull requests; two concurrency settings had left its file with a duplicate key, which GitHub refused (#4339). The Zig format and TensorFold Web workflows cancel a pull request's older run when it gets a newer push, so superseded runs no longer queue, and main pushes all run (#3850). The packaged Web stand-in reports the same version as the engine, so the Web checks pass on main again (#4346). The runtime feed's preparation reads GitHub releases with the workflow's read-only token, so shared hosted runners no longer fail it on the unauthenticated rate limit; transport errors name the host, and a failed preparation step names its error instead of only saying that it failed.
What changes by default
- On CUDA, a filling prompt with more than 32,768 rows left shares its steps with running replies, with a decode share of 0.25 and the 512-row chunk hint (#3486). Nemotron and LFM2 MoE keep a prompt-cache state at each message start (#3696), a resumed prompt skips the images its kept state holds (#3164), and Flash Next loads a bf16 language head's draft rows as 4-bit affine groups of 32 (
TF_FLASHNEXT_DRAFT_HEAD=bf16 keeps them as stored) (#3477).
- Each image takes a fixed share of the visual-token budget (1,024 tokens at the defaults), a lone image included (#2717). A streamed reply whose client falls 1,024 tokens behind sits out rounds, and one whose client reads nothing for 60 seconds is cancelled (#3234). Concurrent identical prompts join a leader's fill (#3693).
- On Metal, LFM2-MoE, LFM3, MiniMax-M3 and Qwen3.5-2B fill new prompts a chunk at a time between other streams' decode rounds (#3058, #3088, #3089, #3090). GLM on Macs without tensor units uses the new vectorized prompt, KDA and indexer kernels (#3921). Qwen3.8-27B and 9B on M1 to M4 size the prompt frame to the 128-row passes they run (#2797), a lone sampled Qwen3.8-27B stream ranks its DFlash2 drafts with its own keyed draw (#3978), and Gemma 4 holds each linear's bytes once (#3350).
- Flash Next on an integrated HIP GPU caps its default kept-prompt cache at the memory the streams leave, at most 16 GiB (#3645). Flash Next ranks at
--tp 2 on CUDA wait up to 10 minutes for a peer to load (#2972). MOSS-Image's bf16 RoPE tables change from row 565 (#1998), and Muse Spark is refused by name on Windows (#4171).
- Opt-in only:
serve --execution-witness (#2901, #2956, #2958), --tool-think-close (#4157), TENSORFOLD_PARALLEL (#3685), TENSORFOLD_LINUX_SANDBOX=1 (#3364), TF_DSV41_COPY_DRAFTER=1 (#2638), TF_DSV41_FLOAT_LINEAR (#4076), TF_GLM_DRAFT_SPLIT=1 (#3616), TF_QWEN_KV_HOST=1 (#2637), TF_QWEN27_METAL_KV_PAGES=1 (#3377, #3180), --mtp-drafts above 3 for Nemotron on HIP (#3613) and a DFlash2 --drafter DIR for GLM-5.3-Flash on Metal (#3920).
Thanks
@AbrahamCete, @AIFlowML, @BadAd84, @bikepresentball, @Bizuayeu, @Ch0mpaa, @christopherowen, @CodingEmil, @devopsfortunato, @eXsoR65, @Flyinpiz-dev, @garvrao80, @gilby, @hobbesd, @javierbits, @jayleaton, @jschmied, @kotevcode, @laudney, @MaverickKB, @MiaAI-Lab, @millaguie, @Nipale-ai, @nuk3s, @plotarmordev, @SergiioB, @supernovae, @swbiggart, @tacosta, @ThinkOffApp and @ultradaoto.
|