TensorFold v1.3.3

TensorFold is an open-source local AI inference engine designed to run large language models directly on personal computers and GPU servers. Developed primarily in Zig, it provides native GPU execution through Metal on Apple Silicon and CUDA on NVIDIA hardware, without requiring a Python or MLX installation to serve models.

Unlike tools that simply wrap existing inference frameworks, TensorFold implements its own model execution and GPU kernels. It also provides an API server compatible with popular AI clients, making it possible to use locally hosted models with applications designed for OpenAI or Anthropic services.

The project goes beyond basic model inference by including a desktop workspace, web interface, model management tools, and experimental capabilities for modifying model weights while running.

Features

Native Local LLM Inference

TensorFold runs language models directly on supported hardware using native GPU backends.

Its main advantages include:

  • Metal acceleration for Apple Silicon.

  • CUDA acceleration for NVIDIA GPUs.

  • Native HIP support for selected AMD hardware.

  • Direct model checkpoint loading and tokenization.

  • No Python or MLX installation required for native serving.

  • Configurable context length and generation parameters.

  • Support for concurrent inference requests.

The engine is designed to make local inference more accessible without requiring users to maintain a complicated Python-based machine learning environment.

Hardware support varies by backend and model architecture. Not every model is compatible with every GPU, so users should consult the project's model compatibility information before downloading large checkpoints.

Model Management

TensorFold includes commands for downloading and managing models from Hugging Face and other supported sources.

For example, users can download a model and start an inference server with:

tensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit
tensorfold serve mlx-community/Qwen3.5-2B-MLX-4bit --name local-model

The application also supports existing local checkpoint directories and models already present in the Hugging Face cache.

Model configuration options include context length, maximum output tokens, temperature, sampling parameters, and reasoning settings. This provides flexibility when adapting a model for coding, chat, document processing, or other workloads.

An important consideration is that model weights have their own licenses. TensorFold's Apache 2.0 license does not automatically grant permission to use every model distributed through external repositories.

OpenAI and Anthropic-Compatible API

One of TensorFold's most useful features is its local API server.

It supports several popular API formats, including:

  • OpenAI Chat Completions.

  • OpenAI Responses API.

  • OpenAI Completions.

  • Anthropic Messages API.

This allows compatible applications to communicate with locally hosted models instead of relying on cloud AI providers.

For example, after starting the server, users can test it with:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "local-model",
    "messages": [
      {
        "role": "user",
        "content": "Explain how local LLM inference works."
      }
    ],
    "max_tokens": 256
  }'

The default server address is 127.0.0.1:8080. Authentication options are available for deployments that need API access control.

This compatibility makes TensorFold useful for integrating local models with coding assistants, chat interfaces, and other applications that already support standard AI APIs.

Concurrent Requests and Performance Controls

TensorFold includes controls for managing inference workloads and available GPU memory.

Its serving configuration supports:

  • Parallel inference requests.

  • Automatic concurrency selection.

  • Request queuing.

  • Context-length limits.

  • Prompt caching.

  • Maximum output token limits.

  • Sampling and reasoning configuration.

  • Memory diagnostics.

The automatic parallelism option attempts to determine how many requests can run concurrently based on available resources. Explicit concurrency limits are also supported.

Prompt caching can reduce repeated processing when conversations share common prefixes. Additional options allow users to control memory allocation and retained prompt data.

These features are particularly useful when operating a local inference server for multiple applications or users.

TensorFold Desktop Workspace

TensorFold also provides a desktop application for managing local AI workloads.

The workspace combines several functions:

  • Local model discovery and downloads.

  • Model execution and chat.

  • Hardware recommendations.

  • Live generation metrics.

  • Coding agent management.

  • Document editing.

  • Local inference management.

This provides a more approachable interface for users who prefer graphical applications over terminal-based workflows.

The macOS desktop application is distributed as an Apple Silicon beta. Its mobile clients remain preview software, so users should not expect the same maturity as the native inference engine.

TensorFold Web

For users who prefer a browser interface, TensorFold includes a local web application.

It provides a graphical environment for selecting models, running inference, and chatting with locally hosted models.

The web interface complements the command-line server and desktop application, giving users another way to interact with their local AI environment.

AI Coding Workers

TensorFold's desktop workspace also includes coding workers designed to assist with development tasks.

This extends the application beyond conventional chat-based inference. Users can manage coding-oriented workflows alongside their local models and documents.

The desktop workspace is still evolving, so its agent functionality should be evaluated separately from the more established inference engine.

Living Weights

One of TensorFold's more experimental capabilities is Living Weights.

Instead of keeping newly learned information exclusively in an external database or conversation history, this feature attempts to write selected facts directly into a model's checkpoint files.

Users can provide information through a dedicated API endpoint, and TensorFold processes the material as lessons that modify the model's weights.

The project also provides a 3D visualization interface showing learned facts and their associated model components.

However, Living Weights is explicitly experimental. The project documents limitations involving interference between related facts, declining recall after learning multiple facts, and reduced decoding performance on some hardware.

It is an interesting research feature, but it should not be considered a replacement for established retrieval-augmented generation systems or a reliable persistent memory solution.

Download TensorFold v1.3.3 - Software Mirrors

TensorFold v1.3.3 for macOS

tensorfold-1.3.3-macos-arm64.tar.gz | 7.82 MB

TensorFold v1.3.3 for Linux

tensorfold-1.3.3-linux-x86_64.tar.gz | 54.13 MB

tensorfold-1.3.3-linux-aarch64.tar.gz | 53.61 MB

TensorFold v1.3.3 Source Code

TensorFold v1.3.3 Source code (zip)

TensorFold v1.3.3 Source code (tar.gz)

TensorFold v1.3.3 Release Notes:

TensorFold 1.3.3

1.3.3 makes GLM faster. On CUDA, GLM-5.3-Flash drafted decode after a 1,040,384-token prompt is 13.8% faster at TP=4. On Macs without tensor units, GLM cold prompts take 5.17% to 5.68% less time. It adds picture generation for MOSS-Image on CUDA through POST /v1/images/generations, GLM-OCR on HIP, DFlash2 drafting for GLM-5.3-Flash on Metal, EXL3 mcg experts at 1 to 8 bits for GLM-5.3-Flash on CUDA, structured output for DeepSeek-V4.1 on CUDA, and Idol's image encoder on Metal. The server gains an opt-in execution witness, a prompt-cache flush route, per-request deadlines (deadline_s), structural tags, deferred and namespace tools, and TENSORFOLD_PARALLEL. By default, long prompts on CUDA now share their steps with running replies, each image takes a fixed share of the visual-token budget, and LFM2-MoE, LFM3, MiniMax-M3 and Qwen3.5-2B on Metal fill new prompts between other streams' decode rounds. The speed changes below keep the same tokens, bits or bytes where their bullets say so.

Speed

  • GLM-5.3-Flash on CUDA (long-context decode): drafted decode after a 1,040,384-token prompt takes its DSA index scores from tf_glm_index_scores. It is 13.8% faster at TP=4 (172.39 against 151.51 tok/s median), with byte-identical scores and the same tokens. Shorter windows keep tf_glm_scores (#3689).
  • GLM-5.3-Flash on CUDA (TP=3, 5, 6 or 7): at these widths a rank's fused KDA input is not a multiple of eight columns. The stack is now padded at load to its own width rounded up to eight columns and stays row-major bf16, so prompt windows take the staged B16 tiles with every byte as before. TF_GLM_KDA_PAD=1 still tiles the stack for glm_mm (#3697, for #3516).
  • GLM-5.3-Flash on CUDA (split drafter): TF_GLM_DRAFT_SPLIT=1 splits the DFlash2 drafter's MLP across three ranks or more where each share is 256-aligned, without TF_ROCE=1. The ranks must set it alike, two ranks split as before, and unset changes nothing. On three GB10s whose gathers stay on NCCL, decode with --drafter is about 5% faster at one stream (four streams within their start-to-start spread), with the same replies and the same tokens a round on the measured prompts. As with TF_ROCE=1, the split drafter adds its down projection's rank shares in another order, so its proposals can differ at near-ties (#3616, for #3581).
  • GLM on Metal before M5 (prompts): vectorized prompt, KDA and indexer kernels on Macs without tensor units cut cold prompt time 5.17% to 5.68% (2K to 64K tokens), with 0/64 token differences. (#3921, refs #3152).
  • GLM on Metal (DFlash2): decoding skips inactive draft rows and their tap capture, and the tap mean writes into its ring in place. This cuts sampling overhead without changing verified tokens (#4257).
  • DeepSeek-V4.1 on Metal (multi-stream decode): a decode round over several streams runs each per-stream stage (window key, compressed rows and index keys, ratio-2 pooling, decode attention, index scores, candidate blocks and the top 512) as one dispatch over a table of the streams' state. There is one barrier between stages instead of one a stream, and the bits are the same (#2954, refs #2984).
  • DeepSeek-V4.1 on Metal (sampling and drafts): DSpark drafts are selected over the whole vocabulary on the GPU with an adaptive block size (#3405). Sampling keeps its scratch between picks and scans greedy logits in vectors, choosing the same IDs (#3288).
  • DeepSeek-V4.1 on CUDA (copy drafter): opt-in TF_DSV41_COPY_DRAFTER=1 turns on the lookup planner with a copy drafter that ranks matches within a length band by how much continuation they offer. It offers 8 drafts, widens to 15 after a round that keeps every copy, and offers a copy only when it is expected to keep at least as many tokens as the other drafter. It is refused when TF_DSV41_LOOKUP is off or blank, and unset leaves TF_DSV41_LOOKUP_PLAN=1 as it was (#2638).
  • Flash Next on CUDA (draft head): a bf16 language head's draft rows load as 4-bit affine groups of 32 by default. TF_FLASHNEXT_DRAFT_HEAD=bf16 keeps them as stored, and a malformed value stops the start by name. Replies are unchanged because drafts are verified (#3477, refs #1870).
  • Flash Next on CUDA (--tp 2 control link): a rank's control-link wait polls with a 50 µs timeout for its first 50 ms and then blocks as before, so drafted rounds pick up the next header sooner. Replies are unchanged (#2971).
  • Flash Next on HIP (prompts and load): on one GPU, a prompt's first token goes out before its last chunk's kept prompt states are saved, and they are saved before any later step writes the stream's caches. Kept-state copies pass through one pinned 32 MiB buffer, and on Linux each kept state sits on small pages of its own (#3647). With the prompt cache on, a prompt chunk runs on through the cache marks its side buffers can hold, writing each marked row's state aside, instead of ending at the first mark (#3646). The first stream's caches are made at load, so the first prompt no longer waits on allocating them. Each layer's weights are read past the page cache by threads, with the next layer's read under way, and the n-gram gather spreads over all of the pool's parts instead of at most 32 (#3644).
  • Flash Next on XPU: prompt and decode passes run on Xe2 DPAS tile kernels, with prompt chunks sized to the solo stream and MTP chains that stop when draft confidence falls. Replies stay bit-exact across chunk sizes (#3603, refs #3583).
  • Qwen3.8-27B on Intel GPUs (attention): decode and verify attention (attn_decode) runs on a (rows, heads, slices) grid, so the tree rows of one key slice run back to back. Each work group's arithmetic is unchanged, so replies keep their tokens. The XPU kernel contract holds attn_decode and attn_merge to the previous kernels' bytes, with a planted control for the launch order (#2595, refs #2543).
  • Qwen3.8-27B on Metal (sampled drafts): a lone sampled stream's DFlash2 draft tree ranks its candidates with the stream's own keyed draw, so the drafts follow what the target samples. The target still decides every token, and greedy and multi-stream rounds are unchanged (#3978).
  • Qwen 3.5/3.6/3.8 on CUDA (prompt chunks): a prompt chunk before the last that nothing reads (no last-layer tap, learning, MLP or pocket reader, prompt int8 path or image) stops its last layer after its keys, values and RoPE. It skips that layer's query projection, attention, output projection, MLP and head, with the same reply tokens. The final chunk is unchanged (#3060, fixes #2941).
  • CUDA (images on resume): a resumed prompt encodes only the images its kept state does not hold (Gemma 4, Qwen3.5 and Flash Next on one rank), so a follow-up turn skips the vision tower for images the restored prefix already holds. A resume never cuts an image, and Flash Next at --tp 2 keeps encoding every image as before (#3164).
  • CUDA (long prompts beside replies): a long prompt now shares its steps with running replies by default. A filling prompt with more than 32,768 rows left takes a decode share of 0.25 and the 512-row chunk hint, as --decode-share 0.25 does, while a shorter prompt fills whole as before. --decode-share and TENSORFOLD_PREFILL_SHARE keep their meanings (#3486, refs #3397).
  • Nemotron and LFM2 MoE on CUDA (message starts): the prompt cache keeps a state at each message start by default. The next request in a conversation resumes from where its last message began instead of reading the conversation again, within the existing prompt-cache budget and slots (#3696, part of #3617).
  • Nemotron on HIP (MTP drafts): at --tp 1, --mtp-drafts takes up to 15, widening a lone stream's verify window up to 16 rows. Unset or at 3 or fewer, the window stays at four rows. Rounds shared by several streams chain at most three drafts a stream, and above --tp 1 more than three is refused by name (#3613, refs #3394).
  • Metal (new prompts beside replies): LFM2-MoE (#3058), LFM3 (#3088, part of #3021), MiniMax-M3 (#3089, part of #3021) and Qwen3.5-2B (#3090, refs #3021) fill a new prompt a chunk at a time between other streams' decode rounds, instead of stalling those replies for the whole prefill. On LFM3, MiniMax-M3 and Qwen3.5-2B, chunks are cut as before, so replies are unchanged, and a cancel keeps the state at the last chunk's end. On MiniMax-M3, a stepping drafter absorbs each chunk before the next. LFM2-MoE declares prefill_step and cancel_prefill beside reports_fill.
  • MiniMax-M3 on Metal (drafters): the DFlash2 and DSpark drafters hold up to eight streams' blocks in one pass and one wait, instead of a pass per stream, and absorb no longer waits for the GPU. Drafted replies are unchanged (#3121).
  • LFM2-MoE and LFM3 on Metal (rounds): a keep that drops no rows skips the conv-state and logits copy in Cache.keep, since commit and run already wrote those bytes; a partial keep still restores through a new take() (#3078). On LFM3, every stream's qk in a shared round runs in one dispatch instead of once per stream per attention layer, with the same arithmetic per row, so replies are unchanged (#3079).
  • Spark 2.5 on CUDA: gate and up load as one stacked weight and decode in one launch on every CUDA chip instead of two launches, with the bits unchanged. Only 4-bit checkpoints take this path (#3130).
  • HIP (routed prompts): a routed prompt plan keeps the stacked shared expert whole up to the GEMM tile instead of cutting it into 8-row items. On gfx1151, a prompt pass of 33 rows or more takes the GEMM tile for every expert. Decode, lane rounds and the MTP draft head keep the 8-row tile, and outputs keep their bytes (#3521, fixes #3161).
  • Lane host (identical prompts): concurrent identical prompts join a leader that is filling a prompt state it will keep, so they reuse its whole fill and return the same tokens they would alone. A prompt whose state would not be kept, a zero budget, the leader's cancellation or a timeout still runs on its own (#3693, refs #3397, #3487).
  • Tokenizer: a request gap with no normalized added spelling reuses the tokenizer's memo instead of re-encoding cold. Long multi-message requests on tokenizers with normalized added tokens (DeepSeek-style ones) cost less per call. Gaps carrying a spelling keep the mask, and every id is unchanged (#2844).

New

  • MOSS-Image on CUDA: the checkpoint's Qwen3.5 prompt encoder and VAE decoder run beside the transformer, so the native server serves pictures through POST /v1/images/generations. The same request twice gives the same bytes (#2281, part of #1573).
  • OCR on HIP: GLM-OCR on native HIP serves POST /v1/ocr. It returns normalized page Markdown with pipe or spanned-HTML tables and $/$$ math, blocks: [] because it has no geometry, and named refusals for header and footer extraction (#2572).
  • GLM-5.3-Flash on Metal (DFlash2): it can draft with a published DFlash2 checkpoint behind --drafter DIR (and --drafter-bits). Drafted replies equal plain ones, and the default stays MTP (#3920).
  • EXL3 mcg checkpoints on CUDA: GLM-5.3-Flash loads EXL3 mcg routed experts at any whole-bit width from 1 to 8, mixed by projection within a layer, where only 4-bit was admitted, and its prompt expert kernels gain a 3-bit form; the 4-bit path is unchanged (#2094, refs #2093). Qwen3.5 admits EXL3 checkpoints with the mcg codebook marker alongside mul1 ones, and unknown markers are still refused as an unsupported quantization (#3412).
  • DeepSeek-V4.1 on CUDA (structured output): in the dsv41-cuda source build, structured output runs on the lane core's token grammars with drafts on. This covers response_format, the guided_* fields, tool_choice: "required" or a named function, and strict tools under tool_choice: "auto". A grammar's stop token ends the reply with finish_reason stop, even with ignore_eos (#3026, closes #1664, from #1665).
  • Qwen3.5/3.8 and Flash Next on CUDA (quantized vision towers): --vision loads a tower whose linears ModelOpt stores as MXFP8 or NVFP4, as nvidia/Qwen3.8-Flash-Next-FP4-Local does, where it stopped at the first quantized linear. They are decoded to bf16 on the host at load, one linear at a time, so the tower runs in bf16 as before. A floating-point tower is still read as stored (#3727).
  • Drama 3 on CUDA (NVFP4): compact NVFP4 checkpoints load as stored, held experts are staged in slot order, and routing is masked to their original ids (#4144).
  • Nemotron 3.5 Lightning on HIP (MTP head): the MTP head file is read at the width its own metadata states, so it drafts beside an oQ checkpoint of any width. A head at a width the launches do not read is refused by name (#3251, part of #3061).
  • Idol on CUDA (decoders): Idol's octree and Gaussian decoders run on CUDA beside the Metal decoders, driven by the same tf-idol-decode case driver and gates. The stem, to_kv and the head norms run in fp32, input_layer and the head in true fp32 (no TF32), and the blocks through the denoiser's CUDA linears and norms. Attention keeps fp32 scores and sums and rounds P to nearest fp16 as torch's CUDA kernels do, and runs fp32 throughout at 8 query rows or fewer. The six kernels and 13 controls are registered with the kernel contract (tf-cuda-test idol-decoder-contract). It admits sm_121 (GB10) and sm_120 and refuses other GPUs. It compiles for both and its host tests pass, but it has not yet run on a GPU (#1858, after #1773 and #1767, for #1757).
  • Idol on Metal (image encoder): Idol's FLUX.2 VAE image encoder runs natively in fp32, from one 1,024 x 1,024 picture to the denoiser's 4,101 condition rows, with the posterior's noise as an input. It takes 0.185 s an encode on an M5 Ultra, against 0.095 s for the SDK's bf16 encoder on torch MPS, and lies far closer to a float64 run: worst row 5.1e-5 against the SDK's 2.7e-2. tf-idol-vae gates it from a case that tools/zig/idol_case.py vae records, with the ungated FLUX.2-klein-4B weights (#1774, after #1766, for #1757).
  • MiniMax-H3-Image on Metal: tf-h3-image compares the Metal transformer steps with exported picture cases (#2766). It now runs the transformer (MiniMax H3's, without the audio layers, on M5 tensor units) from a picture case through its six distilled steps, each with its own output head, beside the engine. The VAE's rules (config, tiles and blend, latent statistics, pixels) run on the host, core/picture gains Pillow's Lanczos-3 resampling, and clip cases run as before (#2775). tf-h3-image-encoder turns an edit's reference pictures into their clean rows with the VAE's encoder in fp32 on any Apple GPU, with a selftest of each kernel against a float64 host reference; the reference rules (sizes, crops, grids, encoder layers, clean and anchor rows) run on the host (#2778, on #2775).
  • Ling-3.0-tiny on Metal (diagnostic): a bf16 forward diagnostic runs all 24 layers of the published checkpoint on kernels that already exist, with no new GPU kernel source. zig build test-ling-forward, given TF_LING_CHECKPOINT and a Python with torch and safetensors (TF_LING_PYTHON), checks whole, chunked and resumed runs byte for byte and reports the distance from float64. Nothing is served yet (#3919, part of #1247).

Server and API

  • Execution witness: serve --execution-witness (off by default) adds GET /v1/execution/witness under the read key scope. It returns a snapshot naming the running binary, the checkpoint's text files, the serve defaults and TF_* settings, the engine's epoch and a per-process instance, or unknown with its reason. It needs an API key or --unix-socket and hashes only after the server is ready (#2901, refs #2741). With the flag on, "execution_witness": true on a chat completion or completion adds an execution_witness object to the reply (on a stream, its finish chunk). It holds the witness read when the engine takes the request and again at the end, the SHA-256 of the prompt ids, the resolved request and the reply tokens, the finish reason and a binding over them, or unknown with a reason (#2958, refs #2741). Native Qwen3.5-2B on CUDA (one rank, MLX affine 4-bit, text only) attests its path: once the server is ready it reads every checkpoint shard, after stamping the files before the load, and binds them with the kernel images, GPU, driver, family facts and resolved profile values. Other engines still read unsupported (#2956, refs #2741).
  • Image token budget: every image takes a fixed share of the visual-token budget, --vision-image-tokens / --vision-max-images (1,024 tokens at the defaults), whatever the request's image count. A turn that adds an image no longer re-sizes earlier ones, so the prompt cache can resume from the first image. A lone image gets that share, not the whole budget, and a budget that leaves no share is refused by name, naming the image and both flags (#2717, fixes #2510).
  • Prompt cache flush: POST /v1/prompt_cache/flush drops the prompt states the server keeps for resuming conversations, in RAM and on disk when it keeps them there (--prompt-cache-disk, --learn-conversations or --spill-gib), once no request is running, so replies in flight are unchanged. It needs a full or [admin] key when API keys are on (a loopback caller without keys). It answers 503 if requests have not ended within 10 minutes and 501 on an engine without the prompt cache, and it leaves --learn's shared-prefix states on disk (#3210, fixes #3183).
  • Request deadlines: chat and legacy completions requests take deadline_s (seconds, more than 0 and at most 1,000,000; any other value is refused before the engine). At that time the server cancels the engine itself, and the reply ends with finish_reason: "deadline", keeping every token the engine gave and its usage. A reply the engine ended first, or a cancel that came first, keeps its own ending, and without the field nothing changes (#3438).
  • Structural tags: response_format: {"type": "structural_tag", ...} on the native server keeps the reply free text, while each structure's begin opens a region that must be JSON matching its schema and close with its end, with optional triggers. Past 64 structures or triggers, 256-byte tags or the 512 KiB grammar limit, the request is refused with HTTP 400 before any stream starts (#3208).
  • Deferred tools on /v1/messages: /v1/messages and its count_tokens accept deferred tools. A tool with defer_loading: true stays out of the prompt until a tool_reference block in a tool_result loads it. It is rendered by a template that reads both fields (GLM-5.3's), or else as the tool's name, description and parameters in text. A reference to a tool the request does not offer, a non-boolean defer_loading or all tools deferred is refused, and requests without these fields are unchanged (#3212, fixes #3115).
  • Namespace tools on /v1/responses: one level of namespace tools is accepted. Each function is offered to the model as <namespace>__, and its calls come back with name and namespace, whole, streamed, stored and over WebSocket. Requests without namespace tools are unchanged (#3610, fixes #3266).
  • Thinking before tool calls: opt-in --tool-think-close bounds thinking for required or named tool calls, so the call has room in max_tokens. The flag defaults off and keeps smaller thinking budgets (#4157).
  • Default lanes: TENSORFOLD_PARALLEL sets the default lanes when serve has no --parallel: a whole number 1 to 4096 or auto. The flag wins when both are given, a blank value keeps the default, and any other value is refused by name at startup (#3685, for #3684).
  • Slow stream readers: a streamed reply whose client falls 1,024 tokens behind sits out rounds until the reader is within 512 again. One whose client reads nothing for 60 seconds is cancelled like a disconnect, so a stalled reader stops holding its lane. The text a reader receives is unchanged (#3234).
  • Qwen3.8 Flash Next on Metal (cancelled prompts): a request cancelled while its prompt is still filling keeps the prompt state it had already landed at a resumable cut, so a follow-up with the same prefix resumes from there instead of starting over (#3657, refs #3618).

Desktop and Web

  • App on Linux (sandbox): with TENSORFOLD_LINUX_SANDBOX=1 and bubblewrap installed, the app's commands run in a bubblewrap sandbox. It keeps every repository's config, config.worktree, hooks and info read-only, nested repositories included, and freezes worktree .git pointers. A workspace with no repository, a worktree whose gitdir or shared directory lies outside the workspace, a path the sandbox cannot inspect, or a walk past 32 levels or 50,000 directories is refused with code 126 before anything runs. Without the setting, Linux commands stay refused as before (#3364, fixes #1693).
  • App (hardening): a packaged build ignores VITE_DEV_SERVER_URL (#3049). Explore and Oracle child agents refuse a tool outside the list they were given, so a read-only run stays read-only (#3040). Only the owner token resolves a tool approval on the remote-control server (#3034).
  • Mobile: a pairing link opened from outside the app asks first and names its host (#3036).
  • Web: Qwen3.6-35B-A3B is listed for CUDA only, and starting it on a machine without CUDA is refused before anything launches, so a Mac is no longer pointed at the Python engine for it (#3738, refs #3710).

Platforms and settings

  • Flash Next on HIP (integrated GPUs): on an integrated GPU that shares the host's pages (no firmware carve-out), the default kept-prompt cache takes no more than the memory the streams leave, at most 16 GiB and none when nothing is left. An explicit --prompt-cache-gib still sets it (#3645).
  • CUDA (RoCE with TF_ROCE=1): the RoCE all-gather pairs each peer's stripes with the HCA on that peer's IPv4 /24 (the rule fabric/tp uses, now one copy for both) when every rank's plan holds. Three GB10s cabled as a switchless ConnectX-7 triangle now take the RDMA path instead of falling back to NCCL. A switch or two ranks keep the same-numbered pairs, and a TF_ROCE_HCA of more than two names is refused by name instead of cut to two (#3518, fixes #3411).
  • Qwen dense and MoE hybrids on CUDA (KV in host memory): opt-in TF_QWEN_KV_HOST=1 keeps the KV pages in mapped pinned host memory, with the state and verify runs on the device, so a long window can fit on a card with little room left beside the weights. It needs TF_QWEN_KV_PAGES, TF_QWEN_KV_PAGES_ALL=1, --parallel 1 and a discrete GPU, and takes an optional TF_QWEN_KV_HOST_GIB cap. It refuses spill, disk snapshots and TF_QWEN_KV_SHARE by name. It trades speed for room, since attention reads the pages over the device interconnect; unset, the plan and every path stay as they were (#2637).
  • Qwen3.8-27B on Metal (TF_QWEN27_METAL_KV_PAGES=1): a stream that opens a slot is promised the KV pages of its prompt and one widest pass, instead of every slot being promised its whole window at load. A pass past that promises the rest before it takes a page, and fails with LaneMemory before any page is taken when the pool cannot meet it. A stream that leaves, a failed or requeued prompt pass, a reset and a move to slot 0 give back the slot's pages and promise (#3377, refs #1293). Macs with tensor units (M5) now take the flag instead of refusing it, and a paged prompt pass finds each 512-key chunk's two pages once instead of reading the page table at every 32-key step, with every token and cache row unchanged (#3180, part of #1293). Without the flag nothing changes.
  • Qwen3.8-27B and 9B on Metal (M1 to M4): the default prompt frame is sized to the 128-row passes these chips run, instead of 512 rows, so the frame, attention and GDN scratch no longer hold room that is never used (#2797, refs #2089).
  • Qwen3.8-27B on Metal (rollback bank): the GDN rollback bank is leased from the shared recurrent workspace (#4178).
  • Gemma 4 on Metal (memory): each linear's bytes are held once instead of three times (the shard, the lane or row pack and a prompt-chunk copy). The linears are staged from their files during load and the files close after it, with the same bytes and the default prompt-cache budget unchanged (#3350).
  • DeepSeek-V4.1 on CUDA (float linears): opt-in TF_DSV41_FLOAT_LINEAR runs backbone projections kept as float matrices on the plain path. With it unset, such a pack is still refused (#4076).

Docs

  • Optimization ledger: rows 34 to 37 are added (FP8 KV prefill waits, Qwen normalization and head, text metadata uploads and greedy readback), each failed on one GB10 against the 3% served-speed bar with equal tokens, and the SM120 follow-up is recorded as inconclusive (#4054, fixes #3741).
  • README build section: a Linux zig build native compiles for the build machine's CPU, and -Dcpu=baseline or -Dcpu=x86_64_v3 builds a binary for another machine (#3179, #3301).
  • Desktop release notes: Desktop's packaged release notes move to app/CHANGELOG.md, apart from the engine's, with a link each way (#4284).

Fixes

  • Flash Next on CUDA (--tp 2 start-up): a rank now waits for its peer to finish loading before the first collective, up to 10 minutes, instead of failing at the collective's 30 s bound. A peer that fails or closes its link ends the wait with RankDisconnected (#2972, fixes #2970).
  • Qwen3.8-27B (DSpine drafter): tencent/Qwen3.8-27B-DSpine-b16, whose current revision names its architecture DSpineDraftModel, loads again on Metal and CUDA instead of stopping with UnsupportedDraftConfig. Any other name is still refused (#3959, fixes #3819).
  • CUDA (capabilities): capabilities --json now advertises modelopt-mixed and modelopt-nvfp4 for the dense Qwen3.5 family, matching the ModelOpt exports its loader serves (#4192). The dense Qwen3.8-27B row advertises them too, modelopt-mixed (nvidia/Qwen3.8-27B-NVFP4) and modelopt-nvfp4, beside mlx-q4g64 and exl3, so a client that filters on capabilities --json no longer hides a checkpoint the engine serves (#4150).
  • Prompt cache (expiry): states displaced by backend expiry are saved to the eviction-only disk tier, avoiding stale resumptions (fixes #2669).
  • MOSS-Image and Step-5 (RoPE tables): the vision encoders take their RoPE frequencies from the shared exact helper. MOSS-Image's bf16 cos and sin tables change only from row 565, so shorter prompts read the same tables, and Step-5's bf16 tables are unchanged. zig build test-moss-image-rotary-host checks the MOSS-Image tables without a device (#1998, part of #1852).
  • Cluster bring-up: bring-up reports the leader, epoch and view it agreed on and counts only acknowledgments for that view, so a rank that finishes after a peer says goodbye no longer reports a smaller view on slow runners (#4135, fixes #4123).
  • Windows (CUDA builds): the native CUDA product compiles again. Muse Spark, which needs two TP ranks over the POSIX-only TP fabric, is refused by name on Windows (TP across ranks runs on Linux only) instead of breaking the build, and CI now type-checks the Windows CUDA product (zig build compile-native-cuda-windows) (#4171, fixes #4105). The Windows Qwen3.5 diagnostic build compiles again too, with named refusals where a POSIX stage or pipe is needed; 5/5 Windows GNU cross-build steps pass (#4112, refs #4105).

Tools and tests

  • Kernel contract (coverage): the HIP relu² stream tile is checked at 5, 6 and 8 bits in groups of 64 as well as 4, with gates 1 to 4, a planted control for each wider width, and gate-5 records on the four RDNA chips (#3163, part of #3061). The CUDA contract's NVFP4_NZ5 stress fill holds +5s among its 4s and 6s, its float64 reference reads code 0b1000 as 5, and a second planted control goes red at the sums gate, so that gate covers the +5 code; no kernels change (#3543, refs #2894).
  • Kernel contract (nvcc 13.3 records): resource records for 121 launches clear 408 stale lines on five-SM fatbins (#3953). The GLM record for df_sumn under nvcc 13.3 (40 registers) is the one checked, and a stale generic 33-register entry no longer overrides it (#4291).
  • Code: Drama 3's codec weight map, decoder stage plan and decode and encode schedules live once in codec_net.zig, and the CUDA and Metal backends each bind only their own kernels. A new check-drama-codec-bindings build step, run by zig build test, compiles both bindings on any host, and the waveform and codes are byte-identical to before on the author's CUDA and Metal runs (#2893, refs #2118). The HIP gfx12 WMMA step pieces move from tiles/wmma12_gemm.hpp into a shared header, tiles/wmma12_step.hpp, for the coming gfx12 affine tile; it is only a move, so the kernels' bytes do not change (#3862, refs #2401). Lanes gain an optional normalized-row view of the prompt features, whose null default keeps every existing drafter path as it is (#4132, refs #2485).
  • Tools: tools/comfyui_runtime installs the ComfyUI MOSS sampler on a GB10 to run without a network. It checks the release CLI and the pinned native source, builds only moss-image-cuda, and switches to a new runtime only after integrity and run checks, keeping the active one when setup fails. A separate tool writes fused BF16 checkpoint metadata without changing tensor payloads (#2737).
  • Tests (timing): the last real-clock multi-rank bring-up tests run on the lockstep clock, so a busy CI runner cannot change their outcome (#4193). The served-evidence concurrency test pairs its two generated streams in the fake server, so a busy runner can no longer decode them one after the other and turn it red; other conformance fixtures keep their pacing. The Web loopback startup test sets its fixture ready before probing, so its health, model-list and relay checks no longer depend on warm-up timing (#3753, fixes #3752).
  • Tests (host): DeepSeek's checkpoint reader in load prefetch makes the same number of allocations on every run, so the allocation-failure host test passes on macOS (#4344). The serve-config fixtures no longer depend on a group-friendly umask, while the permission refusals stay; 39/39 host tests pass under umask 002 and 022 on Linux and macOS (#4115, fixes #4113). The three bounded top-k selection tests run in the fast test-lanes-topk-host step, with a planted failure confirming they can fail by name (#4095, refs #2460).
  • CI and release tooling: the Zig format workflow runs again on pushes and pull requests; two concurrency settings had left its file with a duplicate key, which GitHub refused (#4339). The Zig format and TensorFold Web workflows cancel a pull request's older run when it gets a newer push, so superseded runs no longer queue, and main pushes all run (#3850). The packaged Web stand-in reports the same version as the engine, so the Web checks pass on main again (#4346). The runtime feed's preparation reads GitHub releases with the workflow's read-only token, so shared hosted runners no longer fail it on the unauthenticated rate limit; transport errors name the host, and a failed preparation step names its error instead of only saying that it failed.

What changes by default

  • On CUDA, a filling prompt with more than 32,768 rows left shares its steps with running replies, with a decode share of 0.25 and the 512-row chunk hint (#3486). Nemotron and LFM2 MoE keep a prompt-cache state at each message start (#3696), a resumed prompt skips the images its kept state holds (#3164), and Flash Next loads a bf16 language head's draft rows as 4-bit affine groups of 32 (TF_FLASHNEXT_DRAFT_HEAD=bf16 keeps them as stored) (#3477).
  • Each image takes a fixed share of the visual-token budget (1,024 tokens at the defaults), a lone image included (#2717). A streamed reply whose client falls 1,024 tokens behind sits out rounds, and one whose client reads nothing for 60 seconds is cancelled (#3234). Concurrent identical prompts join a leader's fill (#3693).
  • On Metal, LFM2-MoE, LFM3, MiniMax-M3 and Qwen3.5-2B fill new prompts a chunk at a time between other streams' decode rounds (#3058, #3088, #3089, #3090). GLM on Macs without tensor units uses the new vectorized prompt, KDA and indexer kernels (#3921). Qwen3.8-27B and 9B on M1 to M4 size the prompt frame to the 128-row passes they run (#2797), a lone sampled Qwen3.8-27B stream ranks its DFlash2 drafts with its own keyed draw (#3978), and Gemma 4 holds each linear's bytes once (#3350).
  • Flash Next on an integrated HIP GPU caps its default kept-prompt cache at the memory the streams leave, at most 16 GiB (#3645). Flash Next ranks at --tp 2 on CUDA wait up to 10 minutes for a peer to load (#2972). MOSS-Image's bf16 RoPE tables change from row 565 (#1998), and Muse Spark is refused by name on Windows (#4171).
  • Opt-in only: serve --execution-witness (#2901, #2956, #2958), --tool-think-close (#4157), TENSORFOLD_PARALLEL (#3685), TENSORFOLD_LINUX_SANDBOX=1 (#3364), TF_DSV41_COPY_DRAFTER=1 (#2638), TF_DSV41_FLOAT_LINEAR (#4076), TF_GLM_DRAFT_SPLIT=1 (#3616), TF_QWEN_KV_HOST=1 (#2637), TF_QWEN27_METAL_KV_PAGES=1 (#3377, #3180), --mtp-drafts above 3 for Nemotron on HIP (#3613) and a DFlash2 --drafter DIR for GLM-5.3-Flash on Metal (#3920).

Thanks

@AbrahamCete, @AIFlowML, @BadAd84, @bikepresentball, @Bizuayeu, @Ch0mpaa, @christopherowen, @CodingEmil, @devopsfortunato, @eXsoR65, @Flyinpiz-dev, @garvrao80, @gilby, @hobbesd, @javierbits, @jayleaton, @jschmied, @kotevcode, @laudney, @MaverickKB, @MiaAI-Lab, @millaguie, @Nipale-ai, @nuk3s, @plotarmordev, @SergiioB, @supernovae, @swbiggart, @tacosta, @ThinkOffApp and @ultradaoto.

Performance and Compatibility

TensorFold is built with Zig and uses native GPU execution rather than depending on a Python-based inference stack for its serving engine.

This architecture reduces the number of runtime dependencies and gives the project direct control over model execution, memory management, and GPU kernels.

Performance depends on the model architecture, quantization format, GPU memory, context length, and available hardware acceleration.

Apple Silicon: TensorFold uses Metal for GPU execution on supported Apple Silicon Macs. Unified memory can be useful when running larger models, although the available memory must accommodate model weights, context data, and concurrent requests.

NVIDIA GPUs: The Linux release supports CUDA-based inference. Its documented Linux release requirements include an NVIDIA driver version R580 or newer and CUDA 13.0 compatibility. Supported GPU architectures and model combinations vary.

AMD GPUs: Native HIP support is available for selected hardware and models. The project documents specific qualification results rather than guaranteeing compatibility with every AMD graphics card.

The project also advertises Vulkan support, but users should verify the actual backend and model compatibility for their target hardware rather than assuming every advertised backend supports every model.

TensorFold supports Linux and macOS for its documented native workflows, with additional Windows support through its native Windows build documentation.

System Requirements

macOS

  • Apple Silicon Mac.

  • Compatible Metal GPU support.

  • Sufficient unified memory for the selected model.

  • Homebrew for the recommended installation method.

Linux

  • 64-bit Linux system.

  • glibc 2.28 or newer for the documented NVIDIA release archives.

  • NVIDIA GPU and R580-or-newer driver for the documented CUDA release.

  • CUDA 13.0-compatible runtime environment.

  • Sufficient GPU or unified memory for the selected model.

Windows

  • Supported Windows environment.

  • Compatible NVIDIA GPU and driver for the CUDA workflow.

  • Follow the project's native Windows build and installation documentation.

There is no universal minimum RAM requirement because memory consumption depends heavily on the model, quantization, context size, and concurrency settings.

Pros and Cons

Pros

  • Free and open-source under Apache 2.0.

  • Native inference engine written in Zig.

  • Metal acceleration for Apple Silicon.

  • CUDA support for NVIDIA GPUs.

  • Native HIP support for selected AMD hardware.

  • Does not require Python or MLX for native serving.

  • Supports OpenAI-compatible and Anthropic-compatible APIs.

  • Includes model downloading and management.

  • Supports concurrent requests and prompt caching.

  • Provides desktop and browser-based interfaces.

  • Includes performance diagnostics and memory controls.

  • Experimental Living Weights functionality.

Cons

  • Hardware and model compatibility vary considerably.

  • NVIDIA Linux releases require a relatively recent driver.

  • Large models require substantial memory.

  • Some desktop and mobile functionality remains in beta or preview.

  • Living Weights is experimental and has documented limitations.

  • The project has a learning curve for users unfamiliar with local LLM deployment.

  • Not every model available through Hugging Face is compatible with its native engine.

How to Install

macOS

The recommended installation method for Apple Silicon is Homebrew:

brew install ashhart/tensorfold/tensorfold

Download a model and start the server:

tensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit

tensorfold serve mlx-community/Qwen3.5-2B-MLX-4bit \
  --name local-model \
  --context 8192 \
  --no-thinking

Linux with NVIDIA GPU

Download the appropriate release archive for your CPU architecture. The project provides separate archives for x86_64 and ARM64 systems.

For example:

V=1.2.8

curl -fLO "https://github.com/ashhart/TensorFold/releases/download/v$V/tensorfold-$V-linux-x86_64.tar.gz"

tar -xzf "tensorfold-$V-linux-x86_64.tar.gz"

export PATH="$PWD/tensorfold-$V-linux-x86_64/bin:$PATH"

Download a compatible model and start inference:

tensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit

tensorfold serve mlx-community/Qwen3.5-2B-MLX-4bit \
  --name local-model \
  --context 8192 \
  --no-thinking

Check the latest release instructions before installation because package names, supported architectures, and hardware requirements can change.

Verify the Installation

After starting the server, open another terminal and test the API:

curl -fsS http://127.0.0.1:8080/v1/models

The command should return the models currently available through the running server.

For additional options, use:

tensorfold --help
tensorfold capabilities --json
tensorfold serve --help

Final Verdict

TensorFold is an ambitious local AI inference project that combines a native Zig engine, GPU acceleration, model management, and API compatibility in one ecosystem. Its ability to serve local language models without requiring Python or MLX is a major advantage for users who want a more self-contained inference environment.

The OpenAI and Anthropic-compatible APIs make it practical for integrating local models with existing applications, while its desktop and web interfaces provide alternatives to command-line operation. Advanced features such as prompt caching, concurrency controls, and memory diagnostics also make it interesting for more demanding inference workloads.

Its main limitation is hardware and model compatibility. Users must verify that their GPU, driver, and chosen checkpoint are supported before expecting good results. The desktop application and Living Weights feature also have different maturity levels from the core inference engine.

Overall, TensorFold is a promising choice for developers and enthusiasts who want to run local LLMs with native GPU acceleration and a flexible API server.

Sources


TensorFold v1.3.3
Free
Software Informations:
Developer:

Operating System:
macOS / Linux
Date Added:
2026-11-22T00:07:23.081Z
Categories:

Post a Comment/Report Broken Link: