TensorFold v1.2.10

TensorFold is an open-source local AI inference engine designed to run large language models directly on personal computers and GPU servers. Developed primarily in Zig, it provides native GPU execution through Metal on Apple Silicon and CUDA on NVIDIA hardware, without requiring a Python or MLX installation to serve models.

Unlike tools that simply wrap existing inference frameworks, TensorFold implements its own model execution and GPU kernels. It also provides an API server compatible with popular AI clients, making it possible to use locally hosted models with applications designed for OpenAI or Anthropic services.

The project goes beyond basic model inference by including a desktop workspace, web interface, model management tools, and experimental capabilities for modifying model weights while running.

Features

Native Local LLM Inference

TensorFold runs language models directly on supported hardware using native GPU backends.

Its main advantages include:

  • Metal acceleration for Apple Silicon.

  • CUDA acceleration for NVIDIA GPUs.

  • Native HIP support for selected AMD hardware.

  • Direct model checkpoint loading and tokenization.

  • No Python or MLX installation required for native serving.

  • Configurable context length and generation parameters.

  • Support for concurrent inference requests.

The engine is designed to make local inference more accessible without requiring users to maintain a complicated Python-based machine learning environment.

Hardware support varies by backend and model architecture. Not every model is compatible with every GPU, so users should consult the project's model compatibility information before downloading large checkpoints.

Model Management

TensorFold includes commands for downloading and managing models from Hugging Face and other supported sources.

For example, users can download a model and start an inference server with:

tensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit
tensorfold serve mlx-community/Qwen3.5-2B-MLX-4bit --name local-model

The application also supports existing local checkpoint directories and models already present in the Hugging Face cache.

Model configuration options include context length, maximum output tokens, temperature, sampling parameters, and reasoning settings. This provides flexibility when adapting a model for coding, chat, document processing, or other workloads.

An important consideration is that model weights have their own licenses. TensorFold's Apache 2.0 license does not automatically grant permission to use every model distributed through external repositories.

OpenAI and Anthropic-Compatible API

One of TensorFold's most useful features is its local API server.

It supports several popular API formats, including:

  • OpenAI Chat Completions.

  • OpenAI Responses API.

  • OpenAI Completions.

  • Anthropic Messages API.

This allows compatible applications to communicate with locally hosted models instead of relying on cloud AI providers.

For example, after starting the server, users can test it with:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "local-model",
    "messages": [
      {
        "role": "user",
        "content": "Explain how local LLM inference works."
      }
    ],
    "max_tokens": 256
  }'

The default server address is 127.0.0.1:8080. Authentication options are available for deployments that need API access control.

This compatibility makes TensorFold useful for integrating local models with coding assistants, chat interfaces, and other applications that already support standard AI APIs.

Concurrent Requests and Performance Controls

TensorFold includes controls for managing inference workloads and available GPU memory.

Its serving configuration supports:

  • Parallel inference requests.

  • Automatic concurrency selection.

  • Request queuing.

  • Context-length limits.

  • Prompt caching.

  • Maximum output token limits.

  • Sampling and reasoning configuration.

  • Memory diagnostics.

The automatic parallelism option attempts to determine how many requests can run concurrently based on available resources. Explicit concurrency limits are also supported.

Prompt caching can reduce repeated processing when conversations share common prefixes. Additional options allow users to control memory allocation and retained prompt data.

These features are particularly useful when operating a local inference server for multiple applications or users.

TensorFold Desktop Workspace

TensorFold also provides a desktop application for managing local AI workloads.

The workspace combines several functions:

  • Local model discovery and downloads.

  • Model execution and chat.

  • Hardware recommendations.

  • Live generation metrics.

  • Coding agent management.

  • Document editing.

  • Local inference management.

This provides a more approachable interface for users who prefer graphical applications over terminal-based workflows.

The macOS desktop application is distributed as an Apple Silicon beta. Its mobile clients remain preview software, so users should not expect the same maturity as the native inference engine.

TensorFold Web

For users who prefer a browser interface, TensorFold includes a local web application.

It provides a graphical environment for selecting models, running inference, and chatting with locally hosted models.

The web interface complements the command-line server and desktop application, giving users another way to interact with their local AI environment.

AI Coding Workers

TensorFold's desktop workspace also includes coding workers designed to assist with development tasks.

This extends the application beyond conventional chat-based inference. Users can manage coding-oriented workflows alongside their local models and documents.

The desktop workspace is still evolving, so its agent functionality should be evaluated separately from the more established inference engine.

Living Weights

One of TensorFold's more experimental capabilities is Living Weights.

Instead of keeping newly learned information exclusively in an external database or conversation history, this feature attempts to write selected facts directly into a model's checkpoint files.

Users can provide information through a dedicated API endpoint, and TensorFold processes the material as lessons that modify the model's weights.

The project also provides a 3D visualization interface showing learned facts and their associated model components.

However, Living Weights is explicitly experimental. The project documents limitations involving interference between related facts, declining recall after learning multiple facts, and reduced decoding performance on some hardware.

It is an interesting research feature, but it should not be considered a replacement for established retrieval-augmented generation systems or a reliable persistent memory solution.

Download TensorFold v1.2.10 - Software Mirrors

TensorFold v1.2.10 for macOS

tensorfold-engine-1.2.10-darwin-arm64-metal.tar.gz | 7.27 MB

tensorfold-1.2.10-macos-arm64.tar.gz | 7.3 MB

TensorFold v1.2.10 for Linux

tensorfold-1.2.10-linux-x86_64.tar.gz | 49.87 MB

tensorfold-1.2.10-linux-aarch64.tar.gz | 49.38 MB

TensorFold v1.2.10 Source Code

TensorFold v1.2.10 Source code (zip)

TensorFold v1.2.10 Source code (tar.gz)

TensorFold v1.2.10 Release Notes:

TensorFold 1.2.10

1.2.10 brings faster DeepSeek-V4.1 prompts on Metal, faster Qwen3.8-27B decode, background responses, tensorfold rm and tensorfold launch, Qwen3-Embedding and Unlimited-OCR-2 on Metal, and fixes for wrong and empty replies. Every speedup below keeps replies unchanged; the GLM-5.3-Flash FP8 figure compares two expert formats.

Speed

  • DeepSeek-V4.1 on Metal:
  • Cold prompts run about 5-7% faster on an M5 Ultra (2K tokens 600 to 631 tok/s, 8K 855 to 917, 30K about 1,000 to
-- 1,063), now that the target's own buffers join the weights' residency set, inside its load limit (#3321).
  • A cold prompt's first chunk reads its Engram rows while the GPU runs the layers before each Engram layer, so on an
-- M5 Ultra cold prompts start about 0.3 s sooner at 2K tokens, 0.55 s at 8K and 0.8 s at 30K (#3318).
  • On M5 GPUs (generation 17, macOS 27) the prompt's routed experts run on the tensor units, bit-equal to the decode
-- kernels: on an M5 Ultra a 4096-row chunk of a layer takes 155 ms instead of 185 ms. Other Macs keep the simdgroup -- kernels (#3330).
  • The dense FP8 prompt projections take 14 to 19% less time on an M5 Ultra; a 4096-row prompt chunk of release layer
-- 2 drops from 185 to 175 ms (#3322).
  • Decode launches of 6 to 16 rows run the dense FP8 projections on a simdgroup-matrix tile, so two drafted streams
-- decode 3.7 to 5.5% faster on an M5 Ultra and 3.6 to 5.4% faster on an M3 Ultra; other stream counts are level -- (#2973).
  • The server keeps the model's ~300 GiB of weights wired between requests. On an M5 Ultra the first request after a
-- restart reaches its first token in 0.28 s instead of 1.8 s, and a cached follow-up after 15 s idle in 0.02 s -- instead of 1-2.5 s (#3384).
  • The prompt cache also keeps a prompt's state before its first user turn (once the system part before it is 256
-- tokens or more) and, for a thinking prompt, before its final <think>. A new conversation with the same system -- prompt and tools resumes after them, and the next turn of a thinking conversation resumes from the last one instead -- of reading the whole conversation again (#3345).
  • TF_DSV41_CALIB_FILE now prices draft depth from a cost table measured on the serving machine; with a measured
-- table, prose decode on an M5 Ultra rose 0.7% greedy and 2.4% sampled (#3365).
  • Qwen3.8-27B:
  • On CUDA with paged KV and the shared prompt store, streams resumed from one prompt read each shared prefix key tile
-- once a forward instead of once a stream: at a 32K shared prefix on a GB10, decode is 57% faster at eight streams -- and 28% at four. Every other forward runs the same kernels as before (#3653).
  • On Metal before M5, a drafted window's rows run the decode attention in pairs over the keys below the window: at a
-- 32K prompt on an M3 Ultra, drafted decode is 24% faster at one stream and 29% at eight. TF_QWEN27_JOIN=0 keeps -- the unpaired pipelines (#3661).
  • On M1 to M4, the 4-bit prompt matrix multiply reads each input row's group sums and half-precision operands from a
-- prepass, bit-equal to the existing tiles: prompts run 3% to 4% faster on an M3 Ultra from 2K to 64K tokens, with -- decode level (#3215).
  • On Intel XPU, the prompt gdn_out kernel takes 22.6% less time on an Arc Pro B70 (736.3 to 570.2 ms over a 32K
-- prompt), with byte-identical outputs, its fallback included (#2921).
  • Qwen3.8-Flash-Next:
  • On Metal (M1 through M4), the scale index is now the default for 4-bit experts in groups of 32;
-- TF_FLASHNEXT_SCALE_INDEX=0 turns it off. On an M3 Ultra with the MLX 4-bit checkpoint, memory in use once loaded -- falls from 129.6 to 122.5 GiB, drafted decode on one stream is 2.0% to 4.2% faster (prose 138.6 to 142.0 tok/s, a -- copy-heavy edit 282.0 to 293.8), four streams are 2.6% faster (203.1 to 208.3 tok/s) and a cold prompt's first -- token arrives about 1% sooner; load takes about 0.7 s longer (#2805).
  • On HIP (gfx1151), the tf_fn_idx_scores kernel's time falls from 276.0 ms to 141.5 ms at 8K and from 2,873.9 ms
-- to 2,091.4 ms at 32K (#3601).
  • On HIP, when several streams draft in one round, the MTP head drafts them together, as on CUDA and Metal: on a
-- Strix Halo iGPU, distinct-prompt decode is 2.8-3.9% faster at 2 streams and 6.6-7.1% at 4, with one stream -- unchanged (#2696).
  • At TP=2 on CUDA, the draft head, head-group and rollback passes replay as CUDA graphs on both ranks: with NVFP4 and
-- MTP on GB10s, decode is 1.2-1.4% faster and prompts 0.5-2.1% faster (#2837).
  • On two GB10s, the CUDA recipe now lists both RDMA devices of each ConnectX-7 port in NCCL_IB_HCA, with
-- NCCL_IB_ADDR_RANGE covering both. On two GB10s joined by one cable, this setting made cold 2,048 to 32,768-token -- prompts at --tp 2 26.5% to 28.1% faster (1,147 to 1,469 tokens/s at 8,192), with decode speed and tokens -- unchanged (#2969).
  • GLM-5.3-Flash on Metal: checkpoints whose routed experts are not 4/64 read each expert once a drafted window for
every row that picked it, and the shared expert once: on an M5 Ultra, an 8-bit build's greedy decode goes from 50.0 to 53.0 tok/s and sampled from 46.5 to 48.8 (#3631).
  • Flint on Metal: --drafter takes the published incoai/Flint-DFlash2 draft model through the DFlash2 runtime the
27B uses, and every reply equals drafts off: greedy decode on an M5 Ultra goes from 174.6 to 233.0 tok/s at one stream and 221.4 to 289.7 at two (#3511).
  • Nemotron 3.5 Bolt on CUDA: the shared expert runs on the side stream in decode windows, as on Super, so decode on
a GB10 is 2.9-3.0% faster at one stream and 1.4-2.1% at four (#2847).
  • Prompt cache, opt-in: TF_CACHE_EVICT=decayed_resume (off by default) gives a resumed snapshot bounded recency
credit when the RAM store evicts, so agent conversations that keep returning outlive one-off prompts. Through the real store at 16 GiB, agent traces rebuild 24.4-34.5% fewer prompt rows, and at 4 GiB 3.8-8.0% fewer (interleaved chats 17.7% fewer), with none worse (#3258, #3241).
  • Follow-ups and busy servers:
  • Qwen3.5 on CUDA (qwen3_5 and qwen3_5_moe) declares warm turns, so the server prefills a follow-up's prompt in
-- the background and the next turn resumes past the previous reply (#2824). Qwen3.5-2B on Metal declares them too, -- so a follow-up that re-renders earlier rows resumes instead of prefilling the whole conversation again (#3662).
  • Qwen 3.5, 3.6 and 3.8 on CUDA: a background prompt fill stops between chunks when a foreground request is waiting,
-- keeps the prefix it has filled and resumes from there (#2873).
  • Nemotron on Metal fills a new prompt a chunk at a time between other streams' decode rounds, instead of stalling
-- them for its whole prefill (#3054).
  • Fewer launches and less repeated work:
  • CUDA: a dense Qwen 3.5 MLP's NVFP4 prompt gate, up and activation run in one launch for prompt rows past 16
-- (#3252); Spark 2.5 and MiniMax-M3 run a prompt's last layer only on the rows something reads, as Qwen's already did -- (#3122); Nemotron 3.5 Super at TP=2 sends rank 1 one header a decode window and one exact-size message a layer, so -- it queues a window's layers with no host read between them (#2839).
  • Metal: on chips with tensor units, core row projections over windows of 4 or more rows run 8 rows at a time, then
-- 4, 2 and 1, so each weight code is unpacked once for up to eight rows (#3129); Qwen3.8-27B's projections on M1-M4 -- drop their inner barriers, and XS sums compile out before tensor units (#2913, #2900).
  • HIP: at head size 256 on spans of 16,384 keys or more, the planned decode walk scores keys in its own kernels
-- without per-element tests or a per-key branch, for Qwen3.6-35B-A3B and Qwen3.8-27B (#3508); on gfx1151, a plain -- prompt of 1 or 2 rows takes the 2-row K-parallel tile (#3273).

Models and features

  • Unlimited-OCR-2 on Metal: Baidu's Unlimited-OCR-2 is served natively from its BF16 release on M5-class Macs, page
images through serve --vision, with the family's chat template built in. On five reference pages, every departure from the release's own reply is a one-step tie (#2074).
  • Qwen3-Embedding on Metal: 0.6B, 4B and 8B serve through /v1/embeddings on chips with tensor units, with the same
refusals and MRL widths as CUDA; vectors match an FP32 reference within 0.001 cosine distance, and a batch equals its single inputs bit for bit (#3494).
  • Nemotron 3.5 Transcribe on CUDA: the language model loads from the bf16 release as published, with exact bf16
caches and no conversion, and the audio encoder (log-mel, the ASR encoder, the streaming speaker encoder and its cache, fusion and projection) runs on the GPU. The server does not take audio input for it yet (#1631).
  • Qwen3.8-27B drafts on its own: serve drafts with DFlash2 when z-lab/Qwen3.8-27B-DFlash2 is already in the
Hugging Face cache, and one line names it. Nothing is downloaded, a drafter that does not open leaves the target serving alone, and --drafter none serves without it. The target checks every drafted token, so replies are the same tokens (#3204).
  • GLM-5.3-Flash:
  • On Metal, routed experts stored in the official FP8 checkpoint's format (E4M3 codes, 128 x 128 f32 block scales)
-- load as stored, and a combined checkpoint can point at whole FP8 shards. On an M5 Ultra, FP8 experts beside the -- 8-bit backbone decode prose at 60.8 tok/s against 55.7 with MLX 8-bit experts; the two expert formats match on -- prompt per-token loss (0.542561 against 0.542376, inside the 0.070% standard error) (#3682).
  • On Metal, a checkpoint that stores layers.45.shared_head.head drafts MTP tokens through it on one Mac while
-- lm_head verifies every token, so replies are unchanged (#3632).
  • On CUDA, NVFP4_NZ5 checkpoints load, their block-32 routed experts reading E2M1's code 0b1000 as +5; every other
-- recipe and mix is refused by name, and E2M1 checkpoints load exactly as before (#2894).
  • On CUDA, TF_GLM_EXPERT_FORM=classic or factored names the expert form and skips the load's race between the
-- two, so launches and builds compare at one form; unset keeps the race (#3151).
  • Kept conversations on more CUDA models: Gemma 4, Dory and Spark2.5 can keep conversations on disk through the
shared kept-image store (#3135). LFM2.5 takes its attention pages from the shared KV pool and keeps chat states in the prompt cache, with the copies off the first token's path (#1503), and LFM2.5-24B-A2B's prompt-stream writes wait for those copies first (#2989).
  • tensorfold rm REPO[@REVISION] removes a checkpoint pull cached, or one of its snapshots with the refs that
name it and the blobs no other snapshot links to. It lists what goes and the bytes freed, then asks (--yes skips the question, --dry-run only lists). It never deletes outside the repo's cache folder or through a link, and refuses snapshots a running server holds (#3525).
  • tensorfold launch starts Claude Code, Codex, OpenCode or Hermes against the local server, with the API
base URL and key set only in the agent's environment; no agent config or home-folder files change (#3248).
  • tensorfold device reports this machine's hardware, supported backends and formats, memory budget and the cached
models that fit; a saved device profile also supplies the help summary and certain pre-load refusals (#3107).
  • A Grafana dashboard for the engine's Prometheus metrics: TTFT and TPOT quantiles, decode rate, running and waiting
requests, KV use, draft acceptance, prompt cache, memory and admission (#3141).
  • Benchmark receipts: managed runs on a listed Apple silicon Mac record its coarse family in hardware.form_factor
(such as Mac mini or MacBook Pro), mapped on the device from a fixed table so the product identifier is never written; other Macs, Linux and Windows leave the field out and keep the same receipt bytes. benchmark inspect in 1.2.9 and earlier refuses receipts that carry it (#2009).
  • Groundwork, not used by any served model yet: dots.mocr's image tower weights on CUDA (#2769), Metal kernels for
DeepSeek V4.1's vision tower (#1721), a host KDA reference for Ling-3.0-tiny with a Metal check of Kimi's k3_kda (#3055), shared SSD expert code ready for DeepSeek V4.1 experts on SSD, with Flash Next unchanged (#3683), KV pool pages returned once their readers are done (#3353), and message start positions recorded in rendered chat prompts (#3679). Qwen3.5's dense family plans each rank's share of the 27B for tensor parallelism, and --tp above 1 is refused by name until ranks serve (#3383).
  • Docs: the README is now a short quick start and docs list, with setup, serve flags and model tables in the new
run guide and supported models pages (#3170); a guide to using TensorFold from OpenClaw through the openclaw-tensorfold plugin (#3341); and recipe updates for Nemotron 3.5 Lightning on a Strix Halo iGPU with native HIP (#3560), GLM-5.3-Flash in long agent sessions (#3576), GLM-5.3-Flash's FP8 latent cache diagnostics, with the cache not qualified (#3432), and the loads TF_GLM_COPIES=1 sped up and slowed, with the default staying off (#3335).

Server and API

  • Background responses: background: true on POST /v1/responses answers at once with a queued response and
generates the reply after the POST returns; GET /v1/responses/{id} polls it, and POST /v1/responses/{id}/cancel or DELETE stops it. At most 64 are queued or running per server (16 per key, 256 MiB of copied requests) before a create answers 429, and stream: true with background is refused (#3220).
  • Rerankers also answer /score, TEI's /rerank with texts and Cohere's /v2/rerank, each translated onto the
existing /v1/rerank path (#3203).
  • /v1/completions takes a prompt list of 2 to 64 strings and gives one choice a prompt, each the reply that
prompt gets alone, streamed or not, with usage summed over all of them; a longer list is refused (#3194).
  • Images: POST /v1/images/generations with stream: true streams each denoising step's progress
(image_generation.progress) and, with preview_interval, partial previews, then the PNG (#3245). It also takes n from 1 to 4, each picture with its own seed (the request's seed plus its index, or 42 plus its index) reported on its data item, and each PNG is the one an n: 1 request with that seed returns (#3244).
  • Browsers: --cors-origin ORIGIN (repeatable, exact origins only, no wildcard or credentials mode) lets a page on
that origin call the API, while keys are still required. /v1/apply-template returns the prompt text a chat request's messages render to, special tokens included, without running the model (#3166).
  • Grammars: a request waits at most 2,000 ms for its grammar and past that gets HTTP 400 naming the limit, and each
model runs at most 4 compiles at once, so a slow compile no longer holds up other requests (#3231). A grammar over 512 KiB is refused with HTTP 400 before it is compiled (#3038).
  • Wait reasons in metrics: every queued request carries one wait reason, and DeepSeek-V4.1 names its own
(kv_pool, memory_floor, prefill_concurrent). /metrics gains requests_waiting_by_reason, request_refusals_total and request_failures_total by code, and restore_fallbacks_total; /v1/load gains waiting_by_reason and refusals (#3014).
  • Drama speech: an explicit --max-frames is honored exactly or refused at open, so --context - 2 (once trimmed
silently to --context - 3) is refused by name; an overlong prompt's 400 names the assembled-row limit, the request text's tokens and the reference frames; and GET /v1/models lists a speech object with the loaded budgets (#2143).
  • Image URLs: a host lookup stops at 3 seconds (or the request's deadline, if sooner) even when the system resolver
keeps retrying, at most 64 lookups run at once, and a refused connection falls through to the host's next checked address (#3127).
  • Chat templates may use Jinja's float filter, float(value, default=0.0), which reads text as ASCII only,
instead of the template being refused (#3577).
  • Docs: the Ollama-compatible routes, with what each takes and refuses, the NDJSON replies and how each differs from
its OpenAI-style route (#3496); strict tools under tool_choice: "auto" and TENSORFOLD_TOOL_GRAMMAR, including the legacy DeepSeek alias (#3235).

Platforms

  • CUDA on WSL2 and Windows: an integrated GPU such as the RTX Spark N1X admits a model against its CUDA memory pool
instead of WSL's share of RAM in /proc/meminfo, so a checkpoint the GPU holds is no longer refused. TENSORFOLD_UNIFIED_HOST_COUNTS=1 keeps /proc/meminfo and =0 reads the pool; any other value is refused at startup, and native Linux admits as before. Drama 3 on native Windows opens cuBLAS as cublas64_13.dll too (#2107).
  • CUDA memory for Qwen 3.5: the forward scratch is reserved at load in its own allocation, sized to --context
(capped at the largest prompt pass) plus the verify window, and a pass past it is refused before any kernel runs. With a short --context the model starts on a smaller GPU: the 4B in MLX 4-bit at --context 1024 with a 0.5 GiB memory reserve starts on a 4 GiB RTX 3050 Laptop GPU where it was refused, and on a GB10 the 27B's tokens and speed are unchanged (#1879). A dropped kept prompt state's pinned host buffer goes to a spare list of at most two, which the next save or recall blank of the same size reuses instead of pinning new memory (#2841).
  • Metal memory: Flash Next with an explicit --context smaller than the model's full context sizes its caches and
selectors to that context, so it holds less memory, with tokens unchanged (#3691). Qwen3.8 Flash Next with SSD residency on (--ssd-experts, --ple-on-ssd) no longer allocates the second prompt workspace that only two-segment prompts use, 6.16 GiB less at the same window (#3690, #3667). Nemotron without tensor units skips the lane tiles no kernel reads (#3375).
  • Metal loading: H3's Model.load reads every transformer shard ahead of its mappings and fails the load on a
missing or short shard; TF_LOAD_PREFETCH sets the reader count and 0 turns read-ahead off (#2999).
  • Intel XPU, Qwen3.8-27B: logprobs and top_logprobs are served for a lone drafted stream, plain streams and
shared rounds, greedy and sampled, with tokens bit-equal to a request without them; prompt_logprobs is still refused. Decode costs 2.9% to 5.4% for a lone drafted stream and 12.9% for two streams at once (a total rate, prefill included), as rough figures from one Arc Pro B70 (#3051 via #3722). The target's and the DFlash2 drafter's RoPE tables now come from core's exact helper and hold the reference's fp32 words; 15 of 32 and 18 of 64 values move by 1 ulp (#2000).
  • Libraries on macOS and Windows: a library named without a folder is never loaded from the directory the server
was started in; on macOS, name its folder in DYLD_FALLBACK_LIBRARY_PATH if it was only found there (#3100).
  • Settings checked by name: on Metal, the Qwen KV paging and slot-resume switches and the slide checks refuse a
malformed value by name, and TF_SLIDE_TAKE_OUT_CHECK is read once at start with --slide (#3018). On CUDA, a malformed TF_FLASHNEXT_CHUNK or TF_FLASHNEXT_CHUNK_SOLO names the variable, the supplied text and the accepted range when the engine loads, and a value that is empty or only spaces and tabs now reads as unset (#3308, #3591).

Desktop and Web

  • Kanban and Rooms: tickets have a Details action to read and edit their title, description, type, priority and
effort, and archived tickets can be restored (#3464); a new ticket shows once, even when its update event arrives before the create reply (#3363); returning a parked ticket to Ready releases its previous assignee, so another Room seat can take it (#3469); new ticket worker chats appear in the sidebar as soon as they are created, for solo and Room workers (#3461); and Room health shows each Room's worktree path and Git branch (#3463).
  • Effort and modes: confirming an ultrathink effort change now applies it (#3502), bots keep their own reasoning
effort instead of sharing the app-wide setting (#3500), and selecting Supervisor or applying the Supervised build preset is accepted instead of failing with collaborationMode has an invalid value (#3472).
  • Conversations and models: Approvals hides Open on ticket reviews whose conversation is archived, deleted or not
loaded yet, and a thread whose conversation is missing shows "Conversation unavailable" instead of a blank window (#3393). Readiness and Command Centre check a local provider such as Ollama or LM Studio against its live model list at its configured address, so a stopped server or one with no models no longer shows as Ready (#3605). An idle task's disabled Planning only control opens the model picker on Local, so a planning provider no longer strands the user (#3357).
  • Search: the built-in agent's web_search and web_fetch accept provider: "keenable", an opt-in Keenable
provider with no account or key; only the query or URL goes to Keenable, and the defaults are unchanged (#3637).
  • Sandbox and tools on macOS: Git keeps working in the sandbox when macOS selects Xcode without a legacy
xcode_select_link (#3479); MCP servers bundled inside macOS apps can load their own frameworks, with the bundle read-only and other app bundles out of reach (#3423); and an oversized scroll delta is refused as a bad request instead of stopping the computer-use helper (#3470).
  • Studio accepts local benchmark receipts with the optional hardware.form_factor field and keeps it in saved
history and exports; unknown labels, raw product IDs and extra hardware keys are still rejected (#2693).

Fixes

  • DeepSeek V4.1 on CUDA: a prompt whose prefill segment is 18 to 32 rows no longer replies with repeated
begin-of-sentence tokens, because the second streams buffer is now planned for 32 rows; replies at every other length are unchanged (#3366, #3388).
  • DeepSeek V4.1 on CUDA: a request whose prompt and reply don't fit the free KV pool now waits in the queue until a
running request ends instead of running unreserved (one that can never fit fails at once with a reason), and at TP=2 the ranks agree on each new KV page and roll back together; replies are unchanged (#1682).
  • DeepSeek-V4.1 on Metal: draft trees (TF_DSV41_TREE=2 to 4) no longer end a reply after a few tokens; each
sibling branch runs as its own chain on a shadow slot, so replies are unchanged. Trees stay off by default: on an M5 Ultra they were flat on the 16-prompt bench (greedy 89.3 to 89.2 tok/s, sampled 72.7 to 72.5 with TF_DSV41_TREE=2) (#3326).
  • Empty thinking replies: a reply that ends at the model's own end token inside a think block that never closed,
with no tool call, now answers with its text instead of an empty answer, on chat completions, /v1/messages and /v1/responses, streamed and whole. The text stays in the reasoning too, and chat completions replies say tensorfold.reasoning_recovered: true. Stop strings, max_tokens, the loop guard, forced tool calls and structured output keep today's reply, and token ids, token_sha256 and token counts are unchanged (#3699).
  • Tool calls in code: tool-call markup quoted in Markdown inline code spans and fenced code blocks stays prose, in
whole replies and in streaming, instead of opening, hiding or running as a live tool call (#3588).
  • Flash Next tool results: when several tool calls' results arrive out of call order, each result is now placed
after its own call's ID before the prompt is rendered, since Flash Next's template gives the model anonymous result blocks in order. Only a contiguous run with exactly one result per call is reordered, and every other request and backend renders as before (#3775).
  • Kolibri-1: a server with default flags no longer panics with an integer overflow on its first request when a
cache sits below the origin (#3561).
  • Prompt cache: on Qwen3.8-27B on Metal, the optional pin that keeps a reply's history for a same-prompt
regeneration can no longer evict another conversation's newest state in a tight cache (#3556); radix pages carry their reuse identity, so pages are only reused where they were written for the same request shape (#2888); and with --learn-conversations, an image conversation's kept states stay in RAM as before and the log now says so once per conversation, with no prompt text or image content logged (#3008).
  • GLM-5.3-Flash on CUDA with TF_KV_SHARE and no --prompt-cache-gib: a room that holds one more whole window no
longer keeps fewer states; Jammed XSFP4 at TP=3 on three GB10s keeps all 32 at TF_KV_SHARE=32 instead of 8, with the same tokens (#3125).
  • Shared draft pricing: a chain-cut, unmeasured tail is priced by the rule's rates instead of as a measured zero,
and the cache's cut marker survives allocation errors (#2979); shared rounds price a plain round apart from drafted rows, level on gfx1151 with Qwen3.6-35B-A3B and MTP with tokens unchanged, and opt-in TF_SHARED_PRICE_LOG=1 prints each shared round's prices (#2687); fixed DFlash2 block work stays separate from verification rows (#3755).
  • tensorfold device: when the memory reserve leaves a known zero bytes for weights, a nonempty checkpoint is
reported too large and the text report shows 0.0 GiB (#3773).
  • Files and pulls: temporaries a crash left beside notes and the --slide-graph file are cleared when the server
opens their folders, only its own and only when their process is gone (#3132). A Living Weights fact graph save whose final rename fails no longer leaves its temporary file behind (#3096), and a fact graph write the OS refuses names its cause (#3126). A hub file path refused for a reserved name or a path escape names the path and the reason, and the pull exits 1 (#3097); a cluster pull plan asks for each file's safetensors header, and a file read in full is fetched whole, so its manifest hash is checked (#3095).
  • macOS: the native binary imports _memcpy from libSystem again, and the saved device profile is decoded with
checked decimal arithmetic and keeps its format (#3761).
  • Under the hood: every tokenizer test runs on the host, and the parity corpus adds the 22 scheduled Indian
languages (#3082, #1988); a seeded explorer for lane schedules (#3233); new and repaired kernel-contract and HIP invariance checks (#3580, #3343, #3157); host CI split into two jobs and a redundant nightly shard removed (#3474, #3276); shared code moves with unchanged output (#3214, #3269); and test, tool and comment cleanups (#3086, #3263, #3489, #3465, #3504, #3503, #3572, #3012, #3094, #3633).

What changes by default

  • Qwen3.8-Flash-Next on Metal (M1 to M4): the scale index is on for 4-bit experts in groups of 32;
TF_FLASHNEXT_SCALE_INDEX=0 turns it off (#2805).
  • Qwen3.8-27B: serve drafts with DFlash2 when z-lab/Qwen3.8-27B-DFlash2 is already in the Hugging Face cache;
--drafter none serves without it (#3204).
  • Qwen3.8-27B on Metal before M5: drafted windows run the decode attention in pairs; TF_QWEN27_JOIN=0 keeps the
unpaired pipelines (#3661).
  • Server: a reply that ends inside a think block that never closed answers with its text instead of an empty
answer (#3699).
  • Server: a request waits at most 2,000 ms for its grammar before HTTP 400, and a grammar over 512 KiB is refused
(#3231, #3038).
  • Settings: TENSORFOLD_UNIFIED_HOST_COUNTS takes only 0 or 1 (#2107), and malformed Qwen KV, slide and
TF_FLASHNEXT_CHUNK settings are refused by name (#3018, #3308).
  • macOS and Windows: a library named without a folder is no longer loaded from the server's starting directory
(#3100).
  • Benchmarks: benchmark inspect in 1.2.9 and earlier refuses receipts that carry hardware.form_factor (#2009).
The first three keep replies unchanged. The fourth changes the answer text, not the tokens. The grammar and settings changes turn a slow or malformed request into a named refusal, and on macOS a library found only in the starting directory now needs its folder in DYLD_FALLBACK_LIBRARY_PATH.

Thanks

@AbrahamCete, @AIFlowML, @akol1, @BadAd84, @Bizuayeu, @BobClawblaw, @cshintov, @currencycaver29, @cwschroeder, @devopsfortunato, @dinadur, @drowzeys, @EugeneClaw, @garvrao80, @gilby, @ilya-bogin-keenable, @jayleaton, @jschmied, @laudney, @Mirrdhyn, @Nipale-ai, @plotarmordev, @robertosantosx2, @srh3877, @swbiggart, @sx4im, @tacosta and @ThinkOffApp.

Performance and Compatibility

TensorFold is built with Zig and uses native GPU execution rather than depending on a Python-based inference stack for its serving engine.

This architecture reduces the number of runtime dependencies and gives the project direct control over model execution, memory management, and GPU kernels.

Performance depends on the model architecture, quantization format, GPU memory, context length, and available hardware acceleration.

Apple Silicon: TensorFold uses Metal for GPU execution on supported Apple Silicon Macs. Unified memory can be useful when running larger models, although the available memory must accommodate model weights, context data, and concurrent requests.

NVIDIA GPUs: The Linux release supports CUDA-based inference. Its documented Linux release requirements include an NVIDIA driver version R580 or newer and CUDA 13.0 compatibility. Supported GPU architectures and model combinations vary.

AMD GPUs: Native HIP support is available for selected hardware and models. The project documents specific qualification results rather than guaranteeing compatibility with every AMD graphics card.

The project also advertises Vulkan support, but users should verify the actual backend and model compatibility for their target hardware rather than assuming every advertised backend supports every model.

TensorFold supports Linux and macOS for its documented native workflows, with additional Windows support through its native Windows build documentation.

System Requirements

macOS

  • Apple Silicon Mac.

  • Compatible Metal GPU support.

  • Sufficient unified memory for the selected model.

  • Homebrew for the recommended installation method.

Linux

  • 64-bit Linux system.

  • glibc 2.28 or newer for the documented NVIDIA release archives.

  • NVIDIA GPU and R580-or-newer driver for the documented CUDA release.

  • CUDA 13.0-compatible runtime environment.

  • Sufficient GPU or unified memory for the selected model.

Windows

  • Supported Windows environment.

  • Compatible NVIDIA GPU and driver for the CUDA workflow.

  • Follow the project's native Windows build and installation documentation.

There is no universal minimum RAM requirement because memory consumption depends heavily on the model, quantization, context size, and concurrency settings.

Pros and Cons

Pros

  • Free and open-source under Apache 2.0.

  • Native inference engine written in Zig.

  • Metal acceleration for Apple Silicon.

  • CUDA support for NVIDIA GPUs.

  • Native HIP support for selected AMD hardware.

  • Does not require Python or MLX for native serving.

  • Supports OpenAI-compatible and Anthropic-compatible APIs.

  • Includes model downloading and management.

  • Supports concurrent requests and prompt caching.

  • Provides desktop and browser-based interfaces.

  • Includes performance diagnostics and memory controls.

  • Experimental Living Weights functionality.

Cons

  • Hardware and model compatibility vary considerably.

  • NVIDIA Linux releases require a relatively recent driver.

  • Large models require substantial memory.

  • Some desktop and mobile functionality remains in beta or preview.

  • Living Weights is experimental and has documented limitations.

  • The project has a learning curve for users unfamiliar with local LLM deployment.

  • Not every model available through Hugging Face is compatible with its native engine.

How to Install

macOS

The recommended installation method for Apple Silicon is Homebrew:

brew install ashhart/tensorfold/tensorfold

Download a model and start the server:

tensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit

tensorfold serve mlx-community/Qwen3.5-2B-MLX-4bit \
  --name local-model \
  --context 8192 \
  --no-thinking

Linux with NVIDIA GPU

Download the appropriate release archive for your CPU architecture. The project provides separate archives for x86_64 and ARM64 systems.

For example:

V=1.2.8

curl -fLO "https://github.com/ashhart/TensorFold/releases/download/v$V/tensorfold-$V-linux-x86_64.tar.gz"

tar -xzf "tensorfold-$V-linux-x86_64.tar.gz"

export PATH="$PWD/tensorfold-$V-linux-x86_64/bin:$PATH"

Download a compatible model and start inference:

tensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit

tensorfold serve mlx-community/Qwen3.5-2B-MLX-4bit \
  --name local-model \
  --context 8192 \
  --no-thinking

Check the latest release instructions before installation because package names, supported architectures, and hardware requirements can change.

Verify the Installation

After starting the server, open another terminal and test the API:

curl -fsS http://127.0.0.1:8080/v1/models

The command should return the models currently available through the running server.

For additional options, use:

tensorfold --help
tensorfold capabilities --json
tensorfold serve --help

Final Verdict

TensorFold is an ambitious local AI inference project that combines a native Zig engine, GPU acceleration, model management, and API compatibility in one ecosystem. Its ability to serve local language models without requiring Python or MLX is a major advantage for users who want a more self-contained inference environment.

The OpenAI and Anthropic-compatible APIs make it practical for integrating local models with existing applications, while its desktop and web interfaces provide alternatives to command-line operation. Advanced features such as prompt caching, concurrency controls, and memory diagnostics also make it interesting for more demanding inference workloads.

Its main limitation is hardware and model compatibility. Users must verify that their GPU, driver, and chosen checkpoint are supported before expecting good results. The desktop application and Living Weights feature also have different maturity levels from the core inference engine.

Overall, TensorFold is a promising choice for developers and enthusiasts who want to run local LLMs with native GPU acceleration and a flexible API server.

Sources


TensorFold v1.2.10
Free
Software Informations:
Developer:

Operating System:
macOS / Linux
Date Added:
2026-11-19T00:07:23.417Z
Categories:

Post a Comment/Report Broken Link: