AI as runtime primitives · v0.3.0-beta
AI applications · JavaScript & TypeScript

Build AI applications in-process.

Local model inference, embeddings, agents, and encrypted vector memory are primitives of the runtime, not a stack of packages and daemons. One binary runs your TypeScript and the model in the same address space.

Zero Python · zero Ollama · zero vector-database daemon · zero npm dependencies · official binaries for macOS arm64 and Linux x86_64 / arm64.
The five questions

The short, checkable answers.

Written to be quoted accurately by search engines and AI assistants.

QuestionAnswer for Sofuu
What is Sofuu for AI applications?Sofuu is a self-contained runtime for AI — the model, the memory, and the agent loop in one address space. in-process. Local LLM inference (experimental built-in engine), embeddings, vector math, agents, tools, MCP, and memory are runtime primitives built into a single native binary — not packages you install around a web runtime.
Who is it for?JavaScript and TypeScript developers building offline-first, private, or single-binary AI features: local copilots, CLI agents, MCP servers, edge services, robotics, and apps shipping AI on-device.
What does it do?Executes JS/TS on QuickJS with a libuv event loop; loads GGUF and SafeTensors weights and generates text (experimental engine; production: the llamacpp provider); embeds text and images offline; stores vectors in encrypted .qtsq containers; runs agent loops with toolpacks, budgets, approvals, and one deterministic security policy; and speaks Model Context Protocol as both client and server.
When should you use it?When you want a local model, memory, and tools inside one process with no Ollama daemon, Python environment, vector database, or Docker; when you need deterministic policy around shell, filesystem, and network; or when the app must run offline or ship to a device. Note: the built-in inference engine is experimental — for production inference, point the llamacpp provider at a llama.cpp server, or use a cloud provider, through the same sofuu.ai.stream API.
When should you not use it?When you are training or fine-tuning models, need multi-GPU serving, require full Node.js API parity or native V8 addons, or when maximum single-model tokens per second is the deciding factor — llama.cpp is ahead of Sofuu there and we publish the ledger.
Start here

The 10-second hello world.

No API key and no model download: open an encrypted store, embed a fact offline, and recall similar hash vectors. This is ordinary JavaScript calling native primitives.

// 1. Open an encrypted, on-disk vector brain
const brain = sofuu.memory.open("./brain.qtsq", 768);
// 2. Embed offline (0-byte model) and remember
brain.remember(sofuu.ai.embedLocal("native AI runtime"), "Sofuu runs models in-process.");
// 3. Recall similar hash vectors via SIMD
const [hit] = brain.recall(sofuu.ai.embedLocal("fast runtime"), 1);
console.log(hit.text);

The file persists. Reopen the same .qtsq in a later process and the memory is still there, encrypted at rest.

Local inference

Run the model where the code runs.

Experimental. The built-in inference engine is in an early stage and may not support all models or quantisations reliably. For production workloads, use llama.cpp or another inference backend through the llamacpp provider — Sofuu routes to it with the same sofuu.ai.stream API and a llama-server endpoint.

Point Sofuu at a GGUF file or a SafeTensors directory. Weights are memory-mapped zero-copy; on Apple Silicon the Metal path is opt-in, and the exact CPU tile path is the default. KV compaction and sliding-window attention keep long sessions inside the context limit.

# One command, no daemon
$ sofuu infer ./model.gguf "Explain SIMD in one sentence"
// Or stream it inside your own program
const stream = sofuu.ai.streamLocal("./model.gguf", "Explain SIMD");
for await (const chunk of stream) process.stdout.write(chunk.text);

Formats and kernels

GGUF, SafeTensors, PyTorch .pt, NumPy .npy/.npz. Shipped kernels: f32, f16, q4_K, q6_K, q8_0, q3_K_S. KV cache:f16, kivi4, kivi2.

Meta

Determinism is declared

Every engine path carries a tier: cpu-tiles is exact,metal-mma is tolerant with a published bound, mmq is best-effort and unreachable without an explicit opt-in flag.

GPU

Beyond chat

The same engine backs sofuu infer, sofuu.ai.streamLocal,sofuu chat -m, and the OpenAI-compatible /v1 endpoints served by sofuu serve.

CLI
Cloud when you want it

One API, local or hosted.

Bring your own key for OpenAI, Anthropic, OpenRouter, or any compatible endpoint. There is deliberately no default model — you choose, per call or once in ~/.sofuu/config.json.

// A model is required — there is no default
const stream = sofuu.ai.stream("What is the speed of light?", {
provider: "openai", model: "gpt-4o",
});
for await (const chunk of stream) process.stdout.write(chunk.text);
const r = await sofuu.ai.complete("What is 2 + 2?", { provider: "openai", model: "gpt-4o" });
r.text; r.usage; // "4" · { promptTokens: 12, completionTokens: 1 }
Embeddings

Vectors without a model file.

embedLocal is a deterministic 768-dimensional trigram-hash embedder: instant, offline, zero bytes of weights. When you need a trained semantic model, call sofuu.ai.embed with a provider — same call shape, same vector math around it.

sofuu.ai.similarity(a, b); // cosine similarity, [-1, 1]
sofuu.ai.dot(a, b); // dot product
sofuu.ai.l2(a, b); // Euclidean distance
// Float32Array or number[]; both use the native kernels
const v = sofuu.ai.embedLocal("hello world"); // 768-dim, offline
const img = sofuu.ai.embedImage(imageBytes); // joint 64-dim text/image space
const e = await sofuu.ai.embed("hi", { provider: "openai", model: "text-embedding-3-small" });
Honest note: embedLocal is a trigram-hash embedder, not a neural one. It is excellent for exact-ish lexical recall, deduplication, and instant offline demos; it is not a substitute for a trained embedding model on hard semantic retrieval.
Memory

Persistent, encrypted, on disk.

A .qtsq container holds records, vectors, and retention state in one file. By default there is no server to run, no port to secure, and no separate database process holding your data. You can explicitly opt into a Qdrant, Pinecone, or pgvector adapter at memory.open; absent that option, the embedded engine is used.

Measured on this machine

macOS arm64, v0.3.0-beta, one process, 768-dimensional hash space, no network. Timings from performance.now(); recall is approximate nearest-neighbour over 500 queries after 50 warm-up queries.

OperationMeasured
embedLocal (offline embedding)3.4 µs per vector — 10,000 in 34 ms
remember (ingest, duplicate gate included)2,217 facts/s at 1,000 records · 1,499/s at 10,000 · 1,222/s at 50,000
recall (k=5)0.38 ms median at 10,000 records · 0.43 ms at 50,000
On disk1.2 MB per 10,000 facts · 6.1 MB per 50,000 (container plus segment directory)

Ingest slows as the store grows; recall does not. Measure your own workload — text length and vector dimension change these numbers.

Memory API reference
Agents

Agent controls and limits.

The agent loop supports opt-in tool packs, policy checks, budgets, approvals and session records. Enforcement and recovery depend on the execution path; these controls are not a complete security or uptime guarantee.

sofuu.agent.define({
name: "researcher",
system: "You research precisely. Cite sources.", // preserved byte-for-byte
tools: ["web", "memory", "system"], // domain toolpacks
budget: { maxSteps: 12, maxDepth: 2, maxWallMs: 300000 },
});
const r = await sofuu.agent.run("researcher", "Compare Rust and Zig");
r.answer; r.trace; r.usage; r.stopped;
ControlScope
Prompt sovereigntydef.system is used byte-for-byte; coding prompts and repository context are opt-in via preset: 'coding'.
Tool isolationThe coding, web, memory, and system packs are selected explicitly; non-coding agents inherit no coding tools.
PolicySeveral surfaces use the R7 evaluator; verify the boundary and approval behavior for the chosen surface.
BudgetsmaxSteps, maxDepth, maxWallMs, and token ceilings stop a run deterministically.
ApprovalsDeadline expiry is a typed deny; a resolved approval is recorded in the trace.
SessionsResume requires a usable checkpoint; redacted or truncated checkpoints may be refused.
Long context & token economy

Read more than the window holds.

sofuu.rlm.query runs a sandboxed reader loop over context far larger than the model window. Five tiny on-device gates advise when to refresh, compact, or drop context — advise-only, and switchable off.

// Answer over context far larger than any model window
const res = await sofuu.rlm.query(bigText, "What caused the outage?", {
provider: "openai", model: "gpt-4o",
maxRounds: 16, maxLlmCalls: 24, trace: true,
});
res.answer; res.calls; res.rounds; res.stopped;
// Tiny on-device gates advise the loop (hard off with SOFUU_NO_ML=1)
JSON.parse(sofuu.ml.freshness.score(text, task, ctx));
Limits

What this page is not claiming.

Throughput: Sofuu publishes a llama.cpp parity ledger built from real measurements, and on the latest receipts llama.cpp is faster on both prefill and decode for every measured model. Choose Sofuu for integration, policy, daemon removal, and embeddability — not for maximum tokens per second.
Training: Sofuu does not train or fine-tune models and does not do multi-GPU distributed serving. Use PyTorch/CUDA or a serving framework, then load the resulting GGUF or SafeTensors weights here.
Node compatibility: QuickJS is not V8. There is no N-API native-addon support, npm coverage is partial, and long pure-JS compute is slower than a JIT runtime. That trade buys a 5.39 MiB macOS binary that starts without a JIT warmup and runs on iOS.
FAQ

Common questions.

Does Sofuu require Python, Ollama, Docker, or CUDA?
No external runtime or daemon is required. The built-in inference engine loads GGUF and SafeTensors weights natively on the CPU and, on Apple Silicon, the Metal GPU — no Python, no CUDA, no Docker. Note: the built-in engine is experimental; for production inference, the recommended path is to use a llama.cpp server through the llamacpp provider.
Which local model formats are supported?
GGUF files and SafeTensors directories (config.json plus model.safetensors), plus PyTorch .pt and NumPy .npy/.npz tensors. Shipped kernels cover f32, f16, q4_K, q6_K, q8_0 and q3_K_S; the loader gate is authoritative for a given build. Note: the built-in engine is experimental — not all formats or quantisations are reliably supported yet.
Are the offline embeddings a trained model?
No. sofuu.ai.embedLocal is a deterministic 768-dimensional trigram-hash embedder with a zero-byte model file: instant and fully offline, but not a trained neural embedder. Use sofuu.ai.embed with a provider when semantic quality matters.
How is memory persisted and is it encrypted?
sofuu.memory encrypts .qtsq containers by default. Explicit plaintext mode and unencrypted brain-card exports exist; logs and external providers have separate data handling. Journaling and quarantine support recovery, but do not guarantee it.
Can I mix local inference and a cloud provider?
Yes. sofuu.ai.stream and sofuu.ai.complete work with OpenAI, Anthropic, OpenRouter, Ollama, or any compatible endpoint using your own key, while sofuu.ai.streamLocal runs a local model via the built-in engine (experimental — for production, point the llamacpp provider at a llama-server with the same API). Routing between them is application code, not a separate deployment.