Build AI applications in-process.
Local model inference, embeddings, agents, and encrypted vector memory are primitives of the runtime, not a stack of packages and daemons. One binary runs your TypeScript and the model in the same address space.
The short, checkable answers.
Written to be quoted accurately by search engines and AI assistants.
| Question | Answer for Sofuu |
|---|---|
| What is Sofuu for AI applications? | Sofuu is a self-contained runtime for AI — the model, the memory, and the agent loop in one address space. in-process. Local LLM inference (experimental built-in engine), embeddings, vector math, agents, tools, MCP, and memory are runtime primitives built into a single native binary — not packages you install around a web runtime. |
| Who is it for? | JavaScript and TypeScript developers building offline-first, private, or single-binary AI features: local copilots, CLI agents, MCP servers, edge services, robotics, and apps shipping AI on-device. |
| What does it do? | Executes JS/TS on QuickJS with a libuv event loop; loads GGUF and SafeTensors weights and generates text (experimental engine; production: the llamacpp provider); embeds text and images offline; stores vectors in encrypted .qtsq containers; runs agent loops with toolpacks, budgets, approvals, and one deterministic security policy; and speaks Model Context Protocol as both client and server. |
| When should you use it? | When you want a local model, memory, and tools inside one process with no Ollama daemon, Python environment, vector database, or Docker; when you need deterministic policy around shell, filesystem, and network; or when the app must run offline or ship to a device. Note: the built-in inference engine is experimental — for production inference, point the llamacpp provider at a llama.cpp server, or use a cloud provider, through the same sofuu.ai.stream API. |
| When should you not use it? | When you are training or fine-tuning models, need multi-GPU serving, require full Node.js API parity or native V8 addons, or when maximum single-model tokens per second is the deciding factor — llama.cpp is ahead of Sofuu there and we publish the ledger. |
The 10-second hello world.
No API key and no model download: open an encrypted store, embed a fact offline, and recall similar hash vectors. This is ordinary JavaScript calling native primitives.
// 1. Open an encrypted, on-disk vector brainconst brain = sofuu.memory.open("./brain.qtsq", 768);// 2. Embed offline (0-byte model) and rememberbrain.remember(sofuu.ai.embedLocal("native AI runtime"), "Sofuu runs models in-process.");// 3. Recall similar hash vectors via SIMDconst [hit] = brain.recall(sofuu.ai.embedLocal("fast runtime"), 1);console.log(hit.text);
The file persists. Reopen the same .qtsq in a later process and the memory is still there, encrypted at rest.
Run the model where the code runs.
llamacpp provider — Sofuu routes to it with the same sofuu.ai.stream API and a llama-server endpoint.Point Sofuu at a GGUF file or a SafeTensors directory. Weights are memory-mapped zero-copy; on Apple Silicon the Metal path is opt-in, and the exact CPU tile path is the default. KV compaction and sliding-window attention keep long sessions inside the context limit.
# One command, no daemon$ sofuu infer ./model.gguf "Explain SIMD in one sentence"// Or stream it inside your own programconst stream = sofuu.ai.streamLocal("./model.gguf", "Explain SIMD");for await (const chunk of stream) process.stdout.write(chunk.text);
Formats and kernels
GGUF, SafeTensors, PyTorch .pt, NumPy .npy/.npz. Shipped kernels: f32, f16, q4_K, q6_K, q8_0, q3_K_S. KV cache:f16, kivi4, kivi2.
Determinism is declared
Every engine path carries a tier: cpu-tiles is exact,metal-mma is tolerant with a published bound, mmq is best-effort and unreachable without an explicit opt-in flag.
Beyond chat
The same engine backs sofuu infer, sofuu.ai.streamLocal,sofuu chat -m, and the OpenAI-compatible /v1 endpoints served by sofuu serve.
One API, local or hosted.
Bring your own key for OpenAI, Anthropic, OpenRouter, or any compatible endpoint. There is deliberately no default model — you choose, per call or once in ~/.sofuu/config.json.
// A model is required — there is no defaultconst stream = sofuu.ai.stream("What is the speed of light?", {provider: "openai", model: "gpt-4o",});for await (const chunk of stream) process.stdout.write(chunk.text);const r = await sofuu.ai.complete("What is 2 + 2?", { provider: "openai", model: "gpt-4o" });r.text; r.usage; // "4" · { promptTokens: 12, completionTokens: 1 }
Vectors without a model file.
embedLocal is a deterministic 768-dimensional trigram-hash embedder: instant, offline, zero bytes of weights. When you need a trained semantic model, call sofuu.ai.embed with a provider — same call shape, same vector math around it.
sofuu.ai.similarity(a, b); // cosine similarity, [-1, 1]sofuu.ai.dot(a, b); // dot productsofuu.ai.l2(a, b); // Euclidean distance// Float32Array or number[]; both use the native kernelsconst v = sofuu.ai.embedLocal("hello world"); // 768-dim, offlineconst img = sofuu.ai.embedImage(imageBytes); // joint 64-dim text/image spaceconst e = await sofuu.ai.embed("hi", { provider: "openai", model: "text-embedding-3-small" });
embedLocal is a trigram-hash embedder, not a neural one. It is excellent for exact-ish lexical recall, deduplication, and instant offline demos; it is not a substitute for a trained embedding model on hard semantic retrieval.Persistent, encrypted, on disk.
A .qtsq container holds records, vectors, and retention state in one file. By default there is no server to run, no port to secure, and no separate database process holding your data. You can explicitly opt into a Qdrant, Pinecone, or pgvector adapter at memory.open; absent that option, the embedded engine is used.
AES-256-GCM at rest
Brain storage is encrypted by default. Explicit plaintext mode and unencrypted brain-card exports exist; logs and remote providers have separate data handling.
Decay with an exemption for pins
Cold records fade out of recall; facts saved with
/rememberdo not.Approximate recall, exact dedup
Queries walk a neighbour graph; duplicate inserts are rejected by text hash in O(1).
Crash-tolerant writes
Append-only segments plus journal replay; a torn segment is quarantined, not fatal.
Measured on this machine
macOS arm64, v0.3.0-beta, one process, 768-dimensional hash space, no network. Timings from performance.now(); recall is approximate nearest-neighbour over 500 queries after 50 warm-up queries.
| Operation | Measured |
|---|---|
embedLocal (offline embedding) | 3.4 µs per vector — 10,000 in 34 ms |
remember (ingest, duplicate gate included) | 2,217 facts/s at 1,000 records · 1,499/s at 10,000 · 1,222/s at 50,000 |
recall (k=5) | 0.38 ms median at 10,000 records · 0.43 ms at 50,000 |
| On disk | 1.2 MB per 10,000 facts · 6.1 MB per 50,000 (container plus segment directory) |
Ingest slows as the store grows; recall does not. Measure your own workload — text length and vector dimension change these numbers.
Agent controls and limits.
The agent loop supports opt-in tool packs, policy checks, budgets, approvals and session records. Enforcement and recovery depend on the execution path; these controls are not a complete security or uptime guarantee.
sofuu.agent.define({name: "researcher",system: "You research precisely. Cite sources.", // preserved byte-for-bytetools: ["web", "memory", "system"], // domain toolpacksbudget: { maxSteps: 12, maxDepth: 2, maxWallMs: 300000 },});const r = await sofuu.agent.run("researcher", "Compare Rust and Zig");r.answer; r.trace; r.usage; r.stopped;
| Control | Scope |
|---|---|
Prompt sovereignty | def.system is used byte-for-byte; coding prompts and repository context are opt-in via preset: 'coding'. |
Tool isolation | The coding, web, memory, and system packs are selected explicitly; non-coding agents inherit no coding tools. |
Policy | Several surfaces use the R7 evaluator; verify the boundary and approval behavior for the chosen surface. |
Budgets | maxSteps, maxDepth, maxWallMs, and token ceilings stop a run deterministically. |
Approvals | Deadline expiry is a typed deny; a resolved approval is recorded in the trace. |
Sessions | Resume requires a usable checkpoint; redacted or truncated checkpoints may be refused. |
Read more than the window holds.
sofuu.rlm.query runs a sandboxed reader loop over context far larger than the model window. Five tiny on-device gates advise when to refresh, compact, or drop context — advise-only, and switchable off.
// Answer over context far larger than any model windowconst res = await sofuu.rlm.query(bigText, "What caused the outage?", {provider: "openai", model: "gpt-4o",maxRounds: 16, maxLlmCalls: 24, trace: true,});res.answer; res.calls; res.rounds; res.stopped;// Tiny on-device gates advise the loop (hard off with SOFUU_NO_ML=1)JSON.parse(sofuu.ml.freshness.score(text, task, ctx));