A 744B-Parameter Model Now Runs on 25GB of RAM. That Turns Self-Hosted LLM Deployment Into a Sellable Service for Every Team That Can't Send Data to a Cloud API.

Colibrì is a small, unglamorous GitHub repo. It is also proof that the economics of self-hosted AI just moved.

The README states the goal plainly: "Run GLM-5.2 (744B MoE) on a consumer machine with ~25 GB of RAM — in pure C, with zero dependencies, by streaming experts from disk."

That is not a toy benchmark. GLM-5.2 is a "744B-parameter MoE" model that activates "only ~40B parameters per token" across "21,504 routed experts (75 MoE layers × 256 experts)." Running something that size used to mean a rack of GPUs. Colibrì runs it on a dev machine with "12 cores, 25 GB RAM, NVMe via VHDX."

The business idea

Every company currently paying per-token to a closed API for document review, internal search, or agent workflows has a hidden second option: run a frontier-scale open model on hardware they already own, and keep the data in the building.

That is the service. Not "AI strategy." Not "prompt engineering." Self-hosted LLM deployment, sized honestly against real hardware.

Specifically:

• audit the client's workload against real throughput numbers, not vendor marketing
• pick the model and quantization that fits the client's actual NVMe and RAM budget
• set up expert-streaming inference so a huge MoE model runs without a GPU cluster
• tune caching and batching so cold-start numbers don't kill the pilot
• document the hardware and storage runway so the client isn't surprised by the next model generation

Why this works now

Colibrì's own numbers are the pitch, because they are specific and unflattering in the right places, which makes them credible.

Cold decode is "~0.05–0.1 tok/s cold." That is unusable for live chat. But with warm cache and speculative decoding via MTP, it reaches "2.2–2.8 tok/forward measured." That gap, from unusable to workable, is exactly the kind of tuning problem a client will pay someone else to solve rather than discover themselves mid-project.

The model itself needs "~370 GB" on local NVMe for the int4 weights, "9.9 GB" resident RAM for dense parameters, and a "~30 s" load time. Those are concrete numbers a buyer can budget against, not abstract claims about "efficiency."

Colibrì is MIT-licensed, and GLM-5.2 itself is openly published on Hugging Face, so there is no vendor lock-in or licensing negotiation blocking a pilot.

Best customer profile

This is strongest for teams that:

• handle sensitive documents (legal, healthcare, finance) and cannot send them to a third-party API
• already have decent workstation or server hardware (NVMe storage, 16GB+ RAM) sitting underused
• run batch or async workloads (document review, research synthesis, internal knowledge search) where a few tokens per forward pass is fine, because nobody's waiting on a live chat response
• are nervous about recurring API spend that scales with usage

How to package the offer

1. Self-hosted feasibility audit

A short paid engagement. Map the client's workload, required context length, and existing hardware against real throughput numbers from projects like Colibrì.

2. Reference deployment

Set up expert-streaming inference for one real workflow, such as document extraction, internal search, or an agent backend, on hardware the client already owns.

3. Cache and batching tuning

Close the gap between "~0.05–0.1 tok/s cold" and "2.2–2.8 tok/forward" warm, tuned to the client's actual request pattern.

4. Hardware and storage sizing retainer

Models keep growing. Plan the NVMe and RAM runway before the next open-weight release outgrows the current box.

Source: https://github.com/JustVugg/colibri (via Hacker News)