Someone Just Ran a 397 Billion Parameter Model on a MacBook. The Local AI Business Is Real Now.

A developer called danveloper pushed a project to GitHub this weekend that changes the math on local AI. Flash-MoE is a pure C and Metal inference engine that runs Qwen3.5-397B, a 397 billion parameter Mixture-of-Experts model, on a MacBook Pro with 48GB of RAM. It runs at 4.4 tokens per second. That is fast enough for real-time conversations.

The entire model is 209GB on disk. It streams from the SSD through custom Metal compute shaders. No Python. No PyTorch. No cloud API calls. Just C, Objective-C, and hand-tuned GPU kernels.

This is not a toy demo. The model handles tool calling, structured JSON output, and multi-turn conversations at production quality. The 4-bit quantization preserves enough fidelity that you cannot tell the difference from a cloud API in most use cases.

Why this matters more than benchmarks suggest

The local AI conversation has been building for a while. llama.cpp made it possible to run small models on CPUs. Ollama made it easy. MLX optimized for Apple Silicon. But all of these hit a wall at around 70 billion parameters on consumer hardware.

Flash-MoE broke through that wall by exploiting a key property of Mixture-of-Experts architectures: only a fraction of the model's parameters activate for each token. Qwen3.5-397B has 512 experts per layer, but only 4 fire at any given time. That means you only need to load about 6.75MB per expert, per layer, from the SSD.

The MacBook Pro's NVMe drive reads at 17.5 GB/s. That is fast enough to stream experts on demand without stalling the GPU. The OS page cache handles the rest, keeping frequently used experts in RAM.

This matters because it removes the last argument for cloud-only AI in many business contexts. If a 397B model runs on a laptop, the question is no longer "can we run it locally?" It is "why are we still paying for API calls?"

The privacy premium is real

Every regulated industry has the same problem. Healthcare organizations need AI but cannot send patient data to OpenAI's servers. Law firms want AI-assisted document review but cannot risk client confidentiality. Financial institutions want AI analysis but compliance requires data residency.

The market for on-premise AI is growing fast. Grand View Research valued the on-premise AI market at $15.2 billion in 2025 and projects it to hit $47 billion by 2030. The compound annual growth rate is 25.3%.

Right now, most on-premise AI deployments require expensive GPU servers. NVIDIA A100s or H100s, running $15,000 to $40,000 per card. Flash-MoE suggests a different path: frontier-quality inference on $3,000 to $5,000 laptops and workstations.

Five businesses you can build around local AI

Privacy-first AI consulting

The most immediate opportunity. Law firms, medical practices, financial advisors, and government agencies all need AI and all have data they cannot send to the cloud.

Your offering: assess their AI needs, select and optimize the right local model for their use case, deploy it on their existing hardware or spec new machines, and train their team to use it.

Pricing is straightforward. A discovery call and assessment at $500 to $1,500. Deployment and optimization at $3,000 to $10,000. Ongoing support at $500 to $2,000 per month.

The key differentiator is that you understand both the AI capabilities and the compliance requirements. Most AI consultants know the models but not HIPAA. Most compliance consultants know HIPAA but not the models.

Productized local AI deployment

Instead of bespoke consulting, build a repeatable deployment package. Pick one vertical. Build an installer that sets up the right model with the right configuration for that use case. Document it. Price it.

For example: a "Local AI Legal Assistant" package that installs a quantized model optimized for legal language, comes with pre-built prompts for contract review and case research, includes a simple web UI for non-technical staff, and runs entirely on a Mac Studio.

Sell it for $5,000 as a one-time setup, or $500 per month as a managed service. One person can handle 10 to 20 clients before needing to hire.

Optimized inference engines

Flash-MoE itself is the product here. There are dozens of new model architectures shipping every month, and each one benefits from hardware-specific optimizations. Apple Silicon needs Metal shaders. NVIDIA consumer GPUs need CUDA kernels. AMD GPUs need ROCm support.

Build and sell optimized inference runtimes for specific hardware and model combinations. Charge per model or per hardware platform. Open-source the basic version and sell enterprise support, like llama.cpp but with a commercial layer.

Offline AI applications

Build applications that work without an internet connection. A medical scribe that transcribes and summarizes patient visits on a tablet at the bedside. A field inspection tool that analyzes photos and generates reports in locations without cell service. A legal research tool that runs on a laptop in a courtroom.

The value proposition is not "AI." It is "AI that works everywhere, always, with no data leaving your device." That is worth a premium.

AI workstation builder

The simplest version: spec and sell optimized hardware bundles for local AI. A Mac Studio with 192GB unified memory runs most models natively. A custom-built PC with an RTX 4090 handles CUDA workloads. Bundle the hardware with pre-installed models, a web UI, and setup documentation.

Charge a 20% to 30% markup on the hardware plus a $500 to $1,000 setup fee. Market to small businesses and professionals who want "their own ChatGPT" but cannot or will not use cloud services.

The technical edge that matters

Flash-MoE was built by one person in 24 hours with AI assistance. The project includes a research paper, 90 experiments, and production-quality code. This is the new pace of infrastructure development.

But the techniques are specific and learnable. SSD streaming with parallel pread() calls. FMA-optimized dequantization kernels. Fused Metal shaders for SwiGLU activation and RMS normalization. Deferred GPU compute that overlaps with CPU preparation.

If you learn these techniques now, you are one of maybe a few hundred people worldwide who can optimize inference for consumer hardware. That scarcity is temporary. But while it lasts, it pays well.

What to take from this

The cloud is not going away. But the assumption that serious AI requires cloud infrastructure just took a hit. A 397 billion parameter model running on a laptop at conversational speed means that for many use cases, local is good enough. And for privacy-sensitive use cases, local is better.

The businesses that emerge from this will not be about the technology. They will be about packaging it for people who need AI but cannot or will not send their data to someone else's server.

Pick a vertical. Learn the compliance requirements. Deploy a model. Charge for the expertise.