Every conversation about artificial intelligence eventually reaches the same toll booth: the GPU. Modern AI is assumed to need racks of expensive, power-hungry accelerators from a single supplier, and every hospital, bank and government data centre that wants AI must queue at that gate. In Bengaluru, a deep-tech startup has spent years proving the gate was never necessary. Ziroh Labs’ Kompact AI platform runs modern AI models — text, speech, vision and multimodal — on the ordinary CPUs that already sit inside the world’s servers, workstations, laptops and edge devices.
Ziroh Labs is not a newcomer chasing the AI boom. The company’s roots lie in fully homomorphic encryption — the mathematics of computing on data without ever decrypting it — and in research led by CEO Hrishikesh Dewan, who is affiliated with the Indian Institute of Science. That encryption-era work exposed the team to transformer architectures before the generative AI wave, and when the GPU shortage began throttling who could build with AI, they already understood the problem from first principles. The company is guided by Whitfield Diffie, the Turing laureate and co-inventor of public-key cryptography, who has described the ability to run AI on widely available CPUs as “one of the most exciting things I have seen in quite a number of years.”
The engineering behind Kompact AI is a ground-up rewrite of the AI inference stack. “We wrote the entire stack from the ground up,” senior VP Vineet Mittal told DIGITIMES Asia. “Most optimizations, about 70% of them, came from the science of AI. About 30% came from rewriting system software close to the CPU, carefully managing L1/L2 caches, parallelism, and memory.” The result, the company says, is striking: Kompact AI delivers Nvidia A100-class inference performance on Intel Xeon CPUs, running open-source LLMs at 164 tokens per second on a batch size of one — and approaching H100-class throughput on Intel’s newer Gen 6 servers, per the company’s figures. Batching pushes enterprise workloads to around 2,500 tokens per second, and a proprietary caching layer called Elephant — which detects semantic similarity across queries, intermediate outputs and final outputs — takes repetitive workloads up to 10,000 tokens per second, according to the company.
Just as notably, Kompact AI does all of this in full precision. Where most CPU inference engines quantize models — a step the company warns can introduce subtle semantic errors in long-context work like legal or financial summarisation — Ziroh runs complete models in BF16, the datatype models are trained in. “Customers are already confused by fluctuating quality. It’s too early to sacrifice correctness for speed,” Mittal said. The platform supports models up to 32 billion parameters with 64,000-token context windows, runs on Intel AMX and AVX-512 as well as ARM, and ships OpenAI-compatible REST APIs with SDKs in eight languages.
The platform’s public journey began at IIT Madras, where Kompact AI was first unveiled on April 9, 2025, alongside the Centre of AI Research (CoAIR) established with IITM Pravartak Technologies Foundation to solve India’s compute-accessibility challenge. The team optimized 17 models — including DeepSeek, Qwen and Llama — for CPU inference, benchmarked with IIT Madras, and, the company says, tested by both Intel and AMD. IIT Madras Director Prof. V. Kamakoti summed up the ethos: “We’re demonstrating that you don’t need a revolver to kill a mosquito.” On September 7, 2026, Ziroh Labs globally launched the expanded Kompact AI platform from Bengaluru, taking it across multiple form factors and sectors — including packaged Kompact AI Box solutions for healthcare, education, retail, fintech and sovereign deployments — with partners already testing and implementing the platform, the company says.
The bet is as strategic as it is technical. A single GPU card can draw 750 watts to a kilowatt, Mittal notes — “untenable for India if we scale AI the same way the West is doing.” Kompact AI is designed for power-constrained deployment, data sovereignty and supply-chain resilience: it runs on the commodity x86 and ARM hardware enterprises and governments already own and control, deployable through Docker and Kubernetes across AWS, Azure, Google Cloud and private infrastructure, on-prem or at the edge. “Countries don’t know today which chips will be banned tomorrow,” Mittal said. “It’s important to build infrastructure that runs on open standards and hardware you already control.” The company’s long-term vision circles back to its origins — fully homomorphic encryption for national-security-grade AI — once transformer efficiency opens that window.
More Read
Ziroh Labs is also building the human layer: AI Centres of Excellence with academic institutions, free licences for academic users, and an IIT Guwahati collaboration on a Math Lab initiative and a 40-plus-hour AI and LLM course. Dewan’s framing is simple: “The next phase of AI adoption will require infrastructure that is accessible, efficient, and closer to the people and businesses using it.” In a world racing to concentrate AI inside a handful of GPU clouds, a Bengaluru startup is betting the future runs on the hardware everyone already has.
