Meet OLLM: Bright Python Brand Brillion Brillion of the LLM LLM at 8 GB Consumers GPUS with SSD OffofLoad-Noty required

OLLM Is Hugginggalfall Library Libromers and Pytorch and running large via GPUS transformers by uploading NVIFIs with the loading of local metals. The project is intended to be offline, luggage of single-gpu activity and apparently avoid the weight, using FP16 / BF16 / BF16 instruments with Flashative 2-10 GB within 8-10 GB while handling 100 tokens 100

But what's new?

(1) KV Cache Read / Below That Bypass mmap to lower the RAM use; (2) Dislide-General Support of QWen3-next-80B; (3) LLAMA-3 Flashtatunt-2 of Fitness; and (4) the reduction of the GPT-OSS memory with “Flash-Attention – the Skin and Chunkled MLP. Table published by the Council of Council Reports Last – Memory / I / O / O / O / O Footprints in RTX 3060 TI (8 GB):

• QWEN3-NEXT-80B (BF16, 160 GB Weight, 50k CTX) → ~ 7.5 GB Vram + ~ 180 GB SSD; Noted by the release “≈ 1 tok / 2 s”.
• GPT-OSS-20B (Full of BF16, 10K CTX) → ~ 7.3 GB Vram + 15 GB SSD.
• LLAMA-3.1-8B (FP16, 100K CTX) → ~ 6.6 GB vram + 69 GB SSD.

How does this work

OLLM weight is located directly from SSD on GPU, kneeling offer kV cache in SSD, and optional portalats of shares in CPU. Using flashattings-2 via the internet softmax so the perfect matrix payable never placed in syntuy, and large MLP chunks guess with high memory. This removes a bottle from the Vram and latency, which is why the OLLM project emphasizes NVMS-class SSDs and Kyrio / cufile

Supported models and GPUS

In the box examples for examples cover LLAME 3 (1B / 3B / 8B), GPT-OSS-20Bbeside QWEN3-NEXT-80B. Library intends NVIDIA ampere (RTX 30xx, A-chunks), Adx 40xx, L4), and hopper; QWen3-Next requires the Development Development of the transitions (≥ 4.0.Dev). Especially, qwen3-Next-80B is Sparse Moe (80b Six, ~ 3B is working) Sellers generally set up a number of A100 / H100 items; Ollm's claim is that you know execute Offline on one customer GPU by paying SSD fines and receiving low issues. This puts in contrast with VLMs documents, raising various GPU servers in the same model family.

Installation and Small Use

The project has MIT License and is available on PPSpi (pip install ollm), in addition kvikio-cu{cuda_version} Relatement of high I / O. QWEN3-Next Desert Models, enter transformers from Gitity. Short Example on Ready show Inference(...).DiskCache(...) wiring and generate(...) with a bullbach broadcast text. (PyPPI currently shares 0.4.1; Roadme 0.4.2 Reforms.)

Expecting Works and Trading That Is Never

• Turn: The centerkeeper reports ~ 0.5 Tok the SSD LATENCY ROCK.
• The last pressure: Long conditions require major KV talks; OLLM writes these SSD to keep the Vram flat. This wide mirror applies to the uploading kv (eg
• Hardware testing reality: Running Qwen3-Next-80B “in Consumer Hardware” feasible According to Ollm-Centric Design, but the general detection of the model is still awaiting GPU servers. Carry OLLM as a great shape, offline shape than the restoration of stack productions like VLLM / tgi.

A bit of bitterness

OLLM presses the point of Design Design: Keep the high accuracy, push memory in SSD, and make Ultra-long conditions running on one 8 GB nvidia GPU. It will not be accompanied by the original-statmentpt, but the login and an analysis of the log, a key analysis, or stagnation of the provereration, Pravmatic Way to reach MOE-80B modeling for immediate domain and the 1-1 generation.

Look Gilbub Repo here. Feel free to look our GITHUB page for tutorials, codes and letters of writing. Also, feel free to follow it Sane and don't forget to join ours 100K + ml subreddit Then sign up for Our newspaper.