AI Research & Benchmarking

Breaking the VRAM wall: powerful models with less compute.

Applied research on running dense language models (LLMs) on budget consumer hardware (4 GB VRAM GPU). Combining extreme quantization (INT4/INT3/INT2), Hessian-weighted compensation (GPTQ), non-uniform quantization (Lloyd-Max), and low-rank residual error patches.

View Scoreboard ← Back to portfolio

Reproducible testbed · PyTorch 2.6 CUDA + Ollama · NVIDIA RTX 3050 Laptop GPU

Constrained Hardware (4 GB) RTX 3050 · 2026

The Core Objective

A standard 7B model in FP16 weighs ~14 GB and spills over into system RAM/PCIe, causing severe latency degradation. Compressing to ~2.7 bits reduces it to ~2.4 GB, fitting fully within a budget 4 GB GPU.

1.01× PPL
Lossless INT4
1.15× PPL
Recovered INT3
~2.7 bits
INT2 + Low-Rank Patch
Baseline (1B)
129.6 tok/s
Gemma 1B 100% on GPU (Ollama)
Baseline (3B)
82.6 tok/s
Llama 3.2 100% on GPU (Ollama)
Theoretical Compression
5.8× reduction
From FP16 (16 bits) to ~2.7 effective bits
Wall Broken
12.45× PPL
Pure 2-bit with clipped error GPTQ
Methodology & Findings

4 Core Experimental Discoveries

Finding 1 · Hessian Quantization

GPTQ Solves INT4 and INT3 Completely

Naive uniform quantization collapses catastrophically. By minimizing activation error ‖(W−Q)X‖² using Hessian H = XXᵀ and column-by-column compensation, INT4 reaches 1.01× (indistinguishable from FP32) and INT3 reaches 1.15×, allowing 7B models in ~2.6 GB.

Finding 2 · The INT2 Wall

Why 2-bit Diverges

At 2 bits, classic GPTQ diverges (22,820× error) because the feedback loop amplifies error residuals. Introducing error-feedback clipping at ±2.5σ stabilizes the loop, jumping from 288× to 12.45× at pure 2 bits.

Finding 3 · Orthogonal Stacking

Low-Rank Residual Patches (HADA-LR)

Correcting quantization residuals in activation space with a low-rank approximation drops error drastically: rank-16 reaches 1.54× at 2.67 bits and rank-32 reaches 1.14× at 3.33 bits.

Experimental Scoreboard

Quantitative Results (Perplexity Ratio vs. FP32)

Evaluated on laboratory model (GPT-2, base PPL = 3.013). Lower is better (1.00× = identical to unquantized FP32 baseline).

Target Precision Uniform Quantization Lloyd-Max (Non-uniform) GPTQ (Hessian) GPTQ + Low-Rank Patch Verdict
INT4 (4 bits) 1.14× 1.03× 1.01× 1.01× Solved (Lossless)
INT3 (3 bits) 546× (broken) 1.95× 1.15× 1.05× Solved (Production)
Pure INT2 (2 bits) 2,588× (unusable) 508× 12.45× (damped) 2.32× (rank-8) Frontier Passed
Hybrid (~2.67 bits) — — — 1.54× (rank-16) Sweet Spot (VRAM / Quality)
💡 Technical Takeaway: By stacking Hessian compensation with low-rank activation residual corrections, a 7B parameter model drops from 14 GB down to ~2.4 GB, fitting fully on a 4 GB GPU without PCIe latency penalties.

Engineering Applied to Efficiency

From theoretical mathematics (Cholesky, orthogonal rotations, SVD) to high-speed execution on real consumer hardware.

Contact Back to projects