Can compression make large language models practical on smaller hardware?
A peer-reviewed survey of 165 studies maps how pruning, quantisation, distillation and low-rank adaptation can reduce resource demands. It finds a toolkit, not one proven recipe—and benchmark differences obstruct fair comparison.
The short version
- What happened
- The survey began with about 500 records, screened roughly 300 after deduplication, assessed 210 abstracts and retained 165 studies after full-text review.
- Who it affects
- Smaller models could give schools, clinics, public services and small firms lower-cost or offline tools, but only if task-specific quality and safety survive compression.
- What remains unproven
- This is a qualitative survey of 165 studies, not a meta-analysis with a pooled estimate of quality, latency, memory or energy savings.
Keep in mind
This is a qualitative survey of 165 studies, not a meta-analysis with a pooled estimate of quality, latency, memory or energy savings.
Included studies use heterogeneous models, datasets, benchmarks, hardware and definitions of compression cost.
The initial search cutoff was 18 January 2026, with targeted later additions, in a field where models and toolchains change rapidly.
Average benchmark retention may conceal losses in rare tasks, languages, calibration, security or safety behaviour.
The authors declared no known competing interests and reported Korean government research funding.
Source analysis · 2 linked sources. Primary analysed source date: 14 October 2026.
Read the full report · 8 min The evidence, methods, implications and limitations
The answer: smaller deployment is possible, but every saving moves a trade-off
Compression can make a large language model cheaper to store and run, and targeted fine-tuning can preserve or restore useful task performance. A new peer-reviewed survey finds no universal recipe that guarantees the same quality on smaller hardware. The outcome depends on which parameters are removed or represented more coarsely, what knowledge is transferred, which tasks matter and where the model will run.
The review's central argument is that compression and adaptation should not be designed as separate afterthoughts. Pruning may remove weights or structures; quantisation uses lower-precision numerical representations; knowledge distillation trains a smaller model from a larger teacher; low-rank and other parameter-efficient methods update a limited set of parameters. Combining them can reduce memory, latency or training cost, but the sequence can also compound errors.
For organisations considering local or edge deployment, this is a more useful message than a headline compression ratio. A smaller model that loses rare clinical language, minority dialects or safety behaviour may be cheaper and still be unsuitable. The target is not minimum size; it is the lowest resource cost that preserves performance on the work and people who matter.[1][2]
How the 165-study survey was assembled
The authors searched Google Scholar, arXiv, IEEE Xplore, the ACM Digital Library, ScienceDirect, ACL Anthology, OpenReview and references from earlier reviews. Query families combined terms for foundation and large language models with pruning, quantisation, distillation, low-rank approximation, fine-tuning, parameter-efficient adaptation and alignment. The initial cutoff was 18 January 2026, followed by targeted complementary searches.
About 500 records were identified. After duplicates, overlapping preprint and article versions and clearly irrelevant items were removed, roughly 300 remained. Abstract screening reduced that set to 210, and full-text screening produced 165 included studies. Of those, 145 informed the core methodological synthesis, while the remainder contributed to the discussion of future directions.
This is a substantial map of the field, but it is not a statistical meta-analysis. The paper does not pool a common effect size because studies use different base models, tasks, data, hardware, quality metrics and compression budgets. The counts describe the review process; they do not make each method equally mature or independently replicated.[1]
Pruning and quantisation remove different kinds of cost
Pruning removes parameters, connections, attention heads, layers or other structures judged less important. Unstructured pruning can create many zero weights but may need specialised kernels to deliver real speed gains. Structured pruning removes hardware-friendly units and can improve practical latency, but a coarse cut may discard capabilities that are difficult to recover.
Quantisation keeps the architecture while representing weights and sometimes activations with fewer bits. Moving from high precision to 8-bit, 4-bit or more aggressive formats can sharply reduce memory and data movement. Actual acceleration depends on hardware and software support. A nominally smaller representation may not yield the expected user-visible speed if conversion, dequantisation or unsupported operations become bottlenecks.
Both methods can preserve average benchmark scores while changing the pattern of errors. Evaluation therefore needs more than an overall accuracy number. Teams should test long contexts, rare instructions, tool calls, languages, calibration and safety-sensitive cases at the exact precision and runtime intended for production.[1]
Distillation and low-rank adaptation change what the model learns
Knowledge distillation trains a smaller student to reproduce outputs, intermediate representations or reasoning traces from a larger teacher. It can transfer useful behaviour without copying the full parameter count. It can also transfer the teacher's systematic mistakes, stylistic preferences and blind spots, while the student may struggle most on examples outside the distillation set.
Parameter-efficient fine-tuning methods such as low-rank adaptation leave most base weights fixed and train a small set of additional parameters. That reduces the memory and compute needed to specialise a model and allows separate adapters for different tasks. It does not reduce every inference cost unless adapters are merged or paired with compression, and multiple adapters create versioning and evaluation work.
The survey highlights the interaction between these choices. Quantisation-aware fine-tuning can help a model adjust to low precision; pruning followed by adaptation may recover task performance; distillation can create a smaller base before domain tuning. The best order is conditional on the model, budget and task, and comparative studies do not yet support one default pipeline.[1]
What smaller models could change for people and organisations
Lower memory and compute requirements can make local deployment possible on a workstation, phone or constrained server. That may reduce round-trip latency, permit operation with intermittent connectivity and keep some sensitive inputs inside an organisation. Smaller systems can also broaden access for universities, public services and small businesses that cannot fund a large accelerator cluster.
Those benefits are conditional. Running locally does not automatically protect privacy if logs, telemetry or updates leave the device. A cheaper model can still produce harmful advice, insecure code or biased decisions. Compression may also hide capability loss behind a smooth interface, leaving teachers, clinicians and public servants to discover failures in practice.
Procurement teams should therefore demand results on representative tasks, peak memory, energy per completed task, latency distribution and error severity—not only model size. Human review time belongs in the cost calculation. If a compressed system creates more ambiguous outputs that staff must check, the infrastructure saving may shift work rather than remove it.[1]
The evidence is broad but difficult to compare
The field changes faster than a journal review cycle. The search cutoff was January 2026 with later targeted additions, so models and toolchains released after that point may be absent or only lightly covered. Preprints and conference work form a material part of fast-moving systems research, while replication on current hardware can lag behind claimed improvements.
Benchmark heterogeneity is the deeper limit. A method tested on one model family and general-language benchmark may not carry over to multilingual dialogue, retrieval, coding or a safety-critical domain. Reported compression ratios are not comparable when one paper counts only weights and another includes activations, caches, adapters and runtime overhead.
The authors report no known competing financial interests. Funding came from Korean government-backed information and communications technology programmes, including an AI graduate-school programme and generative-AI workforce development. Funding disclosure does not resolve the comparability problem, but it makes the institutional context visible.[1]
What evidence would change the assessment
A stronger evidence base would use common reporting standards across multiple model families and hardware platforms. Each study should publish the uncompressed baseline, exact model and dataset versions, compression budget, fine-tuning data, total training and inference energy, peak memory, latency percentiles and uncertainty across repeated runs.
Independent evaluations should measure capability retention by task and subgroup, not only an aggregate benchmark. Red-team and reliability tests need to be rerun after every compression and adaptation step because safety behaviour may not degrade in the same way as average accuracy. Open artefacts and executable configurations would make claimed efficiency gains easier to reproduce.
The assessment would become more confident if several independent teams showed that the same combined pipeline preserved specified capabilities and safety thresholds while reducing real end-to-end cost on different hardware. Evidence that benchmark gains disappear under production workloads, or that rare but consequential errors rise, would argue for larger safety margins. Until then, compression is a configurable engineering toolkit—not a universal shortcut.[1][2]