Spectral compression of neural networks: a Transformer-focused decomposition approach

Abstract

This thesis introduces the SQUEEZE (Spectral QUEnch and Eigenvalue ZEroing) framework, a compression method for Transformer architectures grounded in Random Matrix Theory (RMT). The approach reduces computational demands without retraining, enabling the deployment of large-scale encoders in resource-constrained environments. By employing the Marchenko-Pastur (MP) and Tracy-Widom (TW) distributions as noise baselines, the method identifies compressible components through a comparison of scaled singular values against the MP law. Analysis of BERT-base-uncased reveals that the matrix aspect ratio β = m/n governs compressibility: 92% of attention matrices (β ≈ 1.00) reject MP conformity, compared to only 35% of rectangular feedforward network layers (β = 0.25). SQUEEZE achieves an 8–9% size reduction on BERT-base-uncased with an accuracy loss of at most 2.5% across three GLUE tasks, and a 10.4% size reduction on BERT-large-uncased, where it proves to be the only method that avoids catastrophic degradation. However, the method fails on ALBERT-base-v2 due to its cross-layer parameter sharing, which violates the per-layer spectral independence required for MP diagnostics. Although quantization delivers higher compression (59%) on smaller models, SQUEEZE exhibits more graceful quality degradation and becomes the dominant quality-preserving strategy at larger scales. This work establishes rectangularity as a practical indicator of compressibility and per-layer independence as its architectural prerequisite, bridging theoretical spectral analysis with architecture-aware Transformer optimization.