Abstract:

Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method to adapt large foundation models on downstream tasks. However, the weight updates are constrained to be low-rank structures, limiting their expressiveness. Alternatively, low-displacement rank (LDR)-based structured matrices are rank unrestricted, while requiring few parameters and supporting fast matrix- vector multiplication. We propose a new PEFT strategy to construct the weight updates with block-wise LDR matrices by sampling parameters from a hyper net- work framework. Our method, hyper low-displacement rank adaptation (HyDRA), offers high flexibility for choosing the size of a pool of trainable parameters, while not being restricted by the displacement rank. Our experiments demonstrate that the HyDRA can boost the classification accuracy by up to 3.4% and achieve two- fold improvement in parameter efficiency on an image classification benchmark compared with other PEFT variants.