Initialization Techniques for LoRA in 1–3B Models: A Controlled Comparison and Why They Differ
Manuscript in preparation, 2026
About
I'm a graduate researcher in the Department of Computer Science and Electrical Engineering at UMBC. My work focuses on making large language models cheaper to adapt: how parameter-efficient methods like Low-Rank Adaptation (LoRA) behave in practice, and how seemingly minor design choices, such as initialization, change training dynamics and final performance.
I'm also interested in the systems side of AI: running training and fine-tuning workloads efficiently on high-performance and distributed computing infrastructure.
Research Interests
- Parameter-Efficient Fine-Tuning Adapting large pretrained models by training only a small fraction of their parameters.
- Low-Rank Adaptation (LoRA) Initialization, rank, and training behavior of low-rank adapters.
- High-Performance & Distributed Computing Scaling training and inference workloads efficiently across hardware.
- Artificial Intelligence Language models and the practical tradeoffs of training and deploying them.