LoRA

How it is used

The most popular parameter-efficient fine-tuning (PEFT) method. Teams train a LoRA adapter for each task or customer and swap adapters on top of one base model. QLoRA combines it with quantization so that it fits on smaller GPUs.

Example sentences

1. With LoRA we fine-tuned the 8B model on a single GPU in one afternoon.

2. Each customer has its own LoRA adapter, but they all share the same base model.

Common collocations

• LoRA adapter
• train a LoRA
• LoRA rank
• QLoRA
• merge the LoRA weights

Related terms

• fine-tuningExtra training of an already pretrained model on a smaller, task-specific dataset, so that it does better on that task or follows a particular style.
• base modelA model that has only been pretrained, before any fine-tuning or instruction training. It continues text well but does not reliably follow instructions or chat.
• quantizationStoring a model's weights, and sometimes its activations, with fewer bits, for example 8-bit or 4-bit numbers instead of 16-bit, to make it smaller and faster.
• parameterA named input that a function or method accepts, listed in its definition. The actual value you pass in when you call it is called the argument.

This word in other fields

The same English word can have a different meaning in another field:

• LoRA · Artificial Intelligence & Machine Learning

More terms: Machine Learning and Data Processing

• artificial intelligence
• machine learning
• model
• dataset
• training data
• label
• train
• supervised learning
• neural network
• deep learning
• loss function
• overfitting