Soft Prompt

A soft prompt is a sequence of learned, continuous vectors that are prepended to a model's input to steer its behaviour. Unlike a regular (hard) prompt made of human-readable words, a soft prompt exists only as numbers in the model's embedding space — it is not text you can read.

Hard prompts vs soft prompts

A hard prompt is text you write: "You are a helpful customer service agent. Answer the following question politely and concisely." You craft it manually, iterating on wording until the model behaves as desired.

A soft prompt achieves the same goal differently. Instead of finding the right words, you learn a sequence of embedding vectors through training. These vectors are optimised directly to make the model produce the desired behaviour. The result is often more effective than any hard prompt because the optimisation is not constrained to actual words.

How soft prompting works

1. A sequence of virtual tokens (typically 10-100) is initialised randomly.
2. These tokens are prepended to the model's input.
3. The model processes the combined sequence normally.
4. The model's output is compared to desired output, and the soft prompt vectors are adjusted.
5. This process repeats until the soft prompt reliably produces the desired behaviour.

Crucially, the model's weights are frozen during this process — only the soft prompt vectors are updated. This makes soft prompting a very lightweight form of adaptation.

Advantages

• Extremely parameter-efficient: A soft prompt might contain only a few thousand trainable parameters versus billions in the full model.
• Task switching: You can swap different soft prompts in and out to change the model's behaviour without reloading it.
• No readable instructions: The steering mechanism is opaque, which can be an advantage for proprietary prompting strategies.

Limitations

• Not human-readable: You cannot inspect a soft prompt to understand what it "says." It is a black box.
• Model-specific: A soft prompt trained for one model does not transfer to another.
• Limited capability: For complex behavioural changes, fine-tuning or LoRA typically outperforms soft prompting.
• Requires training: Unlike hard prompts, you need a dataset and training pipeline.

Soft prompting vs other adaptation methods

Soft prompting sits between prompt engineering (no training, text-based) and fine-tuning (extensive training, weight changes). It requires some training but changes far fewer parameters than fine-tuning. LoRA has largely overtaken soft prompting in popularity because it offers more capability at a similar cost.