MUNI
FI

Zero-shot, one-shot, few-shot
PA154 Language Modeling (12.1)
Pavel Rychly
pary@fi.muni.cz May 4, 2023
Usage of Large Models
training of big models on huge data is expensive (Long training time)
fine tuning on small data of target task
combining Language modeL with additionaL NN/Layer, training onLy new Layer
■ big modeL is frozen, onLy used
Classification     start        Text Extract
Transformer Linear
Entailment
Similarity
Start	Premise	Delim	Hypothesis	Extract
				
				
Start	Text 1	Delim	Text 2	Extract
-				
Start	Text 2	Delim	Text 1	Extract
Transformer
Linear
Transformer
Transformer
Linear
Pavel Rychlý • Zero-shot, one-shot, few-shot • May 4, 2023
2/8
Usage of LLM without training
Usage of LLM without fine tuning
■ fine tuning is still expensive
■ models can predict reliably
■ using generation for end tasks
■ zero-shot no task-specific data/training
Pavel Rychlý • Zero-shot, one-shot, few-shot • May 4, 2023
3/8
Zero-shot
■ formulate task in natural Language
■ task description + prompt
■Translate English to French: cheese =>
■ Summarize the following paragraph into one sentence:
text
Pavel Rychlý • Zero-shot, one-shot, few-shot • May 4, 2023
4/8
One-shot
■ formulate task in natural Language and show one example
■ task description + example + prompt
■Translate English to French: seq otter => loutre de mar cheese =>
Pavel Rychlý • Zero-shot, one-shot, few-shot • May 4, 2023
5/8
Few-shot
■ formulate task in natural Language and show a few examples
■ task description + examples + prompt
■Translate English to French: seq otter => loutre de mar peppermint => menthe poivree plush girafe => girafe peluche cheese =>
Language Models are Few-Shot Learners
Pavel Rychlý • Zero-shot, one-shot, few-shot • May 4, 2023
6/8
Token is MASK
test sentence
template
POO
labels
I will visit Munich next week. Munich is a [MASK]
P retrained Language Model
predicts (V)
1
company }. QRG team
city
country
man girl
PER
TOKEN is a MASK: Few-shot Named Entity Recognition with Pre-trained Language Models
Pavel Rychlý • Zero-shot, one-shot, few-shot • May 4, 2023
7/8
Available LLM
■ Pythia
■ OPT
■ GALACTICA
■ TO
Pavel Rychlý • Zero-shot, one-shot, few-shot • May 4, 2023
8/8