MUNI FI Zero-shot, one-shot, few-shot PA154 Language Modeling (12.1) Pavel Rychly pary@fi.muni.cz May 14, 2024 Usage of Large Models training of big models on huge data is expensive (Long training time) fine tuning on small data of target task combining Language modeL with additionaL NN/Layer, training onLy new Layer ■ big modeL is frozen, onLy used Classification Entailment Start Text Extract -*■ Transformer Linear Start Premise Delim Hypothesis Extract Transformer Linear Similarity Start Text 1 Delim Text 2 Extract - Start Text 2 Delim Text 1 Extract Transformer Transformer Linear Pavel Ryehly • Zero-shot, one-shot, few-shot • May 14, 2024 2/8 Usage of LLM without training Usage of LLM without fine tuning ■ fine tuning is still expensive ■ models can predict reliably ■ using generation for end tasks ■ zero-shot no task-specific data/training Pavel Ryehly • Zero-shot, one-shot, few-shot • May 14, 2024 3/8 Zero-shot ■ formulate task in natural Language ■ task description + prompt ■Translate English to French: cheese => ■ Summarize the following paragraph into one sentence: text Pavel Ryehly • Zero-shot, one-shot, few-shot • May 14, 2024 4/8 One-shot ■ formulate task in natural Language and show one example ■ task description + example + prompt ■Translate English to French: seq otter => loutre de mar cheese => Pavel Ryehly • Zero-shot, one-shot, few-shot • May 14, 2024 5/8 Few-shot ■ formulate task in natural Language and show a few examples ■ task description + examples + prompt ■Translate English to French: seq otter => loutre de mar peppermint => menthe poivree plush girafe => girafe peluche cheese => Language Models are Few-Shot Learners Pavel Ryehly • Zero-shot, one-shot, few-shot • May 14, 2024 6/8 Token is MASK test sentence template POO labels I will visit Munich next week. Munich is a [MASK] P retrained Language Model predicts (V) 1 company }. QRG team city country man girl PER TOKEN is a MASK: Few-shot Named Entity Recognition with Pre-trained Language Models Pavel Ryehly • Zero-shot, one-shot, few-shot • May 14, 2024 7/8 Available LLM ■ Pythia ■ OPT ■ GALACTICA ■ TO Pavel Ryehly • Zero-shot, one-shot, few-shot • May 14, 2024 8/8