Natural Language Modelling
PA154 Language Modeling (9.1)
Pavel Rychlý
pary@fi.muni.cz April 13, 2023
Deep Learning
■ deep neural networks
■ many Layers
■ trained on big data
■ using advanced hardware: GPU,TPU
■ supervised, semi-supervised or unsupervised
Pavel Ryehly • Natural Language Modelling • April 13, 2023
2/25
Neuron
■ basic element of neural networks
■ many inputs (numbers), weights (numbers)
■ activation (transfer) function (threshold)
■ one output: y = (/>(52T=o wjxj +
Pavel Rychlý . Natural Language ModeWi§g^5?p?i
Neural Networks
■ Input/Hidden/Output Layer
■ Input/output = vector of numbers
■ hidden Layer = matrix of parameters (numbers)
m
yk = <f>(Y^ wkjXj) Y = <p(WXT)
Hidden
Pavel Ryehly • Natural Language Modelling • April 13, 2023
4/25
Activation Functions
■ crucial component of NN
■ non-Linear function
■ many Layers without non-Linear activation functions are equivaLent to single Layer (Linear combination of inputs)
Leaky ReLU
max(0.l£, x)
ReLU
https://en.wikipedia.org/wiki/Activation_function
Pavel Ryehly • Natural Language Modelling • April 13, 2023
One-hot representation
■ words/cLasses: [0 0 0 1 0 0 0 0]
■ for each word/cLass one input
■ one input activated (1), others deactivated (0)
■ whole input vector couLd be Large = size of vocabulary (^30k)
■ sequence of words requires sequence of one-hot vectors
■ first Layer transforms words into word embeddings
■ usually not represented explicitly as vectors, using one single number
■ output:
■ one-hot vector during training = expecting one word/class
■ probability distribution during usage
■ cannot be represented using single number
Pavel Ryehly • Natural Language Modelling • April 13, 2023
6/25
Training Neural Networks
■ supervised training
■ example: input + result
■ difference between output and expected result (Loss function)
■ adjusts weights according to a Learning ruLe
■ backpropagation (feedforward neuraL networks)
■ gradient of the Loss function, stochastic gradient descent (SGD)
Pavel Ryehly • Natural Language Modelling • April 13, 2023
7/25
Training Language Models
■ core function of Language models: predict the following word:
P(X5|X1JX2JX3JX4)
■ input: context = Xi,x2,X3,x4
■ output: probability distribution of the following word
■ training:
■ input: xi,X2,X3,X4
■ output: x5 (one-hot vector)
■ training data:
■ get any text (copus)
■ extract all n-grams
Pavel Ryehly • Natural Language Modelling • April 13, 2023
8/25
Why are NNs better than statistics
■ continues space representation
■ items (words/tags/classes) are not atomic
■ no sparsity problem
■ don't need to store all observed n-grams
■ vectors handles relations
■ many realations, not explicit, unknown
■ NN can represent any function (if deep enough)
■ structure of the function is not pre-defined
Pavel Ryehly • Natural Language Modelling • April 13, 2023
9/25
Why are NNs used only Last 10 years
■ big training data
■ powerful hardware
■ Moore's law: memory size, processor's speed doubles every few years
■ matrix processing using GPU,TPU
■ better Learning strategies, NN optimizatons
■ Adam, AdaFactor optimizers
■ dropout
■ attention
■ ready to use Libraries/framewoks, datasets
Pavel Ryehly • Natural Language Modelling • April 13, 2023
10/25
A fixed-window neural Language Model
v = vocabulary size
d = embedding size
h = hidden Layer size
Input: IDs of words (sparse representation of one-hot vector)
E: embeddings (v x d)
W: hidden Layer (4d x h)
U: output Layer (h x v)
books
laptops
1
zoo
u
OOOOOOllltll
w
OOOO   OOOO   OOOO 0000
the     students opened
their
xm       x(2)        x(t) x(4)
Pavel Rye hly • Natural Language Modelling • April 13, 2023
11/25
A fixed-window neural Language Model
E: embeddings (v x d)
W: hidden Layer (4d x h)
U: output Layer (h x v)
A = E(X)
B=f(WA)
Z = softmax(UB)
books
laptops
1
ill
zoo
u
OOOOOOllltll
w
OOOO   OOOO   OOOO 0000
the     students opened
their
xm       x(2)        x(t) x(4)
Pavel Rye hly • Natural Language Modelling • April 13, 2023
12/25
A fixed-window neural Language Model
books
fixed window is too small
enlarging window enlarges W
window can never be Large enough!
x1 and x2 are multiplied by completely different weights in W.
No symmetry in how the inputs are processed.
laptops
1
ill
zoo
u
OOOOOOllltll
w
OOOO   OOOO   OOOO 0000
the
students opened
x(2) x(3)
their
Pavel Rye hly • Natural Language Modelling • April 13, 2023
13/25
Recurrent Neural Network (RNN)
■ dealing with Long inputs
■ feedforward NN + internal state (memory)
■ finite impulse RNN: unroUto strictly feedforward NN
■ infinite impulse RNN: directed cyclic graph
■ additional storage managed by NN: gated state/memory
■ backpropagation through time
Pavel Ryehly • Natural Language Modelling • April 13, 2023
Problem of Long-Term Dependencies
the clouds are in the sky.
©		©		©		©		©
t		t		t		t		t
A	—►	A	—►	A	—►	A	—►	A
X
X
X
/ grew up in France... I speak fluent French .
©		©		©		©				
t		t		t		t		t		t
A	—►	A	—►	A	-►	A	—►	A	—►	A
Pavel Ryehly • Natural Language Modelling • April 13, 2023
15/25
Long short-term memory (LSTM)
■ LSTM unit: cell, input gate, output gate and forget gate
■ cell = memeory
■ gates regulate the flow of information into and out of the cell
Pavel Rye hly • Natural Language Modelling • April 13, 2023
16/25
LSTM - Gates
Hidden state
H
(-]
Forget
gate
F
Input X
Input
gate
t
Output gate
O
<7
FC layer with activation function
J       Copy —
Concatenate
Pavel Rye hly • Natural Language Modelling • April 13, 2023
17/25
LSTM - Input
Hidden state
Lf-]
Forget
gate
F
Input X,
Input gate
1,
i
I
Input node
Output gate
O
		u		tanh		° i
J
(7
FC layer with activation function
j     copy ~Y
Concatenate
Pavel Rye hly • Natural Language Modelling • April 13, 2023
18/25
LSTM - Memory
Memory tell internal state
Hidden state
lr-]
Forget gate F
Input gate
1
Input ^ node T
1
Input X,
Output gate
O.
1
a		a		tanh		
CT
FC layer with activation function
o
Element wise operator
J       Copy —
Concatenate
Pavel Rye hly • Natural Language Modelling • April 13, 2023
19/25
LSTM - Hidden state
Memory cell internal stale C
i-1
Hidden state It
MB
Forget gate F
1
•G>
Input gate
Input
©
Output
<7	I,	Ü		tanh		<7
1
1
II
Input XJ
FC layer with activation function
o
Element wise operator
J       Copy —
Concatenate
Pavel Rye hly • Natural Language Modelling • April 13, 2023
20/25
GRU, BRNN,...
■ Gated recurrent unit (GRU)
■ fewer parameters than LSTM
■ memory = output
■ Bi-directional RNN
■ two hidden Layers of opposite directions to the same output
■ hierarchical, multilayer
Pavel Ryehly • Natural Language Modelling • April 13, 2023
21/25
Encoder-Decoder
variable input/output size, not 1-1 mapping two components
Encoder: variable-Length sequence —>► fixed size state Decoder: fixed size state —>► variable-Length sequence
Encoder
Decoder
Pavel Ryehly • Natural Language Modelling • April 13, 2023
Sequence to Sequence
Learning
■ Encoder: Input sequence -> state
■ Decoder: state + output sequence
-> output sequence
Encoder
Decoder
T
They
t r
are watching
J
T
<eos>
lis
i
<bos>
regardent
I
IIS
regardent
<eos>
i
Pavel Ryehly • Natural Language Modelling • April 13, 2023
23/25
Sequence to Sequence
Using
Encoder: Input sequence —>► state Decoder: state + sentence delimiter
-> output
Encoder
Decoder
T } t t
They      are watching
T
<eos>
<eos>
lis regardent
I f L_J) L-JT L->7
<bos>
Pavel Rye hly • Natural Language Modelling • April 13, 2023
24/25
Multi-layer encoder/decoder
Encoder
Decoder
n x
Recurrent
Embedding
Sources
Recurrent
I
Embedding
Targets
Pavel Rye hly • Natural Language Modelling • April 13, 2023