Advanced Language Processing Winter School 2021
17–22 January 2021, Grenoble, France
● Seven Zoom Q&A sessions with pre-recorded lectures by the speakers.
● Three student poster sessions.
● A Zoom social session, playing Codenames by Vladimír Chvátil (FI alumnus).
Schedule
Outline
1. Kyunghyun Cho: Language Modeling (Lecture [1, 2, 3], Lab, Q&A)
2. Claire Gardent: Natural Language Generation (Lecture, Q&A)
3. Laurent Besacier: Self-Supervised Learning from Speech (Lecture, Q&A)
4. Yejin Choi: Neuro-Symbolic Common-Sense Knowledge (Lec.[1, 2], Q&A)
5. Grzegorz Chrupała: Visually-Grounded Models of Speech (Lecture, Q&A)
6. Tim Baldwin: NLP for User-Generated Content (Lecture, Q&A)
7. Isabelle Augenstein: Explainability for NLP (Lecture, Lab, Q&A)
8. Poster sessions
● Two Slack lab sessions.
Cho: Language Modeling and Machine Translation
● Cho is one of the authors of the Attention mechanism. (Bahdanau et al, 2016)
● Lecture introduces N-gram language models and motivates neural models:
○ N-gram language models suffer from data sparsity (traditional solutions: smoothing, backoff).
○ N-gram language models can’t capture long-term dependencies (no traditional solutions).
● For language modeling, lecture develops CBOW and RNNs (LSTM, GRU).
● For machine translation, lecture develops LSTM+Attention and Transformers.
● Lab shows how CBOW, recurrent models, LSTM+Attention, and Transformers
can be implemented and used for language modeling & machine translation.
Choi: Neuro-Symbolic Common-Sense Reasoning
● In the lecture, Choi frames common-sense reasoning as language generation:
○ Kahnemann (2003) argues intuitive reasoning is part of System 1, evoked by language.
○ Hofstadter and Sander (2013): “categories [..] outnumber words, require [..] text descriptions.”
● Several works by Choi et al. are introduced:
○ Unsupervised inference-time algorithms:
i. Reasoning through neural backpropagation (Qin et al., 2020)
ii. Reasoning through search with logical constraints (Lu et al., 2020)
iii. Reasoning through distributional neural imagination (West et al., 2020)
○ Supervised knowledge modeling algorithms:
i. Neural and symbolic common-sense knowledge (Hwang et al., 2020)
ii. Visually grounded common-sense knowledge (Park et al., 2020)
iii. Social, ethical, and moral norms (Forbes et al., 2020)
● Common-sense challenges by Choi et al. are introduced:
○ Physical/Social IQA, Visual/Abductive Cms. Reasoning, HellaSwag, WinoGrande, CosmosQA
Chrupała: Visually-Grounded Models of Speech
● In the lecture, Chrupała motivates visually-grounded modeling of speech:
○ Additional modality helps solve the lack of annotated data for most spoken languages.
● Existing datasets for visually-grounded modeling of speech are introduced:
○ MIT Flickr Audio Caption Corpus – 8K Flickr images with ~48 hours of spoken captions
○ Places Audio Captions – 100K images w/ spoken descriptions in English/Hindi (w/o captions)
○ Synthetically Spoken COCO, SPEECH COCO – 300K images w/ synthesized spoken captions
● Existing works on visually-grounded modeling of speech are introduced:
○ Cross-Channel Early Linguistic Learning (Roy and Pentland, 2002)
○ Deep Multimodal Semantic Embeddings for Speech and Images (Harwath and Glass, 2015)
○ Unsupervised Learning of Spoken Language with Visual Context (Harwath et al., 2016)
○ Representations of Language in a Model of Visually Grounded Speech (Chrupała et al., 2017)
○ Language Learning Using Speech to Image Retrieval (Merkx et al., 2019)
● Analyses of visually grounded model representations are discussed:
○ Bottom layers encode phonemes (form), top layers encode meaning. (Chrupała et al., 2020)
Augenstein: Explainability for Natural Lang. Proc.
● In the lecture, Augenstein motivates and defines explainability:
○ Decision Understanding – How does the model arrive at predictions for specific instances?
○ Model Understanding – What features and parameters has a model learnt?
● Several works of Augenstein et al. about explainability are introduced:
○ Decision Understanding:
i. Generating Fact Checking Explanations (Atanasova et al., 2020)
ii. A Diagnostic Study of Expl. Techniques for Text Classification (Atanasova et al., 2020)
○ Model Understanding:
i. Generating Label-Cohesive and Well-Formed Advers. Claims (Atanasova et al., 2020)
ii. TX-Ray: Quantifying & Explaining Model-Knowledge Transfer in (Un-)Supervised Natural
Language Processing (Rethmeier et al., 2020)
● Post-hoc explainability methods and evaluation measures are introduced:
○ Gradient-based, Perturbation-based, Simplification-based (see also the interpretability book)
○ Agreement with Human, Confidence Indication, Faithfulness, Rationale/Dataset Consistency