Abstract

The presentation will consist of two parts. Firstly we will discuss SOTA in Neural Machine Translation and my current progress with it.

We will briefly discuss used tools, techniques, and libraries such as Sentencepiece, Fairseq, and Weights and Biases.

The second part will be more theoretical and will consist of an overview of currently available compression techniques for neural networks. These approaches are general and can be applied o any trained NN model from computer vision through bioinformatic to NLP.

We will discuss the following compression techniques:

• quantization
• distillation
• Mobile BERT
• Lottery ticket hypothesis

Readings

1. Implemented 16-bit training NMT: https://arxiv.org/abs/1806.00187
2. TRAINING WITH QUANTIZATION NOISE FOR EXTREME MODEL COMPRESSION:
   https://arxiv.org/pdf/2004.07320.pdf
3. AND THE BIT GOES DOWN: REVISITING THE QUANTIZATION OF NEURAL NETWORKS:
   https://arxiv.org/pdf/1907.05686.pdf
5. Deep Learning Performance Boost by Intel VNNI
   
https://www.intel.com/content/www/us/en/artificial-intelligence/posts/deep-learning-performance-boost-by-intel-vnni.html?fbclid=IwAR0SjK-y4fYkwqoRwCbrcoqOfcLBphYB4u_lEpSiG-8pgiXTsumV7tzpal4

6. DistilBERT: a distilled version of BERT: smaller, faster, cheaper and lighter:
   https://arxiv.org/pdf/1910.01108.pdf
7. MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices:
   https://arxiv.org/pdf/2004.02984.pdf
8. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks:
   https://arxiv.org/pdf/1803.03635.pdf
9. Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP:
   https://arxiv.org/pdf/1906.02768.pdf