A Roadmap Towards
Machine Intelligence
Some (Not So) Recent Papers by Tomáš Mikolov
Vítek Novotný, 2021-04-22
Tomáš Mikolov (* 1982 Šumperk)
● 2007: Language Modeling for Spech Recognition in Czech (Mgr. thesis, BUT)
● 2010: Stays at Johns Hopkins University (F. Jelínek) and Montreal (Bengio)
● 2010–2014: Google Brain, 2014–2020: Facebook AI Research
● 2020–: Czech Institute of Informatics, Robotics and Cybernetics (CIIRC CTU)
A Roadmap Towards Machine Intelligence (2018) I
1. Introduction
“Given the current availability of powerful hardware and large amounts of
machine-readable data, as well as the widespread interest in sophisticated
machine learning methods, the times should be ripe for the development of
intelligent machines.”
2. Desiderata for an intelligent machine
1. Ability to communicate
2. Ability to learn
A Roadmap Towards Machine Intelligence (2018) II
3. A simulated ecosystem to educate communication-based intelligent
machines
1. High-level description of the ecosystem
■ Agents: Teacher (and Reward) ⟷ Learner ⟷ Environment
■ Interface channels: Input and output bitsteams
■ Reward, Incremental structure, Time off, Evaluation
2. Early stages of the simulation
■ Preliminaries, Notation
(continued on the next slide)
The Learner learns to issue
Environment commands
Associating language to actions
Learning to
generalize I
Learning to
generalize II
Learning to
generalize III
Interactive
communication
■ Algorithmic knowledge
“Thus, we believe that successful construction of intelligent
machinescould automate computer programming, which will likely be
done in the future simply through communication in natural
language.”
Interacting with the
trained intelligent
machine
A Roadmap Towards Machine Intelligence (2018) X
4. Towards the development of intelligent machines
1. Types of learning
2. Long-term memory and compositional learning skills
3. Computational properties of intelligent machines
“Since there are many Turing-complete computational systems, one may
wonder which one should be preferred as the basis for machine
intelligence. We cannot answer this question yet, however we
hypothesize that the most natural choice would be a system that
performs computation in a parallel way, using elementary units that can
grow in number based on the task at hand. The growing property is
necessary to support the long-term memory, if we assume that the basic
units themselves are finite. An example of an existing computational
system with many of the desired properties is the cellular automaton of
Von Neumann et al. (1966). [...]”
Possible Computational Systems for the Intelligent Machine
● KRUSZEWSKI, Germán; MIKOLOV, Tomas. Combinatory Chemistry:
Towards a Simple Model of Emergent Evolution. In: Artificial Life Conference
Proceedings. One Rogers Street, Cambridge, MA 02142-1209 USA
journals-info@ mit. edu: MIT Press, 2020. p. 411-419. Available at DOI:
10.1162/isal_a_00258
● CISNEROS, Hugo; SIVIC, Josef; MIKOLOV, Tomas. Visualizing computation
in large-scale cellular automata. In: Artificial Life Conference Proceedings.
One Rogers Street, Cambridge, MA 02142-1209 USA journals-info@ mit.
edu: MIT Press, 2020. p. 239-247. Available at DOI: 10.1162/isal_a_00277
Combinatory Chemistry: Towards a Simple Model of
Emergent Evolution (2020)
● Introduces combinatory chemistry: an artificial algorithmic chemistry based
on combinatory logic (CL), a minimalistic computational system that was
independently invented by Schonfinkel, Von Neumann, and Haskell Curry.
● A system consists of CL expressions that react following reduction rules, plus
random condensation and cleavages.
● They apply a heuristic approach, which emulate the effects of having a much
larger system, and study the emergence of complex structures.
● They seek to use combinatory chemistry for explaining the emergence of
evolvability, one of the central questions in Artificial Life.
Visualizing Computation in Large-Scale Cellular Automata
● Large-scale cellular automata (CA) have high time and memory complexity.
● They propose several coarse-graining approaches for cellular automata, which
approximate the original cellular automata by merging cells with losing as few
information about the states as possible.
● This allows them to visualize large-scale CAs and see emergent structures.
“Viewing space-time diagrams of cellular automata is akin to visualizing a foreign
computer design. Cellular automata are manipulating information, registers and
instructions in parallel in the form of cell states. [...] Future work could focus on
identifying some known simple computational primitives within cellular automata
and understanding how our visualization can help to find them.”