HUMAN NEUROSCIENCE
PERSPECTIVE ARTICLE
published: 28 October 2011
doi: 10.3389/fnhum.2011.00118
What line drawings reveal about the visual brain
Bilge Sayim* and Patrick Cavanagh
Laboratoire Psychologie de la Perception, Centre Attention Vision, Université Paris Descartes, Paris, France
Edited by:
Luis M. Martinez, Universidade de A
Coruña, Spain
Reviewed by:
Micah M. Murray, Université de
Lausanne, Switzerland
Luis M. Martinez, Universidade de A
Coruña, Spain
*Correspondence:
Bilge Sayim, Laboratoire Psychologie
de la Perception, Université Paris
Descartes, 45 rue des Saints-Pères,
75006 Paris, France.
e-mail: bilge.sayim@parisdescartes.fr
Scenes in the real world carry large amounts of information about color, texture, shading,
illumination, and occlusion giving rise to our perception of a rich and detailed environment.
In contrast, line drawings have only a sparse subset of scene contours. Nevertheless, they
also trigger vivid three-dimensional impressions despite having no equivalent in the natural
world. Here, we ask why line drawings work.We see that they exploit the underlying neural
codes of vision and they also show that artists’ intuitions go well beyond the understanding
of vision found in current neurosciences and computer vision.
Keywords: visual perception, art, picture perception, painting, computer vision
Line drawings have fascinated artists and scientists in various
fields for many centuries with the first line drawings dating back
more than 30,000 years (Figure 1A). The ease and immediacy of
recognizing scenes and objects in simple line drawings suggests
that, for the visual system, line drawings have deep similarities
to other more detailed visual representations as well as to
the real scenes they depict. For example, line drawings of visual
scenes are recognized as fast and accurately as photographs (e.g.,
Biederman and Ju,1988). Sometimes,line drawings convey a stunningly
vivid impression of depth and three-dimensional shape,
even when not much more than the outlines of an object are
drawn.
Line drawings are so common in our everyday life that we seldom
ask why they work. Once we ask that question, however, we
realize that line drawings really are exceptional. In particular,in the
real world, there are no lines around objects (with rare exceptions;
Figure 1B). During the eons over which biological vision systems
evolved, there has been no experience that could have adapted our
visual systems to understand line drawings. Instead, objects are
usually segmented from the background by lightness, texture, or
color differences. So why does the visual system understand line
drawings?
We could imagine that line drawings are a convention of modern
art that we have come to recognize through learning as we
have the alphabet in which this paper is written (e.g., Gombrich,
1969; Goodman, 1976). This account has been controversial
(Kennedy, 1974, 1975; Deregowski, 1989; see also Gibson,
1971, 1979) and there is strong evidence against it. For example,
it has been shown that infants (Yonas and Arterberry, 1994;
see also Hochberg and Brooks, 1962), stone-age tribe members
(Kennedy and Ross, 1975), and even chimpanzees (Itakura, 1994;
Tanaka, 2007) are able to recognize line drawings. We even see
line representation used by insects in bio-mimicry (Figure 1C).
These findings rule out any strong version of culture-based acquisition
for understanding line drawings, although clearly there
are many culturally based conventions used in line drawings
(Figure 1D).
If cultural knowledge is not the key for understanding line
drawings and line drawings are too recent an arrival to have triggered
any special adaptation, what then is the mechanism that
allows us to make sense of these drawings? The likely explanation
is that lines trigger a neural response that has evolved to deal
with natural scenes. This fortuitous co-activation lets lines stand
in for solid edges. Once artists discovered this (Kennedy, 1975),
they quickly adopted this format as an economical and powerful
method for representing scenes and objects. How does this “coactivation”
work? The physiological investigation of the neural
response to contours began with by Hubel’s and Wiesel’s (1962,
1968) transformational discovery that neurons in the primary
visual cortex are tuned to the orientation of contours, responding
to edges, and not to uniform areas. The part of cortex that
analyzes visual information accounts for 30% or more of the cortex
in primates and is located at the posterior pole of the brain.
The visual cortex is further subdivided into several subregions that
process the incoming images along parallel and serial streams. The
first divisions of the visual cortex are labeledV1 throughV4 and,in
all of these,we see the orientation-tuned neurons that can respond
to edges. In areas V1 and V2, the orientation-tuned detectors can
be specific to the attribute defining the contour (color, contrast
polarity, or texture, etc.) but, starting in area V2 (Gegenfurtner
et al., 1996), through V5 (Albright, 1992), and on to object recognition
areas like IT (Sáry et al., 1995), many become indifferent
to the attribute that defines the contour. These orientation-tuned
units evolved to efficiently detect the contours in the natural world
(Olshausen and Field, 1996) but even though the edges in the
world are typically marked by a discrete change in surface attributes
– lighter on one side than the other, for example – these units
respond as well to lines – lighter in the middle and dark on both
sides, for example, or even illusory contours that are suggested
by context but not physically present (von der Heydt et al., 1984;
Lee and Nguyen, 2001; see also Seghier and Vuilleumier, 2006). In
other words, the receptive field structure that efficiently recovers
edges, also works well for lines even though it was not designed to
do so.
Frontiers in Human Neuroscience www.frontiersin.org October 2011 | Volume 5 | Article 118 | 1
Sayim and Cavanagh Line drawings and the brain
Consider then the cortical pattern of response to an object like
a cube (Figure 2). The contour-selective neurons with oriented
receptive fields fire only along the contour and not within the uniform
areas of the object. If we were to look at the visual cortex with
voltage sensitive dyes (Cohen et al., 1968; Tasaki et al., 1968; Blasdel
and Salama, 1986), the pattern of activity for the oriented units
would resemble a sketch of the object (Marr, 1982), distorted by
the cortical anatomy (Tootell et al., 1982). A set of lines that match
the cube’s edges would trigger responses in the same pattern, indicating
that, on a neural level, line representations are equivalent to
the originals they depict. This notion is supported by a number of
FIGURE 1 | (A) Early line drawing representation of a rhinoceros at
Chauvet, France, ca 30,000 BCE. (B) Outlines are infrequent effects of
backlighting in natural scenes but even in this case, internal contours are
never visible as lines. (C) Bio-mimicry used by the Fulgorida of South and
Central America where lines trace the contour between the simulated lips
and mouth. (D) Lines typically stand for discontinuities in surface depth or
orientation but some artists rely on cultural conventions to make them
stand for motion or energy (Keith Haring).
FIGURE 2 | Contour-selective neurons with oriented receptive fields
fire along the contours of the cubes. The analysis of both types of cubes
yields the internal sketch-like representation of the cube, which would, in
actuality, be distorted by the cortical magnification.
recent imaging studies that showed that the activation in response
to line drawings was similar to that for other representations (e.g.,
Ishai et al., 2000; Walther et al., 2011).
Is that all there is to it? No, in fact, there is quite a lot remaining
to explain as can be seen in any line version of a natural
scene. These are typically uninterpretable and the simple image
in Figure 3 shows why. Many of the contours in a scene arise from
accidental illumination edges at the borders of shadows and shading.
These contours, when represented as lines take on a reality
that they should not have. Each line in a standard line drawing
stands for depth or slant discontinuities between surfaces: these
are “object contours.” When the borders of shadows are included
in a line drawing, these contours also get promoted to the status
of object contour – but for locations where there were none. As a
result, the whole image is corrupted, deviating from the structure
of the original objects. Figure 4 shows this even more dramatically
because its original is a representation of a face that can only be
recognizedif theshadowsarecorrectlyprocessed.Renderedonlyas
contours,the light and dark polarity required to interpret shadows
is no longer available and the pattern becomes a meaningless set of
lines. So, while contours are, of course, of prominent importance
for visual perception (e.g., Koenderink, 1984), displaying all the
FIGURE 3 | (A) Original image with objects and shadows. (B) Extracted
contours from the original do not capture the underlying scene very well.
(C) A depth sketch revealing the two actual objects in the scene.
FIGURE 4 | (A) Two-tone image of Kennedy (1997) where the dark areas
could be dark pigment or dark shadow.The face can be recovered only if the
shadows are correctly identified. (B) A line version of the same contours is
unrecognizable. The polarity information required for finding shadows is
lost. The shadow boundaries can no longer be discounted and are taken as
object contours that form meaningless shapes unrelated to the original.
Frontiers in Human Neuroscience www.frontiersin.org October 2011 | Volume 5 | Article 118 | 2
Sayim and Cavanagh Line drawings and the brain
contours of a scene in a line drawing will regularly fail to convey
the essential parts of an image.
There is therefore a critical step between the extraction of
edges and their assignment as “object contours.” The visual system
understands how to determine which contours are the critical
ones and beyond a certain level in the hierarchy of visual cortex,
only those contours should remain. Shadow borders and other
accidental contours must be removed in order to keep only the
object contours. We do not know where or how this happens
in the visual system. No imaging or physiological study has yet
shown the absence of response to a shadow border at some level
of cortex, and yet this must happen. Like the visual system, artists
also understand which contours are the important ones. We see
only the characteristic object contours in their line drawings –
never shadow borders, no matter how prominent they are in the
scene they are drawing. Artists appear to have access to a body
FIGURE 5 | Junctions trigger local interpretations. The T-junctions along
the edge at the base of the middle cube imply that this is the edge of an
opaque surface that hides the bottom of the cube. However, the same
edge then makes X-junctions with the contours of the outer cubes,
indicating that it is a cord with no opaque surface on either side. These
junctions drive local interpretations that do not require consistency across
larger distances to be perceived (from Kennedy, 1974).
FIGURE 6 |Two very different line drawings triggering a familiar
prototype. Both sketches are perceived as faces.
of knowledge – what makes a characteristic contour – that scientists
only dimly understand at present. Future studies of artists’
intuitive understanding of critical contours may lead to important
insights for image understanding.
Following the initial critical choice of lines to include, what
are the elements that are central to the information conveyed by
line drawings? Particularly informative are those parts of an image
where contours touch or intersect (Clowes, 1971; Huffman, 1971;
Albert and Hoffman, 1995) and many authors have shown how
these various junctions form a set of constraints that are often
sufficient to specify the original object (c.f., Barrow and Tenenbaum,
1981; Malik, 1987). For example, a T-junction is formed
when one object interrupts the contours of another object behind
it (the contours meet in a junction as in the letter T); a Y-junction
is seen at the front corner of a cube; an X-junction is formed
when the contours of a transparent material cross those of a background
surface. Contours also may end on their own when smooth
surfaces self-occlude, such as the top of a torus or donut (Koenderink,
1984). These local junction cues are clearly used by the
visual system to make sense of line drawings and we can see their
power when they are in conflict (Figure 5) as the impossibility of
the global shape does not suppress the local interpretations they
trigger – a loophole exploited to great effect by artists like Escher
and Reutersvard and scientists like Penrose. Interestingly, the way
junctions are used in drawings has not changed very much over the
recorded history of art (Biederman and Kim,2008,see T-junctions
where the rhinoceros’s legs meet its body in Figure 1A), suggesting
again that they are informative aspects of the world and not
creations of our culture.
However,while junctions are certainly informative,they are not
necessary for recognizing line drawings. The recognition of many
sketches reveals an important contribution of memory. When a
set of contours matches a familiar prototype, memory serves to
fill in the missing details (Figure 6). These and many other line
drawings show how artists are able to depict such various features
as depth, folds, occlusion, texture, brightness and even odor, mental
energy, or motion by choosing the right lines, revealing that
artists (implicitly or explicitly) understand the code of the visual
system. Scientists have yet to fully understand what artists have
successfully been practicing for thousands of years and for some
questions of the neural codes of vision, we may find that artists
are a more immediate and better source of information than our
most advanced scientific studies, whether of behavior, single cell
recordings, or brain imaging.
ACKNOWLEDGMENTS
This work was supported by a Chaire d’Excellence Grant from the
ANR.
REFERENCES
Albert, M. K., and Hoffman, D. D.
(1995).“Genericity in spatial vision,”
in Geometric Representations of Perceptual
Phenomena: Papers in Honor
of Tarow Indow on His 70th Birthday,
eds D. Hoffman, R. D. Luce,
M. D’Zmura, G. Iverson, and A.
K. Romney (New York: Lawrence
Erlbaum), 95–112.
Albright, T. D. (1992). Form-cue invariant
motion processing in primate
visual cortex. Science 255,
1141–1143.
Barrow, H. G., and Tenenbaum, J. M.
(1981). Interpreting line drawings
as three dimensional surfaces. Artif.
Intell. 17, 75–117.
Biederman, I., and Ju, G. (1988). Surface
versus edge-based determinants
of visual recognition. Cogn. Psychol.
20, 38–64.
Biederman, I., and Kim, J. G. (2008).
17000yearsof depictingthejunction
of two smooth shapes. Perception 37,
161–164.
Blasdel, G. G., and Salama, G. (1986).
Voltage-sensitive dyes reveal a modular
organization in monkey striate
cortex. Nature 321, 579–585.
Clowes, M. B. (1971). On seeing things.
Artif. Intell. 2, 79–116.
Cohen, L. B., Keynes, R. D., and Hille, B.
(1968).Lightscatteringandbirefringence
changes during nerve activity.
Nature 218, 438–441.
Deregowski, J. B. (1989). Real space and
represented space: cross-cultural
perspectives. Behav. Brain Sci. 12,
51–119.
Frontiers in Human Neuroscience www.frontiersin.org October 2011 | Volume 5 | Article 118 | 3
Sayim and Cavanagh Line drawings and the brain
Gegenfurtner, K. R., Kiper, D. C., and
Fenstemaker, S. B. (1996). Processing
of color, form, and motion in
macaque area V2. Vis. Neurosci. 13,
161–172.
Gibson, J. J. (1971). The information
available in pictures. Leonardo 4,
27–35.
Gibson, J. J. (1979). The Ecological
Approach to Visual Perception.
Boston: Houghton Mifflin.
Gombrich, E. H. (1969). Art and Illusion.
A Study in the Psychology of
Pictorial Representation. Princeton,
NJ: Bollingen Series/Princeton University
Press.
Goodman, N. (1976). Languages of Art:
An Approach to a Theory of Symbols,
2nd Edn. Indianapolis: Hackett
Publishing Company.
Hochberg,J.,and Brooks,V. (1962). Pictorial
recognition as an unlearned
ability: a study of one child’s
performance. Am. J. Psychol. 75,
624–628.
Hubel, D. H., and Wiesel, T. N.
(1962). Receptive Felds and functional
architecture in the cat’s visual
cortex. J. Neurosci. 160, 106–154.
Hubel, D. H., and Wiesel, T. N.
(1968). Receptive Felds and functional
architecture of monkey striate
cortex. J. Neurosci. 195, 215–243.
Huffman, D. A. (1971). “Impossible
objects as nonsense sentences,”
in Machine Intelligence 6,
eds B. Melzer and D. Michie
(Edinburgh: Edinburgh University
Press), 295–323.
Ishai, A., Ungerleider, L. G., Martin, A.,
and Haxby, J. V. (2000). The representation
of objects in the human
occipital and temporal cortex. J.
Cogn. Neurosci. 12, 35–51.
Itakura, S. (1994). Recognition of linedrawings
representations by a chimpanzee
(Pan troglodytes). J. Gen. Psychol.
121, 189–197.
Kennedy, J. M. (1974). A Psychology
of Picture Perception. San Francisco:
Jossey-Bass.
Kennedy, J. M. (1975). Drawings were
discovered, not invented. New Sci.
67, 523–527.
Kennedy, J. M. (1997). How the blind
draw. Sci. Am. 276, 60–65.
Kennedy, J. M., and Ross, A. S.
(1975). Outline picture perception
by the Songe of Papua. Perception 4,
391–406.
Koenderink, J. J. (1984). What does the
occluding contour tell us about solid
shape? Perception 13, 321–330.
Lee, T. S., and Nguyen, M. (2001).
Dynamics of subjective contour formation
in the early visual cortex.
Proc. Natl. Acad. Sci. U.S.A. 98,
1907–1911.
Malik, J. (1987). Interpreting line drawings
of curved objects. Int. J. Comput.
Vis. 1, 73–103.
Marr, D. (1982). Vision. A Computational
Investigation Into the Human
Representation and Processing of
Visual Information. San Francisco:
W. H. Freeman and Company.
Olshausen, B. A., and Field, D. J. (1996).
Emergence of simple cell receptive
field properties by learning a sparse
code for natural images. Nature 381,
606–607.
Sáry, G., Vogels, R., Kovács, G., and
Orban, G. A. (1995). Responses
of monkey inferior temporal neurons
to luminance-, motion-, and
texture-defined gratings. J. Neurophysiol.
73, 1341–1354.
Seghier, M. L., and Vuilleumier, P.
(2006). Functional neuroimaging
findings on the human perception
of illusory contours. Neurosci. Biobehav.
Rev. 30, 595–612.
Tanaka, M. (2007). Recognition of
pictorial representations by chimpanzees
(Pan troglodytes). Anim.
Cogn. 10, 169–179.
Tasaki, I., Watanabe, A., Sandlin, R.,
and Carnay, L. (1968). Changes in
fluorescence, turbidity, and birefringence
associated with nerve excitation.
Proc. Natl. Acad. Sci. U.S.A. 61,
883–888.
Tootell, R. B. H., Silverman, M. S.,
Switkes, E., and De Valois, R.
S. (1982). Deoxyglucose analysis
of retinotpic organization in primate
striate cortex. Science 220,
737–739.
von der Heydt, R., Peterhans, E.,
and Baumgartner, G. (1984).
Illusory contours and cortical
neuron responses. Science 224,
1260–1262.
Walther, D. B., Chai, B., Caddigan,
E., Beck, D. M., and Fei-Fei, L.
(2011). Simple line drawings suffice
for functional MRI decoding
of natural scene categories.
Proc. Natl. Acad. Sci. U.S.A. 108,
9661–9666.
Yonas, A., and Arterberry, M. E. (1994).
Infants perceive spatial structure
specified by line junctions. Perception
23, 1427–1435.
Conflict of Interest Statement: The
authors declare that the research was
conducted in the absence of any
commercial or financial relationships
that could be construed as a potential
conflict of interest.
Received: 16 June 2011; accepted: 30 September
2011; published online: 28 October
2011.
Citation: Sayim B and Cavanagh P
(2011) What line drawings reveal about
the visual brain. Front. Hum. Neurosci.
5:118. doi: 10.3389/fnhum.2011.00118
Copyright © 2011 Sayim and Cavanagh.
This is an open-access article subject
to a non-exclusive license between the
authors and Frontiers Media SA, which
permits use, distribution and reproduc-
tioninotherforums,providedtheoriginal
authors and source are credited and other
Frontiers conditions are complied with.
Frontiers in Human Neuroscience www.frontiersin.org October 2011 | Volume 5 | Article 118 | 4