Classroom Response and Communication Systems:
Research Review and Theory
Jeremy Roschelle, SRI International
William R. Penuel, SRI International
Louis Abrahamson, Better Education, Inc.
Paper to be presented at the Annual Meeting of the American Educational Research
Association, San Diego, CA, April 2004.
In How People Learn, Bransford and colleagues (National Research Council, 1999) cite
classroom response system technology and the related pedagogy as one of the most
promising innovations for transforming classrooms to be more learner-, knowledge-,
assessment-, and community-centered. As a step towards guiding practice and advancing
research, we present our review of the research on this and more advanced, but related
technologies, particularly with regard to the popular use of these systems to enhance
questioning and feedback. We also formulate tentative theoretical connections to a
broader scientific literature that could explain how pedagogy and technology together
realize multiple desirable outcomes
Classroom Response and Communication Systems: Research Review and Theory
Interest is growing in technologies that generalize and extend earlier “response systems”
and “classroom communication systems,” and enable teachers to improve instruction in
classroom- or lecture hall-sized groups. Researchers report that instructors use the novel
technological capabilities to enhance questioning and feedback, to motivate and monitor
the participation of all students, to foster discussions of important concepts, and to
energize and activate students’ thinking. In their foundational summary of cognitive
science and education, How People Learn, as well as in a subsequent paper, Bransford
and colleagues cite this technology and the related pedagogy as one of the most
promising for transforming classrooms to be more learner-, knowledge-, assessment-, and
community-centered (Bransford, Brophy, & Williams, 2000; National Research Council,
1999). But as yet, research has taking place by researchers in different sub-communities,
with little cross-fertilization and synthesis. As a step towards advancing future research,
the first half of this paper presents our review of the research literature, particularly with
regard to the popular use of these systems to enhance questioning and feedback. In the
second half, we formulate tentative theoretical connections that could explain how
pedagogy and technology together realize multiple desirable outcomes (see Figure 1).
CATAALYST Review and Theory 1 AERA 2004 Paper Proposal
Figure 1: CATAALYST Pedagogy and Technology Integrate to Enhance Learning
CATAALYST
We have dubbed the mature form of this technology CATAALSYT, standing for
“Classroom Aggregation Technology for Activating and Assessing Learning and Your
Students’ Thinking.” It represents an expansion of earlier “response systems” technology
to include an extensible set of subject-matter-specific representations.
The core capabilities of this technology allow the teacher to more efficiently and
effectively:
1. Present a probing question at the heart of the subject matter.
2. Gather student responses rapidly and anonymously.
3. Quickly assemble a public, aggregate display (e.g. a histogram) that makes salient the
variation in the group’s ideas without disclosing individual contributions.
Review of the literature
Our goal was to understand what aspects of CATAALYST pedagogy might be ready for
research on questions of systematic effectiveness. We found one well-developed strand
that focuses on teacher questioning and feedback. Other, more advanced strands of
theory, technology, and pedagogy are in active development (e.g., Hegedus & Kaput,
2003; Stroup et al., 2002) and could be synergistically integrated later. Our review
organizes the existing literature on the “questioning” strand according to three conditions:
1. Success in addressing an important problem.
CATAALYST Review and Theory 2 AERA 2004 Paper Proposal
2. Converging evidence from a variety of settings.
3. A theory adequate to guide replication and generalization.
With regard to condition 1, we found ample converging evidence that supports a
relationship between CATAALYST and student outcomes, particularly in physics, where
a well-calibrated measure of conceptual change is available. Conceptual change is known
to be an important and difficult classroom outcome to achieve (Confrey, 1990;
McDermott, 1984; Smith, diSessa, & Roschelle, 1993). Mazur shows 10 years of
continuous improvement in the pretest/posttest gains by successive classes of students on
the Force Concept Inventory using his Peer Instruction pedagogy (Crouch & Mazur,
2001; Fagen, Crouch, & Mazur, 2002; Mazur, 1997). This pedagogy includes attention to
motivation and adaptation of the lesson plan in response to formative assessment. At its
heart is a practice of focused teacher questioning, aggregation of responses, and use of a
public display of those responses to foster high-level reasoning among learners.
Investigators have also documented a variety of improvements in student attitudes and
classroom participation using related pedagogies with response systems (Dufresne,
Gerace, Leonard, Mestre, & Wenk, 1996).
With regard to condition 2, we found that consistent findings have been reported in a
wide variety of settings beyond undergraduate physics. University settings include
mathematics, physics, chemistry, biology, premedical education, business, and computer
science. K-12 settings include middle school and high school mathematics, physics and
chemistry, as well as elementary, middle school, and high school reading. A total of 26
studies report outcomes (listed in full paper). The most commonly reported outcomes are:
greater student engagement (16 studies), increased understanding of subject matter (11),
increased enjoyment of class (7), better group interaction (6), helping students gauge their
own understanding (5), and teachers have better awareness of student difficulties (4).
This body of evidence, taken together, is suggestive of a real and important phenomenon
at hand. However, none of the available studies rises to the present specification of
“scientifically based research” that would allow inferences about causal relationships or
that could form the basis for estimating the magnitude of the effect.
With regard to condition 3, our review found that existing research does not connect with
the larger research base in education or psychology, which could be used to create an
explanatory theory or model. We turn to this limitation in the next section.
Connections to a Broader Educational and Psychological Literature
We have developed three connections to prior bodies of theory and evidence, selecting
for inclusion bodies of work that (a) deeply relate to reported CATAALYST
phenomenology, (b) have been associated with strong effects on student outcomes in the
literature, and (c) can provide guidance to further CATAALYST research.
1. Formative Assessment through Questioning and Feedback
Classrooms proceed through a sequential curricular structure. In most cases, the transition
from topic to topic occurs merely because time has elapsed, without feedback to the
CATAALYST Review and Theory 3 AERA 2004 Paper Proposal
teacher that the current topic has been mastered. Students receive little feedback before
the examination. Against this background, one of the classroom interventions known to
have the greatest impact on student outcomes is formative assessment (Black & Wiliam,
1998; Fuchs & Fuchs, 1986). When questioning and feedback is frequent and involves
students actively in reflecting on what they know and how they learn, and when
assessment data are used to inform and adjust the course of instruction, formative
assessment can produce large gains (Black & Wiliam, 1998).
Formative assessment deserves to be the base of our theoretical account because the most
obvious thing that CATAALYST does is to increase the ease with which teachers can
engage all students in frequent formative assessment. Connecting CATAALYST to
formative assessment makes a large literature on the quality of questioning and nature of
feedback available to guide practice (Brady, Taylor, & Hamilton, 1989; Dillon, 1984;
Gall, 1984; Lemke, 1990; Redfield & Rousseau, 1981; Samson, Strykowski, Weinstein,
& Walberg, 1987).
2. Building Concepts through Contrasts and Student Discussion
The formative assessment literature does not tell us why CATAALYST innovators use
particular representational forms to aggregate student responses— e.g., a histogram in
early uses and, more recently, overlaid images or Cartesian coordinate systems. We
propose understanding what kinds of representations work best for a questioning-oriented
pedagogy in terms of two kinds of important contrasts:
1. Contrasts among ideas that are strongly related to the target concept.
2. Contrasts between an individual student’s idea and the ideas of the group.
In both cases, we use “contrast” to mean both similarities that have a positive relationship
to the target concept and differences that have a negative relationship to the target
concept.
The first kind of contrast (among instances in a domain) was studied extensively in early
cognitive psychology, which showed that learners can induce a novel category rapidly
(e.g., “blue squares”), when given a well-chosen sequence of cases (Ross, 2000; Smith,
1989). However, although practitioners use representations that highlight important
contrasts, they do not display enough instances for induction to be the main mechanism at
work. Well-selected contrasting cases, however, can help guide interpretation and create
a basis of readiness for future learning. Experimental results show that contrasting cases
prepare students to learn more from subsequent elaborations (Bransford & Schwartz,
1999).
In the case of CATAALYST, the second kind of contrast, in which each student sees his
or her contribution in the context of the group aggregate is also prominent. We conjecture
that this second contrast complements the first and results in better self-regulation—
students are more able to engage with and make sense of feedback when it is feedback
when they have thought about first and when they have a personal stake in the outcome
(Butler & Winne, 1995). Indeed, engaging with cognitive conflict (or making prior
CATAALYST Review and Theory 4 AERA 2004 Paper Proposal
knowledge visibly problematic) is an important prerequisite to conceptual change
(Posner, Strike, Hewson, & Gertzog, 1982; Webb & Palincsar, 1996).
It could be that contrasts are sufficient to enhance achievement, but reports from the use
of CATAALYST suggest that discussion (which may be among peers, small groups, or
the whole class) emphasizing argumentation, elaboration, explanation, and
comprehension is another essential element of the pedagogy. We conjecture this is
necessary because the target concepts in science and mathematics are not as simple as
“blue square”—learners must work out the relationships of a big idea to many supporting
knowledge elements (Ausubel, 1965; diSessa, 1993; Minstrell, 1992). In learning about
kinematics, for example, conversation can lead to “convergent conceptual change,” in
which students guide each other, seeking a suitable differentiation and integration among
the many ideas that comprise knowledge of the central idea (Roschelle, 1992; Webb &
Palincsar, 1996). Further, research has demonstrated that appropriate technology-based
representations can enhance both the quality of students’ discussion and what they learn
(Cohen & Scardamalia, 1998; Hsi, 1997; Pea, 1994; Ryser, Beeler, & McKenzie, 1995;
Suthers & Hundhausen, 2003). Tools that support manipulating dynamic notations and
models are particularly important in mathematics (Kaput, 1992) and science (Penner,
2001). These tools were not available in simple response systems, but they are available
in CATAALYST. We see CATAALYST as catalyzing the base conditions known to be
necessary for successful group learning of mathematics and science.
3. A Shift to Mastery-Oriented Motivational Incentives
The first two elements of our theory seek to explain why CATAALYST improves
achievement, but do less to explain why CATAALYST creates greater engagement and
broader participation. Motivational “goal theories” focus on analyzing students’ reasons
for engaging in achievement activities (Pintrich & Schrauben, 1992). Students pursue two
major kinds of goals: “performance” goals, in which learners seek to maintain positive
judgments of their ability and avoid negative judgments, and learning or “mastery” goals,
in which students seek to increase their ability or master new tasks (Elliott & Dweck,
1988). Across a range of studies, researchers have found a consistent pattern: the
adoption of learning goals tends to help students of all ability levels sustain engagement
in a task, but the adoption of performance goals for low-ability students tends to lead
them to avoid challenging tasks and attribute failure to poor ability rather than to their
level of effort (Elliott & Dweck, 1988). Students’ individual goal orientations, moreover,
can be shaped by powerful educational interventions like CATAALYST, especially in
cases where students perceive classroom goal structures to be more learning or mastery
focused (Ames & Archer, 1988; Griswold & Urdan, 2001; Maehr & Midgley, 1991).
We conjecture that CATAALYST facilitates students’ adoption of learning or mastery
goals by highlighting the differences among ideas while at the same time downplaying
the individual ownership of those ideas. CATAALYST displays can reduce what
motivation theorists call “performance avoidance” goals, the desire to avoid the
embarrassment of a poor performance. Second, CATAALYST questions tend to be used
in a formative rather than summative fashion: students are expected to use the questions
as challenging prompts for thinking rather than as opportunities to display competence.
CATAALYST Review and Theory 5 AERA 2004 Paper Proposal
Finally, CATAALYST creates accountability for all students—the teacher knows
whether all students contributed and can see what each individual student contributed, if
she likes. Thus, the powerful push of individual accountability is combined with the
powerful pull of a focus on the quality of ideas, not the status of individual students
(Davis, 2002).
Educational and Scientific Importance
Presentation of this review at AERA could be educationally important because the main
effects of CATAALYST appear to address important educational concerns, such as
enhancing student achievement, interest, and participation, and the converging evidence
looks promising. Further, the underlying technology is becoming increasingly mature:
robust, powerful, and affordable. We believe our review is scientifically important
because we have connected the findings about CATAALYST coming from practitioners
to a broader scientific and education literature, thus supporting additional research going
forward.
References
Ames, C., & Archer, J. (1988). Achievement goals in the classroom: Students' learning
strategies and motivation processes. Journal of Educational Psychology, 80(3), 260-
267.
Ausubel, D. P. (1965). A cognitive structure view of word and concept meaning. In R. C.
Anderson & D. P. Ausubel (Eds.), Readings in the psychology of cognition (pp. 103-
115). New York: Holt, Rinehart and Winston, Inc.
Black, P., & Wiliam, D. (1998). Inside the black box: Raising standards through
classroom assessment. London: King's College London.
Brady, M. P., Taylor, R. D., & Hamilton, R. (1989). Differential measures of teachers'
qestioning in mainstreamed classes: Individual and classwide patterns. Journal of
Research and Development in Education, 23(1), 10-17.
Bransford, J., Brophy, S., & Williams, S. (2000). When computer technologies meet the
learning sciences: Issues and opportunities. Journal of Applied Developmental
Psychology, 21(1), 59-84.
Bransford, J. D., & Schwartz, D. L. (1999). Rethinking transfer: A simple proposal with
multiple implications. In A. Iran-Nejad & P. D. Pearson (Eds.), Review of research
in education (Vol. 24, pp. 61-100). Washington, DC: American Educational
Research Association.
Butler, D. L., & Winne, P. H. (1995). Feedback and self-regulated learning: A theoretical
synthesis. Review of Educational Research, 65, 245-281.
Cohen, A. L., & Scardamalia, M. (1998). Discourse about ideas: Monitoring and
regulation in face-to-face and computer-mediated learning environments. Interactive
Learning Environments, 6(1-2), 114-142.
Confrey, J. (1990). A review of the research on student conceptions in mathematics,
science, and programming. In C. B. Cazden (Ed.), Review of research in education
(Vol. 16, pp. 3-55). Washington, DC: American Educational Research Association.
CATAALYST Review and Theory 6 AERA 2004 Paper Proposal
Crouch, C. H., & Mazur, E. (2001). Peer instruction: Ten years of experience and results.
The Physics Teacher, 69(9), 970-977.
Davis, S. (2002). Research to industry: Four years of observations in classrooms Using a
network of handheld devices. Paper presented at the IEEE International Workshop
on Wireless and Mobile Technologies in Education, Växjö, Sweden.
Dillon, J. T. (1984). Research on questioning and discussion. Educational Leadership,
42(3), 50-56.
diSessa, A. A. (1993). Towards an epistemology of physics. Cognition and Instruction,
10(2&3), 105-225.
Dufresne, R. J., Gerace, W. J., Leonard, W. J., Mestre, J. P., & Wenk, L. (1996).
Classtalk: A classroom communication system for active learning. Journal of
Computing in Higher Education, 7, 3-47.
Elliott, E. S., & Dweck, C. S. (1988). Goals: An approach to motivation and
achievement. Journal of Personality and Social Psychology, 54(1), 5-12.
Fagen, A. P., Crouch, C. H., & Mazur, E. (2002). Peer instruction: Results from a range
of classrooms. The Physics Teacher, 40(4), 206-207.
Fuchs, L. S., & Fuchs, D. (1986). Effects of systematic formative evaluation: A metaanalysis.
Exceptional Children, 53(3), 199-208.
Gall, M. (1984). Synthesis of research on teachers' questioning. Educational Leadership,
42(3), 40-47.
Griswold, E., & Urdan, T. C. (2001). Achievement goals and classroom motivation:
Differences in personal motivational variables. Paper presented at AERA, Seattle,
WA.
Hegedus, S., & Kaput, J. (2003, July). The effect of a SimCalc connected classroom on
students' algebraic thinking. Paper presented at the Psychology in Mathematics
Education conference, Honolulu, HI.
Hsi, S. (1997). Facilitating knowledge integration in science through electronic
discussion: the Multimedia Forum Kiosk. Unpublished dissertation. Berkeley:
University of California.
Kaput, J. (1992). Technology and mathematics education. In D. Grouws (Ed.), A
handbook of research on mathematics teaching and learning (pp. 515-556). New
York: Macmillan.
Lemke, J. L. (1990). Talking science: Language, learning, and values. Norwood, NJ:
Ablex.
Maehr, M. L., & Midgley, C. (1991). Enhancing student motivation: A school-wide
approach. Educational Psychologist, 26, 399-427.
Mazur, E. (1997). Peer Instruction: A user's manual. Englewood Cliffs, NJ: Prentice
Hall.
McDermott, L. C. (1984). Research on conceptual understanding in mechanics. Physics
Today, 37, 24-32.
Minstrell, J. (1992). Facets of students' thinking. Paper presented at the International
Workshop: Research in Physics Learning - Theoretical Issues and Empirical Studies,
Kiel, Germany.
National Research Council. (1999). How people learn: Brain, mind, experience and
school. Washington, DC: National Academy Press.
CATAALYST Review and Theory 7 AERA 2004 Paper Proposal
CATAALYST Review and Theory 8 AERA 2004 Paper Proposal
Pea, R. D. (1994). Seeing what we build together: Distributed multimedia learning
environments for transformative communications. Journal of the Learning Sciences,
3(3), 283-298.
Penner, D. E. (2001). Cognition, Computers, and Synthetic Science: Building knowledge
and meaning through modeling. In W. G. Secada (Ed.), Review of Research in
Education (Vol. 25, pp. 1-35). Washington, DC: AERA.
Pintrich, P. R., & Schrauben, B. (1992). Students' motivational beliefs and their cognitive
engagement in classroom academic tasks. In J. Meece (Ed.), Student perceptions in
the classroom: Causes and consequences (pp. 149-183). Hillsdale, NJ: Erlbaum.
Posner, G. J., Strike, K. A., Hewson, P. W., & Gertzog, W. A. (1982). Accommodation
of a scientific conception: Toward a theory of conceptual change. Science
Education, 66(2), 211-227.
Redfield, D. L., & Rousseau, E. W. (1981). A meta-analysis of experimental research on
teacher questioning behavior. Review of Educational Research, 51(2), 237-245.
Roschelle, J. (1992). Learning by collaborating: Convergent conceptual change. The
Journal of the Learning Sciences, 2(3), 235-276.
Ross, B. H. (2000). Concepts: Learning. In A. E. Kazdin (Ed.), Encyclopedia of
psychology (Vol. 2, pp. 248-250). Oxford, UK: Oxford University Press.
Ryser, G. R., Beeler, J. E., & McKenzie, C. M. (1995). Effects of a computer-supported
intentional learning environment (CSILE) on students' self-concept, self-regulatory
behavior, and critical thinking ability. Journal of Educational Computing Research,
13(4), 375-385.
Samson, G. E., Strykowski, B., Weinstein, T., & Walberg, H. J. (1987). The effects of
teacher questioning levels on student achievement: A quantitative synthesis. Journal
of Educational Research, 80(5), 290-295.
Smith, E. E. (1989). Concepts and Induction. In M. I. Posner (Ed.), Foundations of
Cognitive Science (pp. 501-526). Cambridge, MA: MIT Press.
Smith, J. P., diSessa, A. A., & Roschelle, J. (1993). Misconceptions reconceived: A
constructivist analysis of knowledge in transition. Journal of the Learning Sciences,
3(2), 115-163.
Stroup, W. M., Kaput, J., Ares, N., Wilensky, U., Hegedus, S. J., Roschelle, J., Mack, A.,
Davis, S., & Hurford, A. (2002). The nature and future of classroom connectivity:
The dialectics of mathematics in the social space. Paper presented at the Psychology
and Mathematics Education North America conference, Athens, GA.
Suthers, D. D., & Hundhausen, C. D. (2003). An experimental study of the effects of
representational guidance on collaborative learning processes. Journal of the
Learning Sciences, 12(2), 183-219.
Webb, N. M., & Palincsar, A. S. (1996). Group processes in the classroom. In D. C.
Berliner & R. C. Calfee (Eds.), Handbook of educational psychology (pp. 841-873).
New York: Macmillan.