Types of Knowledge of Education -- 1 Types of Knowledge of Education Created through Disciplined Inquiry Theodore Frick Instructional Systems Technology Department School of Education Indiana University Bloomington September 4, 2004 Types of Knowledge of Education -- 2 Abstract Disciplined inquiry in educationshould result in knowledge about education. Types of knowledge about education should not be conflated with inquiry methods commonly characterized as qualitative and quantitative. Inquiry methods, if disciplined, result in knowledge. All knowledge claims are not of the same kind or value. Axiology is a branch of philosophy which concerns judgments about value. In general, values are viewed as instrumental or intrinsic. Instrumental values characterize what something is `good for'. Intrinsic values describe inherent worth, what is `good in itself'. Thus, axiological knowledge of education can be characterized respectively as praxiological or philosophical. Furthermore, knowledge of education which is not axiological is scientific. Scientific knowledge of education characterizes `what is'. Scientific knowledge has epistemic value and characterizes matters of truth, not goodness. Moreover, the scope of knowledge claims can range from that which represents unique or delimited states of affairs (i.e., situated knowledge) to that which is generalizable across time and space (i.e., theoretical knowledge). When crossing these two dimensions, a six-fold typology is created: 1. Situated scientific knowledge of education 2. Theoretical scientific knowledge of education 3. Situated praxiological knowledge of education 4. Theoretical praxiological knowledge of education 5. Situated philosophic knowledge of education 6. Theoretical philosophic knowledge of education. Types of Knowledge of Education -- 3 Each type of knowledge has value in its own right. Different considerations obtain for inquiry methodologies and relevant criteria for creating and extending each knowledge type. For example, empirical data are relevant for justification of Types 1 to 4, whereas Types 5 and 6 require rational argument based on moral or ethical principles and their justification. Finally, it is important to see relationships among the knowledge types so as to not get trapped into an overly narrow perspective regarding research methods and to be careful not to use one knowledge type to draw unwarranted conclusions about another. Overview Research is disciplined inquiry. Disciplined inquiry is the rational way to settle doubt and so to fixate belief (C.S. Peirce, 1958). Disciplined inquiry contrasts with the methods of tenacity, authority and agreeableness to reason. The major purpose for doing disciplined inquiry is to create or extend knowledge. The outcome of research is knowledge. Inquiry methods are a means to that end. We should always begin inquiry by identifying the kind of knowledge we hope to create and the questions we want to answer. Then we should choose whatever inquiry method or methods that will help answer those questions. Hammers Versus Screwdrivers When We Really Need a Saw Much of the debate over qualitative versus quantitative research methods in education might dissipate if distinctions are made between methods of research and outcomes of research (Frick & Reigeluth, 1992). Imagine for a moment three carpenters arguing about which tool is best. Types of Knowledge of Education -- 4 Quanta: "Hammers and nails are clearly superior." Qualia: "I disagree. Screwdrivers and screws are much more effective." Performa: "You're both wrong. Saws are best for cutting wood." Quanta: "Who said anything about cutting wood? I thought we were talking about fastening wood together." Qualia: "Right. Who needs saws?" Performa: "I do. I need to cut this board in half." Clearly, Quanta and Qualia have a different outcome in mind than Performa. From this perspective, their debate is misguided. Yet this debate parallels that among educational research methodologists who fail to distinguish between outcomes of research when comparing methods. We engage in disciplined inquiry to create knowledge (Steiner, 1988; Peirce, 1958). Not all knowledge is the same. The kind of knowledge about education that we create through disciplined inquiry determines what research methods are appropriate and useful. Purpose of Disciplined Inquiry The purpose or intent of a research investigation determines the kind of knowledge that is created. There is a distinction between describing "what is" and "what is of value". Steiner (1988) describes the difference as follows: (S)cience and praxiology differ as to the content they add to knowledge. Science does not add any axiologicalcontent to knowledge as philosophy and praxiology do. Yet the axiological content of praxiology differs from that of philosophy. Praxiology treats of instrumental value, while philosophy treats of intrinsic value. In other words, praxiology treats of effectiveness, while philosophy treats of worthwhileness. To treat of effectiveness is to treat of what means are effective Types of Knowledge of Education -- 5 with respect to a given end or ends. Effectiveness, of course, can be established by sensory observation, but worthwhileness cannot. (p. 25) Steiner (1988) considers science, praxiology and philosophy to be kinds of theoretical knowledge, unbounded by time and space. Scope of Knowledge Created through Disciplined Inquiry There is also a distinction between the particular and the general: between what is unique in contrast to what is common to many instances. This distinction has to do with the scope of knowledge created by disciplined inquiry. If the scope of knowledge is descriptive of a unique case (educational situation, program, product, design), then generalization beyond that case is not warranted. Such knowledge is situated. If the scope of knowledge is unbounded by time and space, then it is theoretical. Cronbach, Gleser, Nanda and Rajaratnam (1972) referred to different universes of generalization in the context of dependability of measures, which became known as generalizability theory (Shavelson & Webb, 1991). Here we are discussing the scope of knowledge claims, not scores or profiles. However, the concept is similar. For example, the scope of the claim, "Socrates is mortal," is the particular human, Socrates. Alternatively, the scope of the claim, "All humans are mortal," is unbounded by time and space. To observe the death of Socrates does not by itself warrant the broader generalization. Different kinds of verification are required for situated and theoretical knowledge claims. I believe that science (`what is'), praxiology (`what is instrumental'), and philosophy (`what ought to be') can also be situated ­ i.e., bounded by time and space, and deal with particulars. Furthermore, the typology that follows also differs from Steiner's notions of qualitative and performative knowledge. Types of Knowledge of Education -- 6 A Six-Fold Typology of Knowledge of Education When crossing these two dimensions ­ the purpose of disciplined inquiry and the scope of the knowledge created ­ there are six types of research outcomes, as illustrated in the table below. Table 1. Six types of knowledge of education Scope of Knowledge of Education Created through Disciplined Inquiry Purpose of Inquiry Situated Theoretical Scientific: `What is?' 1 2 Praxiological: `What is instrumental?' 3 4 Philosophical: `What ought to be?' 5 6 Type 1 and 2 knowledge represent matters of epistemic value ­ i.e., truth. Correspondence between the descriptions and what is observed is important. Type 3 and 4 knowledge represent matters of utility ­ i.e., what something is `good for'. What is important is how well something works ­ either a particular product or program, or a set of generalizable principles, heuristics or guidelines for effective education. Type 5 and 6 knowledge represent matters of intrinsic value ­ i.e., description and evaluation of states of affairs in education that ought to be. Standards, norms or criteria are important in evaluation of specific situations as well as philosophic theory. Types of Knowledge of Education -- 7 Scientific and praxiological knowledge claims rely on empirical justification ­ by comparing knowledge claims with what we observe through our senses, or through measuring devices or instruments of observation which extend our senses. For example, if we want to know if it is raining outside at the moment, or if we want to know if precipitation is associated with the presence of clouds in the sky, we can verify empirically whether or not these claims agree with our observations. Empirical justification does not necessarily imply experimental methods. Naturalistic observation of existing states of affairs in the world is empirical. Experimental methods are empirical as well, but states of affairs are instead manipulated in order to make causal inferences. Philosophic knowledge claims should not rely on empirical justification. To conclude `what ought to be' on the basis of `what is' is to commit the naturalistic fallacy (Steiner, 1988). For example, it is empirically true that murder and wars do exist and have existed. However, these empirical facts do not justify the claim that `humans ought to kill each other.' Indeed, this is inconsistent with the moral principles of benevolence and justice. From these principles (axioms) we would infer that `humans should respect one another,' and if we do respect humans, then we `ought not kill each other.' Type 1 Knowledge: Situated, Scientific This is empirically justified knowledge that describes a specific situation in education. Examples of knowledge claims: King (1998) observed several museum schools in Washington, D.C, New York City, and Minneapolis, Minnesota. In this multiple-case study she described in rich detail and supplemented with photographs how these Types of Knowledge of Education -- 8 particular public schools and museums worked together to provide educational experiences for elementary and secondary students. This is situated knowledge, and describes what was the case in 1997 when she observed these museum schools. A further example of situated, scientific knowledge is statistics about children with disabilities in the United States over a ten-year period: Between 1986 and 1996, the number of students with learning disabilities (LDs) who were educated in regular classrooms increased by nearly 20 percent, whereas the percentage served in resource rooms or separate classes decreased substantially (National Center for Educational Statistics, 1999). (Holloway, 2001, p. 86) Useful research methods: Case study research, historical research methods, naturalistic methods, descriptive statistics. Type 2 Knowledge: Theoretical, Scientific This is empirically justified knowledge which describes elements, patterns or relationships which are true in general. Examples of knowledge claims: William Glasser (1998) has developed what he calls `Choice Theory'. This theory claims that all human behavior is purposeful in order to meet one or more basic needs. He classifies needs for: survival, love and belonging, power, freedom, and fun (Erwin, 2004). Maccia and Maccia (1966) developed a theory of school systems based on general systems theory, information theory, di-graph theory and set theory. One of their 201 hypotheses is 12a: "If school resource increases, then school filtrationness decreases." (p. 139) Another hypothesis is 64a: "If school hierachically orderness increases, then school vulnerableness increases and school flexibleness decreases." (p. 145) Types of Knowledge of Education -- 9 Estep (2003) has developed the Theory of Immediate Awareness. In her theory, she claims that "(k)nowing how is far more fundamental in our intelligence than knowledge that because it is logically, epistemologically, and temporally prior to our knowing propositional (knowledge that) statements." (p. xvii) Useful research methods: Experiments, surveys, observational studies, metaanalyses ­ with statistical inferences from sample to population. Type 3 Knowledge: Situated, Praxiological This is empirically justified knowledge which describes the instrumental value of a particular educational product, program or situation. The focus is on effectiveness how well the design or solution works or functions. Examples of knowledge claims: Corry, Frick & Hanson (1997) described the design process for improving the effectiveness of the Indiana University Bloomington Web site. They conducted usability tests of the existing 1995 Website and of a new design with representative members of the target audiences. Their usability tasks required prospective students, parents, faculty and staff, and current undergraduate and graduate students to attempt to find information based on frequently asked questions at IUB. Findings from usability tests: Success rates. Subjects using the proposed Web site found many more locations containing answers to the most frequently asked questions than did subjects using the existing Web site. Efficiency. When subjects found answer locations in both the existing and the proposed Web sites, in most cases they were able to find the location two or three times faster using the proposed Web site. Subjects of the proposed Web site were able to find most answer locations in less than one minute. Types of Knowledge of Education -- 10 Alphabetical list of links. One of the satellite pages of the existing Web site consistently performed better than the proposed site. This page contained a long, alphabetized list of all on-line departments. (p. 71) As a further example, Shavelson and Towne (2002) discuss instances of evaluations of particular program effectiveness: ... among the education programs whose effectiveness have been evaluated in randomized trials are the Sesame Street television series (Bogatz and Ball, 1972), peer-assisted learning and tutoring for young children with reading problems (Fuchs, Fuchs, and Kazdan, 1999), and Upward Bound (Myers and Schirm, 1999). And many of these trials have been successfully implemented on a large scale, randomizing entire classrooms or schools to intervention conditions. (p. 112) It should be noted here that Shavelson and Towne's use of the notion of `scientific research in education' is a subset of what is being referred to in the present article as `disciplined inquiry' (note 1). Useful research methods: Usability testing, formative and summative evaluation, experiments to compare effectiveness. Type 4 Knowledge: Theoretical, Praxiological This is empirical knowledge which describes the instrumental value of a methodology for creating aneffective educational state of affairs, an educational product, or program. In other words, such knowledge constitutes generalizable prescriptions, processes, principles or heuristics which are means to an end. Examples of knowledge claims: David Merrill (2002) posits five of what he calls `first principles' of instruction. He claims that learning will be less effective if one or more of these principles is absent from instruction: 1. Learning is promoted when learners are engaged in solving real-world problems. Types of Knowledge of Education -- 11 2. Learning is promoted when existing knowledge is activated as a foundation for new knowledge. 3. Learning is promoted when new knowledge is demonstrated to the learner. 4. Learning is promoted when new knowledge is applied by the learner. 5. Learning is promoted when new knowledge is integrated into the learner's world. (pp. 44-45) The means are learner engagement in real-world problem solving, activation of existing knowledge, demonstration, application and integration. The end is promotion of learning. Nelson (1999) prescribes a nine-step process for collaborative problem solving in education. She claims that this process is needed (the means) in order to make collaborative problem solving work optimally (the end): 1. Instructor and learners establish and build their readiness to engage in collaborative group work. 2. Either the instructor or the learners form small, heterogeneous work groups, and then the groups engage in norming processes. 3. Groups engage in a preliminary process to define the problem they will work on. 4. Each group defines what roles are necessary to accomplish the design plan and then assigns them. 5. The group engages in the primary, iterative CPS process. 6. Groups begin to finalize their solutions or projects. 7. The instructor and learners engage in activities to help them reflect and synthesize their experiences. 8. The instructor, and, when appropriate, the learners access their products and processes. 9. The instructor and learners develop an activity to bring closure to the learning event. (pp. 257-266) Useful research methods: Formative research methodology for developing instructional theory (Reigeluth & Frick, 1999), analysis of patterns in time (Frick, 1990). Types of Knowledge of Education -- 12 Type 5 Knowledge: Situated, Philosophical This is knowledge which describes the worthwhileness (intrinsic value) of a unique state of affairs in education. Examples of knowledge claims: Jorgenson & Vanosdall (2002) are critical of the risks in the current rush towards standardized testing and its potential negative impact on student learning in science: Despite the revolutionary results from El Centro and the successes in the other districts, many teachers and school administrators nationwide currently will not or cannot ­ devote attention to science instruction. The vast majority of school systems today are locked in a frenzied struggle to better prepare their teachers and students for the high-stakes standardized tests that are sweeping through the U.S. state by state. Increasingly, politicians, the media, and the public have decried the academic performance of our schools based solely on the results of tests of student achievement. Consequently, preparation for basic skills tests has become the fixation in public school districts. In some Arizona school systems, for example, testing required by the state and individual districts already consumes 20% of a student's total time in class. And the pressure will only intensify with the passage in January of the federal No Child Left Behind Act, with its provision for annual testing in grades 3 through 8 in reading and math. Thus we continue to weigh the elephant again and again, rather than feed it, and still we expect it to grow. (pp. 603-604) Marshak (2003) views No Child Left Behind as good in one sense: that every child should get a high-quality education ­ but bad in another: that many will get harmed in the short term. GEORGE W. BUSH deserves significant credit for one policy achievement in education. No Child Left Behind, his stated goal, has become the title of the education bill he signed into law in January 2002. President Bush is the first American President who has affirmed so clearly that every single child deserves a high-quality education ­ and that no child should be victimized by malignant or benign neglect. The President has articulated a new goal for American public schools. But despite all the hype emanating from Washington, nothing else in schools has really changed, except for a lot more testing to come, a list of prospective penalties, and a sparse handful of dollars per student. And the intensification of standardized testing ­ the key tool that President Bush and his Democratic allies (first and Types of Knowledge of Education -- 13 foremost Sen. Ted Kennedy of Massachusetts, who really should know better) will employ to achieve their goal ­ comes not from the future but from the past. (p. 229) ... the Bush/Kennedy NCLB races foolishly into the past of industrial social forms. It will surely be a disaster, though we can hope that it will so discredit the industrial paradigm of schooling that we can finally let it go ­ and begin to move ahead. Unfortunately, a lot of children and teens and teachers and parents will get hurt in the process. (p. 231) Useful research methods: Logic, philosophic reasoning, and evaluation in a broader sense than often construed: applying criteria to determine merit or worth of a specific situation­ i.e., a critique or criticism. Type 6 Knowledge: Theoretical, Philosophical This is knowledge which describes what ought to be, what is worthwhile in general in education. Examples of knowledge claims: Steiner (1981) provides justification for what education should be, referring to a teaching-learning process that is both guided and intended, and that the content of education should be the best of culture: To be learning within education, the learning must be intended by the learner. The learner must deliberately engage in learner tasks. This follows from education being a process involving human learners. Since human learners are human beings, they are active not reactive learners.... (p. 31) To be sure, if education is to be worthwhile, then the culture that is selected should be the arts and fund of intelligence.... (p. 60). Steiner (1988) later presents her rationale of what education ought to be: Since `education' is derived from the Latin `educo' to lead out, I take education, not in Dewey's sense, but in the sense of both intended and guided learning.... Education, then, becomes the teaching-studenting process. Teaching is a process of guiding learning, and studenting is a learning process of a conscious learner, an I or one intending learning. (p. 16) Types of Knowledge of Education -- 14 Notice that Steiner (1988) subsequently introduced the notion of `studenting' instead of `learning' in order to clarify the difference between intended learning and other kinds of learning, and that guiding of learning does not imply direct instruction such as lecturing. Freire (1993) arrives at a similar conclusion regarding intentionality of teachers and students, but from a different social context: A revolutionary leadership must accordingly practice co-intentional education. Teachers and students (leadership and people), co-intent on reality, are both Subjects, not only in task of unveiling that reality, and thereby coming to know it critically, but in the task of re-creating that knowledge. As they attain this knowledge of reality through reflection and action, they discover themselves as its permanent re-creators. In this way, the presence of the oppressed in the struggle for their liberation will be what it should be: not pseudo-participation, but committed involvement. (p. 51) Useful research methods: Logic, philosophic reasoning ­ i.e., methods of philosophy such as phenomenological method, deduction, and syllogism. Forms of Knowledge Claims A knowledge claim is an assertion. It is a statement that can be tested or verified in some way. For example, "the moon is made of green cheese" is an assertion(Type 1, but not in the domain of education, and a false claim). The examples in Table 2 provide prototypical forms of claims. Note that a claim is represented by a sentence or a group of sentences, not a research study or report. Such studies or reports often assert many knowledge claims, some of which are from review of research done previously and others which are the focus of the study itself ­ what is being added to knowledge or being emended. Types of Knowledge of Education -- 15 Table 2. Forms of knowledge claims that when adequately verified become knowledge of education. Scope of Knowledge of Education Created through Disciplined Inquiry Purpose of Inquiry Situated Theoretical Scientific: `What is?' Type 1: x1, x2 ... are true properties of an existing situation, S1. Type 2: X, Y, and the relationship XY tend to be true in general, independent of time and space. Praxiological: `What is instrumental?' Type 3: unique product, program, or invention P1 is effective. Type 4: X1, X2 ... are methods which are generally effective. Philosophical: What ought to be?' Type 5: x1, x2 ... are intrinsically valuable properties of an existing situation, Z1 what is good or bad about Z1. Type 6: X, Y, and the relationship XY are intrinsically valuable principles ­ in general what is good or bad in itself. Relationships among Types of Knowledge To characterize knowledge as in Table 1 does not imply that types of knowledge are unrelated. For example, instructional design theory (Type 4) can certainly influence the process of development of a particular instructional product (Type 3), and vice versa ­ i.e., repeated experience in developing products can help improve design methodologies. Educational philosophy can influence what is designed as well ­ e.g., what kinds of instructional products are worth developing. Educational philosophy can also influence what kinds of student learning are important. Scientific, theoretical knowledge of education can be useful in creating instructional design theories. Types of Knowledge of Education -- 16 Criteria for Evaluating Adequacy of Knowledge Claims Should Be Different It can also be seen that research methods which are useful for creating some kinds of knowledge are not necessarily useful or appropriate for developing other kinds of knowledge. For example, experimental methods are typically of little or no use for Types 1 and 4 knowledge, but have utility for Type 2 when cause and effect relationships are studied, and for Type 3 when effectiveness of an educational program or product is being tested. Experimental methods are inappropriate for Type 5 and 6 knowledge. Furthermore, criteria for judging what constitutes adequate research will differ from one kind of knowledge to another. For example, criteria pertaining to generalizability from a random sample to a population will not be appropriate for judging the quality of Type 1 and 5 research outcomes, but are appropriate for Type 2 and 3. Moreover, for Types 1, 3 and 5 knowledge, generalizability is restricted to the scope of the situation investigated. Methods of inductive inference (inferential statistics), which are important in Type 2 knowledge and may be of use for verifying Type 3 knowledge, are inappropriate for Types 1, 4, 5 and 6. In short, criteria used for judging the merit of one kind of inquiry are not necessarily appropriate for judging another. There is Value for Each of these Kinds of Educational Knowledge Type 1 research outcomes are useful for learning about what has already been done or is being done in education. Phi Delta Kappa magazine often has articles of this kind. Many articles often tell stories about particular schools or educational programs. Types of Knowledge of Education -- 17 Type 2 research is important for predicting or explaining theoretical relationships in education. Knowledge about these generalizable relationships can inform education practice indirectly. They can also inform types 3 and 4 design and development processes. Type 3 research is important in order to develop useful educationalprograms, products or materials. Case studies of unique designs can inform other designers of how particular problems were solved or overcome. Studies of program or product effectiveness are important for making decisions about their specific utility. Type 4 research is important in order to inform methodology. Principles of design, heuristics, and guidelines can inform the design of new products and can influence extant educational practice. Principles of instruction and instructional theories can also inform educational practice and help improve learning in the classroom. These are generalizable, theoretical prescriptions which have instrumental value. Type 5 research is important for identifying what is intrinsically good or bad about particular educational states of affairs, programs or products. These are not questions of effectiveness, but questions of worthwhileness. For example, punishment as a form of student discipline may be effective in schools in Houston, Texas, but it cannot be justified morally or rationally if one holds benevolence and justice as important values ­ i.e., minimizing harm to humans, and by treating them individually as everyone should be treated (cf. Kant's categorical imperative, 1998). Type 6 research is important for identifying what ought to be done in education. This is important for establishing goals or visions of what education could be. General policies or philosophy can guide the development of educational programs and practice. Types of Knowledge of Education -- 18 They do not tell us how to do education, but what education we should do. These are not empirical questions, but rather what educational ends we should seek. Conclusion Educational practitioners often complain that is difficult to apply educational research outcomes to classroom practice. I believe that this is because the large majority of educational research published has been Type 1 and 2. Descriptions of `what is' are inadequate for deciding what educational ends are worthwhile and for effective means to achieve such ends. The six-fold typology of knowledge of education presented here includes philosophy and praxiology as legitimate domains of disciplined inquiry in education. Scientific research in education is insufficient by itself. Educators can benefit from instrumental knowledge ­ what specific educational programs and instructional products are likely to be effective (Type 3), and what instructional theories (methodologies) are likely to work (Type 4). Educational practitioners can benefit further from philosophic knowledge ­ what we ought to be doing in education. Notes 1. Shavelson and Towne's notion of scientific research omits philosophic inquiry as a legitimate discipline of inquiry in education, and hence is too narrow. They also give little emphasis to theoretical, praxiological knowledge of education itself (Type 4), which is somewhat ironic in that their book is largely concerned with principles for conducting Types of Knowledge of Education -- 19 educational research, which is praxiological knowledge of how to do research in education (Type 4 knowledge of research principles). References Corry, M., Frick. T. & Hansen, L. (1997). User-Centered Design and Usability Testing of a Web Site: An Illustrative Case Study. Education Technology Research and Development, 45(4), 65-76. Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability of scores and profiles. New York, NY: John Wiley. Erwin, J. C. (2004). The classroom of choice: Giving students what they need and getting what you want. Alexandria, VA: Association for Supervision and Curriculum Development. Freire, P. (1993). Pedagogy of the oppressed (translated by M. B. Ramos, new rev. 20th anniversary ed.). New York: NY, Continuum. Frick, T. W. (1990). Analysis of Patterns in Time (APT): A Method of Recording and Quantifying Temporal Relations in Education. American Educational Research Journal, 27(1), 180-204. Frick, T. W. & Reigeluth, C. M. (1992). Verifying Prescriptive Instructional Theory by Analysis of Patterns in Time. At the annual meeting of the American Educational Research Association (AERA), San Francisco, CA, April, 1992. Glasser, W. (1998). Choice Theory: A new psychology of personal freedom. New York, NY: HarperCollins. Types of Knowledge of Education -- 20 Holloway, J. (2001). Research link / Inclusion and students with learning disabilities. Educational Leadership, 58(6), 86-88. Kant, I. (1998). Critique of pure reason (Translated by P. Guyer & A. Wood). New York, NY: Cambridge University Press King, K. S. (1998). Alternative educational systems: a multi-case study in museum schools. Bloomington, IN: Thesis (Ph.D.), Indiana University. Jorgenson, O., & Vanosdall, R. (2002). The death of science? What we risk in our rush toward standardized testing and the three R's. Phi Delta Kappan, 83(8), 601-605. Maccia, E. S. & Maccia, G. S. Development of educational theory derived from three educational theory models (Final report, Project No. 5-0-638). Washington, DC: U.S. Department of Health, Education and Welfare, Office of Education, Bureau of Research. Marshak, D. (2003). No Child Left Behind: A foolish race into the past. Phi Delta Kappan, 85(3). 229-231. Merrill, M. D. (2002). First principles of instruction. Education Technology Research and Development, 50(3), 43-59. Nelson, L. M. (1999). Collaborative problem solving. In C. M. Reigeluth (Ed.), Instructional-design theories and models Volume II: A new paradigm of instructional theory. Mawah, NJ: Lawrence Erlbaum, 241-268. Peirce, C. S. (1958). Values in a universe of chance. (Ed., P. P. Weiner). New York, NY: Doubleday. Reigeluth, C. M. & Frick, T. W. (1999). Formative research: A methodology for creating and improving design theories. In C. M. Reigeluth (Ed.), Instructional- Types of Knowledge of Education -- 21 design theories and models Volume II: A new paradigm of instructional theory. Mawah, NJ: Lawrence Erlbaum, 633-652. Shavelson, R. J. & Towne, L. (Eds.) (2002). Scientific research in education. Washington, DC: National Academy Press. Shavelson, R. J. & Webb, N. M. (1991). Generalizability theory: A primer. Newbury Park, CA: Sage. Steiner, E. (1981). Educology of the free. New York, NY: Philosophic Library. Steiner, E. (1988). Methodology of theory building. Sydney, Australia: Educology Research Associates.