A Profile of Student Performance in Reding 3n4 Science Introduction......................................................................................................272 How reading literacy is measured in PISA............................................272 Student performance in reading...............................................................273 ■ Proficiency at Level S (above 625 score points)........................................276 ■ Proficiency at Level 4 (from 553 to 625 score points).............................278 ■ Proficiency at Level 3 (from 481 to 552 score points).............................278 ■ Proficiency at Level 2 (from 408 to 480 score points).............................278 ■ Proficiency at Level 1 (from 335 to 407 score points) or below (below 335 score points)...............................................................................279 ■ The mean performances of countries in reading.......................................280 ■ Differences in reading performance between PISA 2000 and PISA 2003.................................................................................................282 ■ Gender differences in reading literacy........................................................284 How science performance is measured in PISA.................................286 Student performance in science................................................................293 ■ The mean performances of countries in science........................................293 ■ Differences in science performance between PISA 2000 and PISA 2003.................................................................................................295 ■ Gender differences in science.......................................................................296 Implications for policy..................................................................................298 ■ Reading............................................................................................................298 ■ Science.............................................................................................................299 0 OECD 2004 Learning for Tomorrow's World - First Results from PISA 2003 271 INTRODUCTION The 2003 survey provides an update of reading and science performance. In PISA 2003, the areas of reading and science were given smaller amounts of assessment time than mathematics (the focus of the 2003 assessment), with 60 minutes for each allowing an update on overall performance rather than the kind of in-depth analysis of knowledge and skills shown for mathematics in Chapter 2 .This chapter describes how PISA 2003 measures student achievement in reading and science, examines student outcomes in these two areas, and also compares outcomes for PISA 2003 with PISA 2000. PISA measures students' applied ability to deal with written material... .. .through handling different kinds of texts... .. .and performing different types of reading tasks... .. .in relation to various situations where reading is needed. HOW READING LITERACY IS MEASURED IN PISA Reading literacy focuses on the ability of students to use written information in situations which they encounter in their life. In PISA, reading literacy is defined as understanding, using and reflecting written texts, in order to achieve one's goals, to develop one's knowledge and potential and to participate in society. This definition goes beyond the traditional notion of decoding information and literal interpretation of what is written towards more applied tasks. The concept of reading literacy in PISA is defined by three dimensions: the format of the reading material, the type of reading task or reading aspects, and the situation or the use for which the text was constructed. The first dimension, the text format, classifies the reading material or texts into continuous and non-continuous texts. Continuous texts are typically composed of sentences that are, in turn, organised into paragraphs. These may fit under larger structures such as sections, chapters and books. Non-continuous texts are organised differently from continuous texts as they require a different reading approach and can be classified according to their format. Outcomes of students on two reading scales based on the form of the text were reported in the PISA 2000 report Reading for Change — Performance and Engagement across Countries (OECD, 2002b). The second dimension is defined by the three reading aspects. Some tasks required students to retrieve information — that is, to locate single or multiple pieces of information in a text. Other tasks required students to interpret texts — that is, to construct meaning and draw inferences from written information. The third type of task required students to reflect on and evaluate texts — that is, to relate written information to their prior knowledge, ideas and experiences. In PISA 2000 student performance in these three types of task were each reported on a separate scale. In 2003, however, less assessment time was allocated to reading and results are reported only on a single reading literacy scale that combines the three types of tasks. The third dimension, the situation or context, reflects the categorisation of texts based on the author's intended use, the relationship with other persons implicitly or explicitly associated with the text, and the general content. The situations included in PISA and selected to maximise the diversity of content included in the reading literacy assessment were reading for private use (personal), reading for public use, reading for work (occupational) and reading for education. S OECD 2004 Learning for Tomorrows World - First Results from PISA 2003 A full description of the conceptual framework underlying the PISA assessment of reading literacy is provided in The PISA 2003 Assessment Framework: Mathematics, Reading, Science and Problem Solving Knowledge and Skills (OECD, 2003e). STUDENT PERFORMANCE IN READING The principles for the reporting of results in reading are similar to those applied for mathematics (see Chapter 2). However, unlike in mathematics, where the scales were newly established for the 2003 assessment, the PISA 2003 reading scale is anchored to the results of the 2000 assessment. Since reading was the focus of the 2000 assessment, it was possible to fully develop the instrument for measuring reading literacy at that stage, so the PISA 2000 mean of 500 has been established as the benchmark against which future reading performance will be measured. For reading literacy, PISA 2003 uses an identical framework and a subset of items from PISA 2000. To ensure comparability in calculating trends, the 28 reading items used in PISA 2003 are a subset of the 141 items used in 2000. The subset of items was selected taking the relative balance of aspects of the framework into account; for example, in both years, the proportion of items falling into each task classification is similar (see Table A6.2 for the breakdown of items by the various aspects of the framework). Therefore, the reading literacy results that are presented in this chapter are based on the reading literacy proficiency scale that was developed for PISA 2000 which had a mean of 500 and a standard deviation of 100 for the 27 OECD countries that participated. The PISA 2003 results include 29 OECD countries — the Slovak Republic and Turkey joined PISA in 2003 and the Netherlands met all technical standards in 2003, while the United Kingdom has been excluded from the results as it failed to reach the technical standards required by PISA 2003. For the 25 OECD countries for which comparable data are available for both the PISA 2000 and 2003 assessments, the average performance has essentially remained unchanged.1 However, mainly because of the inclusion of new countries in 2003, the overall OECD mean for reading literacy is now 494 score points and the standard deviation is 100 score points. As in 2000, reading scores in 2003 are reported according to five levels of proficiency, corresponding to tasks of varying difficulty. Proficiency levels are defined by tasks sharing common characteristics including conceptual or substantive as well as statistical ones so that tasks within each level meet certain technical specifications (see Chapter 2). Level 5 corresponds to a score of more than 625, Level 4 to scores in the range 553 to 625, Level 3 to scores from 481 to 552, Level 2 to scores from 408 to 480, and Level 1 to scores from 335 to 407. Students at a particular level not only demonstrate the knowledge and skills associated with that level but also the proficiencies required at lower levels. For example, all students proficient at Level 3 are also proficient at Levels 1 and 2. All students at a given level are expected to answer at least half of the items at that level correctly (see Chapter 2). PISA 2003 measures reading in the framework established in 2000, using a subset of tasks used in the PISA 2000 assessment... .. .and reports results on the same scale that was used in 2000. The scale divides students into five levels of proficiency... ... according to the difficulty of tasks that they can usually answer correctly... 0 OECD 2004 Learning for Tomorrow's World - First Results from PISA 2003 FícjUre 6.1 ■ Summary descriptions for the five levels of proficiency in reading literacy Retrieving information Interpreting Reflecting: and evaluating Locate and possibly sequence or combine multiple pieces of deeply embedded information, some of which may be outside the main body of the text. Infer which information in the text is relevant to the task. Deal with highly plausible and/or extensive competing information. Either construe the meaning of nuanced language or demonstrate a full and detailed understanding of a text. Critically evaluate or hypothesise, drawing on specialised knowledge Deal with concepts that are contrary to expectations and draw on a deep understanding of long or complex texts. Con ontinuous texts: Analyse texts whose discourse structure is not obvious or clearly : the relationship of specific parts of the text to its implicit theme or intention. Non-continuous texts: Identify patterns among many pieces of information presented in a display which may be long and detailed, sometimes by referring to information external to the display. The reader may need to realise independently that a full understanding of the section of text requires reference to a separate part of the same document, such as a footnote. Locate and possibly sequence or combine multiple pieces of embedded information, each of which may need to meet multiple criteria, in a text with familiar context or form. Infer which information in the text is relevant to the task. Contii Use a high level of text-based inference to understand and apply categories in an unfamiliar context, and to construe the meaning of a section of text by taking into account the text as a whole. Deal with ambiguities, ideas that are contrary to expectation and ideas that are negatively worded. Use formal or public knowledge to hypothesise about or critically evaluate a text. Show accurate understanding of long or complex texts. tinuous texts: Follow linguistic or thematic links veral paragraphs, often in the abs of cle discourse markers, in order to locate, interpret or evaluate embedded information or to infer psychological or metaphysical meaning. Non-continuous texts: Scan a long, detailed text in order to find relevant information, often with little or no assistance from organisers such as labels or special formatting, to locate several pieces of information to be compared or combined. Locate, and in some cases recognise the relationship between pieces of information, each of which may need to meet multiple criteria. Deal with prominent competing information. Integrate several parts of a text in order to identify a main idea, understand a relationship or construe the meaning of a word or phrase. Compare, contrast or categorise taking many criteria into account. Deal with competing information. Make connections or comparisons, give explanations, or evaluate a feature of text. Demonstrate a detailed understanding of the text in relation to familiar, everyday knowledge, or draw on less common knowledge. Continuous texts: Use conventions of text organisation, where present, and follow implicit or explicit logical links such as cause and effect relationships across sentences or paragraphs in order to locate, interpret or evaluate information. Non-continuous texts: Consider one display in the light of a second, separate document or display, possibly in a different format, or combine several pieces of spatial, verbal and numeric information in a graph or map to draw conclusions about the information represented. 0 OECD 2004 Learning for Tomorrows World - First Results from PISA 2003 Fl (J U re 6.1 (continued) ■ Summary descriptions for the five levels of proficiency in reading literacy Retrieving information Interpreting Reflecting and evaluating Locate one or more pieces of information, each of which may be required to meet multiple criteria. Deal with competing information. Identify the main idea in a text, understand relationships, form or apply simple categories, or construe meaning within a limited part of the text when the information is not prominent and low-level inferences are required. Make a comparison or connections between the text and outside knowledge, or explain a feature of the text by drawing on personal experience and attitudes. Continuous texts: Follow logical and linguistic connections within a paragraph in order to locate or interpret information; or synthesise information across texts or parts of a text in order to infer the author's purpose. Non-continuous texts: Demonstrate a grasp of the underlying structure of a visual display such as a simple tree diagram or table, or combine two pieces of information from a graph or table. Locate one or more independent pieces of explicitly stated information, typically meeting a single criterion, with little or no competing information in the text. Recognise the main theme or author's purpose in a text about a familiar topic, when the required information in the text is prominent. Make a simple connection between information in the text and common, everyday knowledge. Continuous texts: Use redundancy, paragraph headings or common print conventions to form an impression of the main idea of the text, or to locate information stated explicitly within a short section of text. Non-continuous texts: Focus on discrete pieces of information, usually within a single display such as a simple map, a line graph or a bar graph that presents only a small amount of information in a straightforward way, and in which most of the verbal text is limited to a small number of words or phrases. Students scoring below 335 score points, i.e., those who do not reach Level 1, are not able to routinely show the most basic skills that PISA seeks to measure. While such performance should not be interpreted to mean that those students have no literacy skills at all, performance below Level 1 does signal serious deficiencies in students' ability to use reading literacy as a tool for the acquisition of knowledge and skills in other areas. Similarly, since Level 5 is also unbounded, some students participating in PISA may demonstrate higher reading skills than those measured by the assessment. The establishment of proficiency levels in reading makes it possible not only to rank students' performance but also to describe what they can do (Figure 6.1). Each successive reading level is associated with tasks of ascending difficulty. The tasks at each level of reading literacy were judged by panels of experts to share certain features and requirements and to differ consistently from tasks at either higher or lower levels. The assumed difficulty of tasks was then validated empirically on the basis of student performance in participating countries. .. .plus a sixth group made up of those unable to show basic Junctional reading skills. Tasks in each proficiency level have identifiable Jeatures... 0 OECD 2004 Learning for Tomorrow's World - First Results from PISA 2003 ... with easier tasks requiring basic handlina equi ng basic handling of simple texts... .. .and harder ones involving Increasing complexity and less explicit Information. The reading literacy tasks used in PISA 2003 include the three dimensions previously described and have a diverse range of difficulty. Samples of the reading tasks (a total of 45 items) were released after PISA 2000 and can be found in the publication Sample Tasks from the PISA 2000 Assessment — Reading, Mathematical and Scientific Literacy (OECD, 2002c). Each item includes an indication of the dimension being assessed, and a description of the knowledge and skills being assessed. These descriptions provide some insight into the range of processes required of students and the proficiencies which they need to demonstrate to reach different reading levels. Further sample tasks can also be found at www.plsa.oecd.org. Even a cursory review of these items reveals that tasks at the lower end of the scale require very different skills from those at the higher end. A more careful analysis of the range of tasks provides some indication of an ordered set of knowledge-construction skills and strategies. For example, the easiest of these tasks require students to locate explicitly stated information according to a single criterion where there is little, if any, competing information in the text, or to identify the main theme of a familiar text, or make a simple connection between a piece of the text and everyday life. In general, the information is prominent in the text and the text itself is less dense and less complex in structure. In contrast, harder retrieval tasks require students to locate and sequence multiple pieces of deeply embedded information, sometimes in accordance with multiple criteria. Often there is competing information in the text that shares some features with the information required for the answer. Similarly, with tasks requiring interpretation or reflection and evaluation, those at the lower end differ from those at the higher end in terms of the process needed to answer them correctly, the degree to which the reading strategies required for a correct answer are signalled in the question or the instructions, the level of complexity and familiarity of the text and the quantity of competing or distracting information present in the text. Figure 6.2 presents an overall profile of proficiency on the reading literacy scale, with the length of the bars showing the percentage of students proficient at each level. The hardest tasks are sophisticated and require critical thinking... Proficiency at Level 5 (above 625 score points) Students proficient at Level 5 on the reading literacy scale are capable of completing sophisticated reading tasks, such as managing information that is difficult to find in unfamiliar texts; showing detailed understanding of such texts and inferring which information in the text is relevant to the task; and being able to evaluate critically and build hypotheses, draw on specialised knowledge, and accommodate concepts that may be contrary to expectations. See Figure 6.1 for a more detailed description. .. .measuring the kind of The proportion of students performing at the highest PISA proficiency levels skill needed by high-level in participating countries are of interest as today's proportion of students knowledge workers. performing at these levels may influence the contribution which that country 0 OECD 2004 Learning for Tomorrows World - First Results from PISA 2003 FícjUre 6.2 ■ Percentage of students at each level of proficiency on the reading scale Below Level 1 I Level 1 I Level 2 Level 3 Level 4 Level S Percentage of students 100 75 SO 25 25 50 75 III 100 ■c g 1) i-c 0 C C tí tí i-o i-o e « "13 -S g g -š 1 (S u u ti J3 -3 ^5 « ^T í- 4) ,, JT M" N ^ _^ cc C C M C C D Z 4 >- TJ . c U Z ■o . g ŕ "í >- u c Ň Z i»fi »fl^ g-g «I I Š ^'"Ü rt ťt I ?•! "p § S S "u : ^ § T) Ö £ < "c 3 «Í23 < a>i ^s U*?** "g = 3 C E ft- ^ (ť O D 3 3 -n ^ 3 60 % '- U1 i « i- * S ^ 2 g o .5 g n S- p— D C C ■c c 3 3! Countries are ranked in descending order of percentage of 15-year-olds in Levels 3, 4 and 5. Source: OECD PISA 2003 database,Table 6.1. will make towards the pool of tomorrow's world-class knowledge workers in the global economy. In the combined OECD area, 8 per cent of the students are at proficiency Level 5. More than 16 per cent of the students in New Zealand and more than 12 per cent of the students in Australia, Belgium, Canada, Finland, Korea and the partner country Liechtenstein are at this level. In contrast, less than 1 per cent of the students in Mexico reach Level 5 and this is also true in the partner countries Indonesia, Serbia,2 Thailand and Tunisia (Figure 6.2 and Table 6.1). It is important to keep in mind that the proportion of students performing at Level 5 is influenced not only by the overall performance of countries in reading literacy but also by the variation that exists within countries between the students with the highest and the lowest levels of performance. While there is a general tendency for countries with a higher proportion of students scoring Having more students at this top level does not always go with havingfewer at lower performance levels. 0 OECD 2004 Learning for Tomorrow's World - First Results from PISA 2003 at Level 5 to have fewer students at Level 1 and below, this is not always the case. In Finland, for example, 15 per cent of students reach Level 5 while only 1 per cent are below Level 1. By contrast, in Belgium and New Zealand, which also have high percentages reaching Level 5, a relatively high proportion of students score below Level 1 as well (8 and 5 per cent respectively). Finally, in the partner countries Hong Kong-China and Macao-China, only 6 per cent and 2 per cent, respectively, reach Level 5, while only 3 per cent and 1 per cent, respectively, score below Level 1. Proficiency at Level 4 (from 553 to 625 score points) In some countries around Students proficient at Level 4 on the reading literacy scale are capable of difficult 40 per cent of students reading tasks, such as locating embedded information, dealing with ambiguities can at least do difficult and critically evaluating a text (Figure 6.1). In the combined OECD area, 28 tasks at Level 4, but in per cent of students are proficient at Level 4 or above (that is, at Levels 4 and 5) others very jew can. (Figure 6.2 andTable 6.1). Nearly half of the students in Finland and between 40 and 50 per cent or more of those in Australia, Canada, Korea and New Zealand and the partner country Liechtenstein attain at least Level 4. With the exception of Mexico, the Slovak Republic and Turkey, at least one in five students in each OECD country reaches at least Level 4. In addition, fewer than 5 per cent of the students in four of the partner countries — Indonesia, Serbia, Thailand and Tunisia — reach this level. Proficiency at Level 3 (from 481 to 552 score points) Most students in OECD Students proficient at Level 3 on the reading literacy scale are capable of reading countries have at least tasks of moderate complexity, such as locating multiple pieces of information, moderate reading making links between different parts of a text and relating it to familiar everyday skills... knowledge (Figure 6.1). In the combined OECD area, 55 per cent of students are proficient at least at Level 3 (that is, at Levels 3, 4 and 5) on the reading literacy scale (Figure 6.2 andTable 6.1). In 8 of the 30 OECD countries (Australia, Canada, Finland, Ireland, Korea, the Netherlands, New Zealand and Sweden), and in two partner countries (Hong Kong-China and Liechtenstein), between 65 and 80 per cent of 15-year-old students are proficient at least at Level 3. This level is the OECD modal level — that is, the one at which most students are placed at their highest level of proficiency, with 27 per cent in the OECD combined area. Proficiency at Level 2 (from 408 to 480 score points) .. .and in all but two Students proficient at Level 2 are capable of basic reading tasks, such as locating OECD countries, at least straightforward information, making low-level inferences of various types, 75 per cent can do basic working out what a well-defined part of a text means and using some outside reading tasks. knowledge to understand it (Figure 6.1). In the combined OECD area, 78 per cent of students are proficient at Level 2 or above on the reading literacy scale. In every OECD country except Mexico and Turkey, at least three in four students are at Level 2 or above (Figure 6.2 andTable 6.1). 0 OECD 2004 Learning for Tomorrows World - First Results from PISA 2003 Proficiency at Level 1 (from 335 to 407 score points) or below (below 335 score points) Reading literacy, as defined in PISA, focuses on the knowledge and skills required to apply reading for learning rather than on the technical skills acquired in learning to read. Since comparatively few young adults in OECD countries have not acquired technical reading skills, PISA does not seek to measure such things as the extent to which 15-year-old students are fluent readers or how well they spell or recognise words. In line with most contemporary views about reading literacy, PISA focuses on measuring the extent to which individuals are able to construct, expand and reflect on the meaning of what they have read in a wide range of texts common both within and beyond school.The simplest reading tasks that can still be associated with this notion of reading literacy are those at Level 1. Students proficient at this level are capable of completing only the simplest reading tasks developed for PISA, such as locating a single piece of information, identifying the main theme of a text or making a simple connection with everyday knowledge (Figure 6.1). Students performing below 335 score points — that is, below Level 1 — are not likely to demonstrate success on the most basic type of reading that PISA seeks to measure. This does not mean that they have no literacy skills. Nonetheless, their pattern of answers in the assessment is such that they would be expected to solve fewer than half of the tasks in a test made up of items drawn solely from Level 1, and therefore perform below Level 1. Such students have serious difficulties in using reading literacy as an effective tool to advance and extend their knowledge and skills in other areas. Students with literacy skills below Level 1 may therefore be at risk not only of difficulties in their initial transition from education to work, but also of failure to benefit from further education and learning opportunities throughout life. In the combined OECD area, 14 per cent of students perform at Level 1, and 8 per cent perform below Level 1, but there are wide differences between countries. In Finland and Korea, only 5 per cent of students perform at Level 1, and 1 per cent below it, but these countries are the exceptions. In all other OECD countries, the percentage of students performing at or below Level 1 ranges from 10 to 52 per cent (Figure 6.2 andTable 6.1). One-quarter of the OECD countries have between 2 and 5 per cent of students performing below Level 1. The OECD countries with 20 per cent or more of students at or below Level 1 are (in descending order): Mexico,Turkey, Greece, the Slovak Republic, Italy, Luxembourg, Germany, Portugal, Spain, Austria and Hungary. This is also the case in the following partner countries (in descending order): Indonesia,Tunisia, Brazil, Serbia, Thailand, Uruguay and the Russian Federation. It is notable that among these countries Germany has the relatively high contrasting figure of close to 10 per cent of its students performing at Level 5. In addition, between 25 and 34 per cent of students do not reach Level 1 in Mexico and in the partner countries Brazil, Indonesia and Tunisia.These students are routinely unable to show the most basic skills that PISA seeks to measure. Level 1 represents the simplest Junctional n9 tasks... .. .and those not reaching it may be able to read but have serious problems using reading for learning. Although over nine out often OECD students can at least perform at Level 1... .. .in 11 OECD countries at least one in five are not proficient beyond Level 1. 0 OECD 2004 Learning for Tomorrow's World - First Results from PISA 2003 The mean performances of countries in reading Country performance can be summarised by a mean score... .. .but a comparison of country means is only possible where there is a statistically significant difference. These mean performances span a wide range, with Tinnish students doing best overall. Within each country however, the range of performance is even greater, and some countries manage to contain this difference better than others. The discussion above has focused on comparisons of the distribution of student performance between countries. Another way to summarise student performance and to compare the relative standing of countries in reading literacy is by way of their mean scores. Given that high average performance at age 15 is predictive of a highly skilled future workforce, countries with high average performance will have a considerable economic and social advantage. As discussed in Chapter 2, when interpreting mean performance, only those differences between countries which are statistically significant should be taken into account. Figure 6.3 shows those pairs of countries where the difference in their mean scores is sufficient to say with confidence that the higher performance by sampled students in one country holds for the entire population of enrolled 15-year-olds. Read across the row for a country to compare its performance with the countries listed along the top of the figure. The colour-coding indicates whether the average performance of the country in the row is significantly lower than that of the comparison country, not statistically different, or significantly higher. When making multiple comparisons — for example, when comparing the performance of one country with that of all other countries, an even more cautious approach is required, and only those comparisons that are indicated by the respective symbols in dark shadings should be considered statistically significant for the purpose of multiple comparisons. The figure also shows which countries perform above, at or below the OECD average. In Finland, performance on the reading literacy scale is above that of any other OECD country. Its country mean, 543 score points, is more than half of a proficiency level above the OECD average of 494 score points in PISA 2003. Other OECD countries with mean performances statistically significantly above the OECD average include Australia, Belgium, Canada, Ireland, Korea, the Netherlands, New Zealand and Sweden. Among the partner countries, Hong Kong-China and Liechtenstein are also part of that group. Eleven OECD countries perform around the OECD average: Austria, the Czech Republic, Denmark, France, Germany, Iceland, Japan, Norway, Poland, Switzerland and the United States. The partner countries Latvia and Macao-China also perform around the OECD average.3 Among OECD countries, differences are relatively large — 143 score points separate the two extreme performances (i.e., highest and lowest performing countries) — and when the partner countries are considered, this is 150 points. Although there are large differences in the mean performance between countries, the variation in performance between students within each country is much larger. One of the major challenges faced by education systems is to encourage high performance while at the same time minimising poor performance.The question of poor performance is particularly relevant to reading literacy because levels of literacy have a significant impact on the welfare of individuals, the state of society and the economic standing of countries in the international arena (OECD, 2003c). Inequality in this context can be examined through the performance distribution as 0 OECD 2004 Learning for Tomorrows World - First Results from PISA 2003 FícjUľG 6.3 ■ Multiple comparisons of mean performance on the reading scale Readir scale 543 (1.6; 534 (3.1; 528 (1.7; 525 (2.1; 525 (3.6; 522 (2.5; 515 (2.6; 514 (2.4; 513 (2.9; 510 (3.7; 507 (2.6; 500 (2.8; 499 (3.3; 498 (3.9; 498 (2.2; 497 (2.9; 496 (2.7; 495 (3.2; 492 (2.8; 492 (1.6; 491 (3.4; 491 (3.8; 491 (3.7; 489 (3.5; 482 (2.5; 481 (2.6; 479 (1.5; 478 (3.7; 476 (3.0; 472 (4.1; 469 (3.1; 442 (3.9; 441 (5.8; 434 (3.4; 420 (2.8; 412 (3.6; 403 (4.6; 400 (4.1; 382 (3.4; 375 (2.8; I hi £ 3 II 543 S34 S28 S2S S2S S22 SIS S14 S13 S10 S07 S00 499 498 498 497 496 49S 492 492 491 491 491 489 482 481 479 478 476 472 469 442 441 434 420 412 403 400 382 37Í S.E, (1,6) (3,1) (1,7) (2,1) (3,6) (2.S) (2,6) (2,4) (2,9) (3,7) (2,6) (2,8) (3,3) (3,9) (2,2) (2,9) (2,7) (3,2) (2,8) (1,6) (3,4) (3,8) (3,7) (3.S) (2.S) (2,6) (l.S) (3,7) (3,0) (4,1) (3,1) (3,9) (5.8) (3,4) (2,8) (3,6) (4,6) (4,1) (3,4) (2,8) • AAA • • • • • • H • A A A A A A Range of rank* AAAAAAAAAAAAAA AAAAAAAAAAAAAA AAAAAAAAAAAAAA AAAAAAAAAAAAAA AAAAAAAAAAAAAA AAAAAAAAAAAAAA • AAAAAAAAAAAAA • AAAAAAAAAAAAA • A* «AAAAAAAAAA • •••••••AAAAAA • • • • • «AAAAAA • ••••«••••AAAAAA • • ••••••••AAAAAA • •• •••••••AAAAAA • ••• ••••••AAAAAA • •••• •••••••A*AA • J • • • • ••••AAAAAA • •••••• •••••A*AA • ••••••• •••••••A • •••••••• ••••••A • ••••••••• •••••• • •••• ••••• • ••••• •••• ank 1 2 2 3 4 6 6 6 8 10 10 10 10 10 10 12 14 12 12 14 20 20 21 21 21 23 25 28 29 ink 1 3 4 5 b 8 y y 10 15 17 18 18 18 19 20 20 20 21 21 24 25 25 2b 2b 27 27 28 29 ank 1 2 2 3 2 4 6 7 7 7 y n 12 12 12 12 12 12 15 17 15 14 14 17 24 24 25 25 2b 27 29 32 32 33 35 35 36 37 39 39 ink 1 3 5 b b 7 10 10 11 12 12 18 20 12 19 21 22 2Í 24 24 24 2i 25 25 28 29 29 30 31 31 31 34 34 34 36 37 38 38 40 40 * Because data are based on samples, it is not possible to report exact rank order positions for countries. However, it is possible to report the range of rank order positions within which the country mean lies with 95 per cent likelihood. Read across the row for a country to compare performance with the countries listed along the top of the chart. The symbols indicate whether the average performance of the country in the row is lower than that of the comparison country, higher than that of the comparison country, or if there is no statistically significant difference between the average achievement of the two countries. Source: OECD, PISA 2003 database. Without the I Bonferroni I adjustment: | With the Bonferroni adjustment: Mean performance statistically significantly higher than in comparison country No statistically significant difference from comparison country Mean performance statistically significantly lower than in comparison country Mean performance statistically significantly higher than in comparison country No statistically significant difference from comparison country Mean performance statistically significantly lower than in comparison country Statistically significantly above the OECD average Not statistically significantly different from the OECD average Statistically significantly below the OECD average D OECD 2004 Learning for Tomorrow's World - First Results from PISA 2003 u OJ u ^n -CT CTI E c ex "CT 3 C seen by the gap in performance between the 5th and the 95th percentiles (Table 6.2). Among OECD countries, Finland and Korea show the narrowest distributions in the OECD with this difference equivalent to 267 score points while at the same time these two countries show the strongest overall performance. From the partner countries, Macao-China has a very narrow distribution with only 220 score points separating the bottom 5th to the top 95th percent of students. Furthermore, in Canada, Denmark, Ireland and the Netherlands and in the partner countries Hong Kong-China, Indonesia, Latvia, Liechtenstein, Serbia andThailand the performance gaps are below 300 score points. On the other hand, Belgium and Germany show the OECD largest gaps in the performance of students in the middle of the distribution at 362 and 357 score points, which is almost one standard deviation more than in Finland and Korea. Differences in reading performance between PISA 2000 and PISA 2003 Kesultsjrom the two Figure 6.4 shows the overall reading scores for PISA 2000 and 2003 and PISA surveys should be indicates differences in performance between the two assessments. However, as compared cautiously. explained in Chapter 2, such differences need to be interpreted with caution. c ex < FfcjUre 6.4 ■ Differences in mean scores between PISA 2003 and PISA 2000 on the reading scale Only countries with valid data for both 2003 and 2000 I Mean score in PISA 2003 I Mean score in PISA 2000 Performance on the ead mg scale 600 SSO soo T . I 5 ! ..*....*. ■ ■ »,"■., ■ . T 4S0 É i i ! m m . ■" ■ ■ lll 400 i I . ............ ■ " 3S0 i i 1 o o o o oooooooooooo o + + + + + i i 1 1 1 1 1 1 i i i i i 1 1 + + + + + + a o a rt (S TI r! ^ -r, T 03 O -r, re Norway Denmark zech Republic Finland Australia Sweden Greece Belgium Hungary Switzerland Brazil Germany —' re 3 a c a c X x. u ti s U 3 * 01 "Ô S z 2 a o «c í * » J a oj ^ SP-c ó "■ se =j a *c o * se a c I tí T S a .a £ I s- o l- 15 12 iä a a t) <5 2 5 -S ^ -a. -a a .5 íl M ■% E .3 a: E e .2 '£ 3 < T 1 s« a 3 0 3 Q. íl I tí OH 3 í-C 15 E 53 s« 3 C E X 3 - I J íl íl í- 0 3 Q. íl e % -íi 1 ■c e '(5 C 0 x SÍ a. lí SÍ -c 1- e 3 -a u íi N -a (5 J-e (5 U WI 3 aí Í> <5 15 *3 3 X M> S3 a «; Sou OECD PISA 2003 database,Table 6.S. Figure 6.7 shows the percentages of males and females scoring at or below Level 1 in reading (Table 6.5). In Iceland, while 10 per cent of females score at or below Level 1, the percentage of males is 27 per cent. In the partner countries Serbia and Thailand, there are at least 20 per cent more males than females at or below Level 1. Among the OECD countries, the smallest differences between the percentages of males and females at lower levels of performance are found in Korea and the Netherlands and in the partner countries, these are found in Liechtenstein and Macao-China. The science assessment emphasises the application of knowledge... HOW SCIENCE PERFORMANCE IS MEASURED IN PISA The emphasis of the PISA 2003 assessment of science is on the application of science knowledge and skills in real-life situations, as opposed to testing particular curricular components. Scientific literacy is defined as the capacity to use scientific knowledge, to identify questions and to draw evidence-based conclusions in order to understand and help make decisions about the natural world and the changes made to it through human activity. This definition is based on three dimensions: scientific knowledge or concepts, scientific processes and the situations or context in which the knowledge and processes are assessed. With the limited assessment time that was available for science in 2003, it was not possible to assess all areas of scientific knowledge, so a sample of concepts 286 0 OECD 2004 Learning for Tomorrows World - First Results from PISA 2003 was assessed. The selection of these concepts from the major scientific fields of physics, chemistry, biological science, and earth and space science was guided by a number of principles. First, the knowledge assessed should be relevant to real life situations. Second, the knowledge assessed should have some enduring relevance to life over the next decade at least.Third, the knowledge required to successfully answer a PISA science item should be related to some important scientific process — that is, it should not be an isolated recall of a piece of information. Three main scientific processes are part of the PISA assessment in 2003. The first of these is describing, explaining and predicting scientific phenomena — important facets of the scientific process. Students were given tasks that involved recognising phenomena, giving explanations and making considered judgements as to the impact of these phenomena. The second is understanding scientific investigation, which involves being able to recognise questions and problems that could be solved using scientific methods and what evidence may be needed to achieve this, and may also involve an understanding of the variables that need to be measured and controlled in an experiment. In addition, students were assessed on their ability to communicate these ideas. The third is interpreting scientific evidence and conclusions, which is concerned with the use of scientific findings as evidence for a diverse range of claims and conclusions. Through the media, students are constantly coming into contact with claims made by advertisers, proponents of change and commentators who use scientific evidence as a justification. The third main aspect of the assessment of science in PISA is a consideration of the areas of application. For PISA 2003 these are science in life and health, science in the earth and environment, and science in technology. The range of assessment tasks includes problems that affect people as individuals (such as food and energy use), as members of a local community (such as the location of a power station) or as world citizens (such as global warming). Following PISA 2000, two units, which contained eight items, were released to give an indication of the type of problems that students were encountering (OECD, 2002c). These items were replaced with newly created ones which underwent an extensive field trial process to ensure they had similar levels of difficulty as the released items. A sufficient number of items was retained to allow linking to occur between the assessments carried out different times. Like performance in reading literacy, performance in science was marked in PISA 2000 on a single scale with an average score of 500 score points and a standard deviation of 100 score points. Approximately two-thirds of students across OECD countries scored between 400 and 600 score points.The same scale was used for the PISA 2003 science assessment.The scale measures students' ability to use scientific knowledge (understanding of scientific concepts), to recognise scientific questions and to identify what is involved in scientific investigations (understanding of the nature of scientific investigation), to relate scientific data to claims and conclusions (use of scientific evidence) and to communicate these aspects of science. .. .focusing on a selection of concepts that are central to science, of enduring relevance and important to real life. Students were required to recognise and explain scientific phenomena, understand scientific investigation and interpret evidence... __with tasks drawn from a range of scientific situations. The 2003 science assessment overlapped with that used in 2000... .. .and results were reported on the same scale 0 OECD 2004 Learning for Tomorrow's World - First Results from PISA 2003 u OJ u ^n -CT CTI FícjUre 6.8 ■ A sample of science items used in PISA: Unit DAYLIGHT DAYLIGHT Read the following information and answer the questions that follow. E c ex "CT 3 C OJ 0 ex < Today, as the Northern rise at 5:55 am and set at 8:42 Hemisphere celebrates its pm, giving 14 hours and 47 longest day, Australians will minutes of daylight. experience their shortest. The President of the In Melbourne,* Australia, the Astronomical Society, Mr Perry sun will rise at 7:36 am and set Vlahos, said the existence of at 5:08 pm, giving nine hours changing seasons in the and 32 minutes of daylight. Northern and Southern Compare today to the year's longest day in the Southern Hemispheres was linked to the Earths 23-degree tilt. Hemisphere, expected on 22 December, when the sun will 'Melbourne is g city in Austrglig gt g Igtitu4e of gbout 38 4ec|rees south of the equgtor. 288 0 OECD 2004 Learning for Tomorrows World - First Results from PISA 2003 DAYLIGHT QUESTION 1 Which statement explains why daylight and darkness occur on earth? A. The earth rotates on its axis. B. The sun rotates on its axis. C. The earth's axis is tilted. D. The earth revolves around the sun. Item difficulty 690 Score í(592) ■.............................................................................................................................. The correct answer is option A. This is a multiple-choice item that requires students to be able to relate the rotation of the earth on its axis to the phenomenon of day and night and to distinguish thisfrom the phenomenon of the seasons, which arisesfrom the tilt of the axis of the earth as it revolves around the sun. All four alternatives given are scientifically correct. -a «3 550 QUESTION 2 In the Figure light rays from the sun are shown shining on the earth. Figure: light rays from the sun Suppose it is the shortest day in Melbourne. Show the earth's axis, the Northern Hemisphere, the Southern Hemisphere and the Equator on the figure. Label all parts of your answer. Score 2 (720) Answers which include a diagram with the Equator tilted towards the sun at an angle between 10° and 45° and the earth's axis tilted towards the sun within the range 10° and 45° from vertical, and the Northern and or Southern Hemispheres correctly labelled (or one only labelled, the other implied). Score 1 (667) Answers which include a diagram with: • the angle of tilt of earth's axis between 10° and 45°, the Northern and/or Southern Hemispheres correctly labelled (or one only labelled, the other implied), but angle of tilt of the Equator not between 10° and 45°; or the Equator missing. • the angle of tilt of the Equator between 10° and 45°, the Northern and/or Southern Hemispheres correctly labelled (or one only labelled, the other implied), but angle of tilt of axis not between 10° and 45°; or axis E c 2 C oj O < the angle of tilt of the Equator between 10° and 45°, and angle of tilt of axis between 10° and 45°, but the Northern and Southern Hemispheres not correctly labelled (or one only labelled, the other implied, or both missing). 400 This is an open-response item that requires students to create a conceptual model in the form of a diagram showing the relationship between the rotation of the earth on its tilted axis and its orientation to the sun on the shortest day for a city in the southern hemisphere. In addition they had to include in this diagram the position of the equator at a 90-degree angle to the tilted axis. Full credit is obtained f the students correctly place and label all three significant elements — the hemispheres, the tilted axis and the equator. Partial credit is givenfor a diagram with two of the three elements correctly placed and labelled. Q OECD 2004 Learning for Tomorrows World - First Results from PISA 2003 289 Figuľe 6.9 ■ A sample of science items used in PISA: Unit CLONING -TT C c oj qj ex. c qj "TT 3 0 jj o i_ ex. < CLONING Read the newspaper article and answer the questions that follow. A copying mgch ne for living beings^ Without any doubt, if there had been From that small piece he removed elections for the animal of the year 1997, the nucleus, then he transferred the nucleus Dolly would have been the winner! Dolly into the egg-cell of another (female) sheep is a Scottish sheep that you see in the (sheep 2). But first he removed from that 5 photo. But Dolly is not just a simple sheep. 20 egg-cell all the material that would have She is a clone of another sheep. A clone determined sheep 2 characteristics in a means: "a copy". Cloning means: "copying lamb produced from that egg-cell. Ian from a single master copy". Scientists Wilmut implanted the manipulated egg- succeeded in creating a sheep (Dolly) that cell of sheep 2 into yet another (female) 10 is identical to a sheep that functioned as a 25 sheep (sheep 3). Sheep 3 became pregnant master copy. and had a lamb: Dolly. It was the Scottish scientist Ian Wilmut Some scientists think that within a few who designed the "copying machine" for years it will be possible to clone people as sheep. He took a very small piece from the well. But many governments have already 15 udder of an adult sheep (sheep 1). 30 decided to forbid the cloning of people by law. 290 © OECD 2004 Learning for Tomorrows World - First Results from PISA 2003 CLONING Question 1 Which sheep is Dolly identical to? A. Sheep 1 B. Sheep 2 C. Sheep 3 D. Dollys father Score I (494) ■...................................................................................................................................... The correct answer is option A. This is a multiple-choice question item that assesses the students' understanding of the process by which the cloning takes place. This is described in detail in the text, and the students are required to carefully read this text to extract the information required. They need to know that the nucleus of the cell contains the material that will determine the characteristics of the off-spring. QUESTION 2 In line 14 the part of the udder that was used is described as"a very small piece". From the article text you can work out what is meant by"a very small piece". That "very small piece"is A. a cell. B. a gene. C. a cell nucleus. D. a chromosome. Item difficulty 690 Score 1(512) -.................................................................................................................... The correct answer is option A. This is a multiple-choice item that requires the students to demonstrate an understanding of the structure of cells. «3 550 "■> QUESTION 3 In the last sentence of the article it is stated that many governments have already decided to forbid the cloning of people by law. Two possible reasons for this decision are mentioned below. Are these reasons scientific reasons? Circle either "Yes" or "No"f or each. Reason: Cloned people could be more sensitive to certain diseases than normal people. People should not take over the role of a Creator. Score I (507) Answers which indicate Yes, No, i Scientific? Yes/No Yes/No ĺ that order. This is a complex multiple-choice item that requires students to show that they can distinguish between statements that are scientifically based and those that are not. One of the aspects of the PISA scientific literacy framework is the notion that students understand scientific investigation and reasoning. The question poses two reasons why governments mightforbid human cloning. One of the reasons is concerned with thefact that cloned people might be more susceptible to disease (a reason that could be said to be "scientific"), while the other is statement that people should not take on the role of a Creator (a valid reason for many people, but one which cannot be said to be"scientific"). Full credit is obtained for correctly labelling both statements. E c 2 C oj O < 400 Q OECD 2004 Learning for Tomorrows World - First Results from PISA 2003 291 More difficult tasks involve more complex concepts and greater skill requirements, and demand more sophisticated scientific knowledge. Science is not rated at proficiency levels, but it is possible to define characteristics of difficult, medium and easy scientific tasks. The increasing difficulty of tasks along the scale involves the complexity of the concepts used, the amount of data given, the chain of reasoning required and the precision required in communication. In addition, the level of difficulty is influenced by the context of the information, the format and the presentation of the question. The tasks in PISA require scientific knowledge involving (in ascending order of difficulty): recall of simple scientific knowledge or common scientific knowledge or data; the application of scientific concepts or questions and a basic knowledge of investigation; the use of more highly developed scientific concepts or a chain of reasoning; and knowledge of simple conceptual models or analysis of evidence in order to try out alternative approaches. Unlike for reading and mathematics (see Chapter 2), the science scale cannot yet be defined in terms of proficiency levels. This will only be possible from 2006 onwards, when science becomes the main focus of the PISA assessment for the first time and when a full instrument for measuring and reporting science will be developed. However, the criteria for harder and easier tasks can still be described in relation to items associated with different points on the science scale. ■ Towards the top end of the science scale (around 690 score points) students are generally able to create or use conceptual models to make predictions or give explanations; to analyse scientific investigations in order to grasp, for example, the design of an experiment or to identify an idea being tested; to compare data in order to evaluate alternative viewpoints or differing perspectives; and to communicate scientific arguments and/or descriptions in detail and with precision. ■ At around 550 score points, students are typically able to use scientific concepts to make predictions or provide explanations; to recognise questions that can be answered by scientific investigation and/or identify details of what is involved in a scientific investigation; and to select relevant information from competing data or chains of reasoning in drawing or evaluating conclusions. ■ Towards the lower end of the scale (around 400 score points), students are able to recall simple factual scientific knowledge (e.g., names, facts, terminology, simple rules); and to use common scientific knowledge in drawing or evaluating conclusions. A full description of the conceptual framework underlying the PISA assessment of science is provided in The VISA 2003 Assessment Framework: Mathematics, Reading, Science and Problem Solving Kowledge and Skills (OECD, 2003e). The tasks used for the assessment of science in PISA are quite diverse. Figure 6.8 and Figure 6.9 show examples of the science tasks used in PISA 2003, along with a description of the criteria used to mark students' answers. A more complete set of sample tasks can be found at www.pisa.oecd.org. The science assessment was comprised of 35 items divided into 13 units from which 25 items from 10 units were the same as the ones used in 2000 (see Annex A6,Table A6.3 for the breakdown of the items by the various aspects of the framework). 0 OECD 2004 Learning for Tomorrows World - First Results from PISA 2003 The sample unit Daylight provides verbal information on the variation in the length of daylight between the Northern and Southern hemispheres (Figure 6.8). The change of seasons in these hemispheres is also related to the tilt of the earth's axis. The stimulus for the sample unit, Cloning, features an extract from a newspaper article and a photograph of Dolly, the first sheep to be cloned (Figure 6.9).The questions that follow are probing the students' knowledge of the structure of animal cells and scientific methods of investigation. When taken together, these science units help to illustrate the underlying understanding of science that PISA has adopted in its framework as scientific literacy, in particular the ability to use science knowledge to give explanations. STUDENT PERFORMANCE IN SCIENCE The mean performances of countries in science As previously described in Chapter 2 for the case of mathematics and earlier in this Chapter for reading, the average scores of countries provide an indication of the overall level of performance, keeping in mind that mean scores provide an incomplete picture of performance. As with reading, the outcomes for science are based on the science scale that was developed for PISA 2000 and which had a mean of 500 and a standard deviation of 100. Figure 6.10 shows average performance on the science scale (Table 6.6).The PISA 2003 results include 29 OECD countries — the Slovak Republic and Turkey joined PISA in 2003 and the Netherlands met all technical standards in 2003, while the United Kingdom has been excluded from the results as it failed to reach the technical standards required by PISA 2003. When the 25 OECD countries for which comparable data are available for both the PISA 2000 and 2003 assessments are compared jointly, it is clear that the average performance has remained unchanged (Figure 6.10).6 However, mainly because of the inclusion of new countries in 2003, the overall OECD mean for science is now 496 score points and the standard deviation is 105 score points. The gap in performance between the highest and the lowest performing OECD countries is 143 points.That is, while the average scores of the highest performing countries of Finland and Japan is 548 or about half a standard deviation above the OECD average, Mexico's average score of 405 score points is almost one standard deviation below the OECD average. Finland and Japan have the highest mean scores and rank between first and third on the science scale, but their performance is not statistically significantly different from that in Korea and the partner country Hong Kong-China, who both rank between second and fourth. Other OECD countries that show mean performance in science higher than the OECD average are Australia, Belgium, Canada, the Czech Republic, France, Ireland, the Netherlands, New Zealand, Sweden, Switzerland and among the partner countries Liechtenstein and Macao-China. Countries with performance not statistically different from the OECD average are Germany, Hungary, Poland and the Slovak Republic.7 On average, students did as well in science in 2003 as in 2000, but their results were slightly more spread out. Four countries had the highest performance and their averages are indistinguishable. CT1 E c ex "CT 3 C OJ O ex < O OECD 2004 Learning for Tomorrow's World - First Results from PISA 2003 293 u OJ u -CT CTI E c ex "CT 3 C OJ O ex < FícjUre 6.10 ■ Multiple comparisons of mean performance on the science scale Science scale Japan Hong Kong-Chin Li eclitenstein Australia Macao-China Netherlands Czech Republii New Zealand Canada Switzerland France Belgium Ireland Hungary Germany Poland Slovak Republi United States Austria Russian Fed. Spain Italy Norway Luxem bourg Greece Denmark Portugal Uruguay Serbia Turkey Thailand Mexico Indonesia Tunisia Range of rank* U ň -3 'i, ' ■* 1 I I k I I I J I I S I Š 1 I a *& s 548 S48 S39 S38 S2S S2S S2S S24 S23 S21 £19 £13 ill £09 Í06 SOS 503 502 498 495 495 491 491 48 J f 31 h S 3 1 í ď z J 487 486 484 483 Hill iE, (1,9) (4,1) (4,3) (3.5) (4.3) (2.1) (3.0) (3.1) (3.4) (2.4) (2.0) (3.7) (3.0) (2.5) (2.7) (2.7) (2.8) (3.6) (2.9) (3.7) (l.S) (3.1) (3.4) (4.1) (3.9) (2.6) (3.1) (2.9) (l.S) (3.8) (3.0) (3.5) (2.9) (3.5) (5.9) (2.7) (3.5) (3.2) (4.3) (2.6) ink 1 1 2 4 4 4 4 6 7 y y 10 10 11 11 14 li 16 17 16 19 19 20 22 21 2i 26 28 29 ink 2 3 3 7 8 8 8 y 13 13 13 u li 16 17 iy 21 19 2Í 23 24 2i 2i 2i 26 27 27 28 29 ink 1 1 2 2 i i i i i b 8 10 12 u 13 13 14 14 17 18 19 20 19 20 20 22 22 24 26 2i 30 31 33 33 33 34 37 38 38 Í3 ink 3 3 4 4 11 10 10 11 11 11 12 li 16 16 18 18 iy 21 12 2i 23 27 28 30 29 29 30 30 30 31 32 32 3i 36 36 36 37 S3 40 40 * Because data are based on samples, it is not possible to report exact rank order positions for countries. However, it is possible to report the range of rank order positions within which the country mean lies with 95 per cent likelihood. Read across the row for a country to compare performance with the countries listed along the top of the chart. The symbols indicate whether the average performance of the country in the row is lower than that of the comparison country, higher than that of the comparison country, or if there is no statistically significant difference between the average achievement of the two countries. Source: OECD, PISA 2003 database. Without the I Bonferroni I adjustment: | With the Bonferroni adjustment: Mean performance statistically significantly higher than in comparison country No statistically significant difference from comparison country Mean performance statistically significantly lower than in comparison country Mean performance statistically significantly higher than in comparison country No statistically significant difference from comparison country Mean performance statistically significantly lower than in comparison country Statistically significantly above the OECD average Not statistically significantly different from the OECD average Statistically significantly below the OECD average 294 S OECD 2004 Learning for Tomorrows World - First Results from PISA 2003 FícjUre 6.11 ■ Differences in mean scores between PISA 2003 and PISA 2000 on the science scale Only countries with valid data for both 2003 and 2000 I Mean score in PISA 2003 I Mean score in PISA 2000 Perfi 600 ormance on the science scale SSO S00 I I zi^: I, s s .j....,........................T... 4S0 ■■■ " Is 1 I I I i 400 I ■ 3S0 V I Austria------ Norway BB| Mexico HHI Canada ------ I 0 c oooooooooooooo röröröa^ga<2arö22^-: aaa.sfe.2 -a «gag •£>-&) tí tí tí t) 2 tí 5? a* tí z o &, a *• <» S ř a i; 3 u i .2 a £ M N Ě M á 1 < M ?1lí z 1 * a c K + "š :::::: u h3 a u a t) " a S3 Sa a t« .3 3 »s