Chapter 3

Reporting a National Assessment: Other Instruments to Communicate Findings

In many cases, the only report issued following a national assessment is the type described in chapter 2. Because potential users of assessment data are numerous and different audiences have different requirements, this chapter describes additional ways of reporting findings that are tailored to meet a variety of needs. These methods range from publishing lengthy detailed reports to issuing short press releases and conducting individual briefings that summarize the main findings of an assessment. Although curriculum developers and textbook writers will require detailed information on student learning (more detailed perhaps than is available in a main report), a set of summary findings or highlights should be presented to citizens, who may have limited statistical acumen.

Pérez (2006) has identified the following general principles regarding the communication of research findings that can be considered applicable to national assessment findings:

– Use simple language, preferably in attractive media products such as videos.

– Clearly identify stakeholders, and tailor events and products to their needs.

– Consider recruiting public and credible leaders as advocates.

– Disseminate information to mass media. Events should be well advertised.

– Use slogans and simple messages that are readily understood. For example, a statement such as “an eight-year-old child should be able to read a 60-word story in one minute and answer three questions about its content” illustrates what a standard means.

– Back up all broadcast or large-audience dissemination materials (including PowerPoint presentations) with supporting technical information.

The procedures and instruments that a national assessment team may use in addition to the main report are described in this chapter and include the following: briefing ministerial and senior policy personnel; preparing briefing notes; publishing summary reports, technical reports, and thematic reports; securing media reports; issuing press releases; holding press conferences; conducting individual briefings; posting Web site reports; and making assessment data available.

Product Specification Sheet

The manager of a national assessment will find preparing a product specifi cation sheet useful in planning the preparation of reports. The sheet should include the following elements:

– Product name (for example, main report)

– Summary description of the product

– Intended users

– Product priority

– Detailed product description

– Key production activities, responsibilities, and time schedule

– Production costs

– Projected release date (and venue if appropriate)

– Product price

– Dependence on other products or inputs.

Because the potential number of products and services will almost inevitably outstrip the technical and financial resources available to the national assessment team, products should be prioritized so that those with the greatest potential for instituting reform will be produced. The national assessment team should plan and cost the full spectrum of products and services, however, to allow the team to move rapidly if additional funding becomes available.

Briefing Ministers and Senior Policy Personnel

National assessment teams should prepare briefings (most likely in written form) for the minister and senior officials in the ministry of education. They will want information that captures the main findings of an assessment in a concise form and the possible implications of those findings (Beaton and Johnson 1992). Because ministers tend to get numerous documents to read on a daily basis, a briefing note must be short and to the point.

Ministers and their senior officials are rarely interested in reading full reports, but they do need to be aware of key findings and to be alerted to issues that the media, parliament, or stakeholders in the education system (for example, teachers’ unions) may raise when the report of an assessment is published. They need this information even if the news is “bad.” For some audiences “good” news is not always welcome because it can imply that additional resources are not needed.

Particular attention needs to be paid to how differential outcomes for subpopulations are reported and interpreted. If some groups perform poorly, it can be politically embarrassing for politicians because the results can be interpreted as evidence of neglect of those segments of the population. It may even be a reason for not holding a national assessment at all or, if one is held, not publicizing the results.

Many government ministries have standard formats for ministerial briefings. These formats should be used unless they are considered inadequate for the task. Effective briefing notes on a national assessment might include the following:

– Concisely stated purpose for carrying out the assessment

– One or two sentences on why it is important for the minister to know the results

– Brief description of the background of the assessment (such as who did it and what target population was assessed)

– Key results, especially those that might have policy implications (such as student achievement levels, regional and gender differences, resource allocation to schools)

– Possible next steps that might present options, such as discussing whether the minister should make a public statement about some of the results, recommending study of the results by a curriculum authority or by teacher training institutions, or suggesting a further national assessment in a different curriculum area

– Recommendations to advise the minister on a response to be taken to the listed options

– Information on whether attachments are included, such as a press release, a summary of the national assessment, or the complete report

– Name of the national assessment team member to be contacted in case the minister requires additional information.

Publishing Summary Reports

A summary report for nontechnical readerships is often published in addition to a main report. The summary report can be very brief, as is the case of the state-level summaries or snapshots provided by the U.S. National Assessment of Educational Progress (NAEP) on its Web site. Box 3.1 presents the summary report for the performance of fourthgrade students in Illinois on the 2007 reading assessment.

Some summary reports may be longer. For example, following a national assessment of geography at grades 4, 8, and 12, the National Center for Education Statistics produced a summary of the results in the NAEPfacts series (approximately six pages), under the title “Geography: What Do Students Know and What Can They Do?” The report described what students at the 25th, 50th, and 90th percentiles at each grade level had mastered (Vanneman 1996). Other reports are lengthier and may include a brief description of all the components of the main report.

Main and summary reports are available for the national assessment of English reading in Ireland (Eivers and others 2005a, 2005b). Both publications can be accessed on the Educational Research Centre Web site (http://www.erc.ie). The main report of the U.S. NAEP for history (grades 4, 8, and 12) and a summary report can be downloaded from http://nces.ed.gov/nationsreportcard/ushistory/.

Classroom teachers are often the primary readership for summary reports. In that case, summary reports are likely to include recommendations for teaching that arise from the assessment (see chapter 6). For example, a teacher’s guide to the mathematics achievements of Irish 15-year-olds was published following the administration of the Programme for International Student Assessment (PISA) test in Ireland in 2003 (Shiel and others 2007). The report contains examples of test items along with information on student performance and on the extent to which Irish teachers teach PISA-type mathematics. Box 3.2 contains three of a number of recommendations for teachers that are in the report.

Summary reports may focus on other interest groups, as follows:

– Teachers’ unions represent the collective interests of teachers and can be powerful agents for – or against – change. They often have a strong vested interest in using information to support their positions.

– Community leaders, including local politicians, need information to determine if the education system is producing what the community needs to achieve social, cultural, and economic goals.

– Employers and business leaders need objective information about student learning achievements as an indicator of the quality of preparation of future employees.

– Citizens may use information from a national assessment to judge whether the education system is meeting its goals with reference to access, quality, efficiency, and equity, which may lead to pressure to improve education provision.

– Donor agencies seek objective information to evaluate program effectiveness or to justify assistance to the education sector or to a particular population subgroup or geographic region.

Publishing Technical Reports

Technical reports are a crucial element of a national assessment because they provide members of the research and scientific communities with detailed information about the assessment that allows them to evaluate it critically. Technical reports also act as a record of the activities involved in the assessment, which is needed to implement future cycles of an assessment.

Some national assessments publish only one report, which serves as both a general and a technical report. The Australian report for students’ achievement in information and communication technologies in years 6 to 10 is an example; it contains technical details on proficiency levels and sampling procedures (Ainley, Fraillon, and Freeman 2007).

Other reports focus on the more technical aspects of sampling, item analysis, scoring techniques and criteria, scaling, statistical analyses, and quality control. Examples can be found in The Trends in International Mathematics and Science Study 2003: Technical Report (Martin, Mullis, and Chrostowski 2004) and in Reading Literacy in the United States: Technical Report of the U.S. Component of the IEA Reading Literacy Study (Binkley and Rust 1994).

Table 3.1 lists activities that a technical report should cover. Particular attention should be paid to instrument development, a description of the population or sample that was assessed, item scaling, and statistical analysis.

Publishing Thematic Reports

Thematic reports explore aspects of the findings of an assessment related to a specific theme that are not addressed in detail in the main report. A thematic report could analyze error patterns in students’ responses to particular aspects of the curriculum or to sets of items in an achievement test. Such analyses can help identify where a curriculum needs to be reformed or instruction needs to be strengthened. For example, an analysis of error patterns in 2006 PISA science items was conducted for Qatari students (DataAngel Policy Research 2007). Despite its potential to improve practice, very little analysis of this type is undertaken.

A thematic report can focus on a subpopulation that is of interest to a particular audience or that relates to a particular policy (for example, boys and girls, race or ethnicity groups, students in disadvantaged backgrounds, rural students). Figure 3.1 provides an example that compares the reading achievement scores of students, who are classified by race or ethnicity on the basis of national assessment data in the United States.

Examples of thematic reports that incorporate advanced statistical methods and present results in an accessible, policy-relevant way can be found in studies using data from the Southern and Eastern Africa Consortium for Monitoring Educational Quality, or SACMEQ, and the Programme d’Analyse des Systèmes Educatifs de la CONFEMEN (Conférence des Ministres de l’Education des Pays Ayant le Français en Partage), or PASEC, on the cost-effectiveness of school inputs (Michaelowa and Wechtler 2006) and rural-urban literacy differences (Zhang 2006) and studies using PISA data on 15-year-olds’ “engagement in reading” (Kirsch and others 2002).

Securing Media Reports

Paper-based products in the form of reports are expensive to produce. Besides, this kind of publication may not be appropriate for many individuals who might have an interest in the findings of a national assessment. In contrast, the media— print, radio, television, or videos – can provide an inexpensive way of disseminating the main messages of a national assessment to individuals who probably would not otherwise learn about them. On the one hand, failure to get media coverage may mean that an assessment goes largely unnoticed. On the other hand, when the media are critical of an assessment – or create sensational or inaccurate messages – the national assessment system may be threatened.

Radio can reach large numbers of individuals and may be particularly appropriate for individuals with low literacy skills. Television, too, can reach large numbers and has an important role in arousing public interest. However, radio and television presentations are usually too superfi cial and short to provide any real understanding of the findings of a research study or of their implications.

A more in-depth use of television to inform the public of the results of an assessment was achieved in Latin America. Following the release of the results of an international assessment carried out by the Laboratorio Latinoamericano de Evaluación de la Calidad de la Educación (see volume 1, appendix C3, of this series – Greaney and Kellaghan 2008), details were publicized through a video shown on television throughout the continent (Ferrer and Arregui 2003). The case for using videos and television to report the findings of a national or international assessment is strengthened by the finding in Peru that videos were much more effective than lectures or PowerPoint presentations in dialogue with stakeholders on educational policy (Pérez 2006).

Source: U.S. National Center for Education Statistics 2006b: figure 2.17. Note: Results are based on the national NAEP sample. Black includes African American; Hispanic includes Latino; and Pacific islander includes Native Hawaiian. Race categories exclude Hispanic origin. * indicates that the score is signifi cantly different from that of American Indian and Alaska Native students.

Issuing Press Releases

A press release is a short written statement issued to the media. The format and content will vary depending on who issues it. A press release from the ministry of education will tend to highlight the positive aspects of findings, whereas one issued by a research body will tend to take a more neutral stance. At the outset, the person responsible for drafting the press release should clarify the type of audience being targeted. Audiences can include the general public, government officials, or experts. Knowing the audience will help determine the amount of technical information to include and the tone of the press release.

Preparing a press release helps reduce, but does not eliminate, the tendency of reporters to oversimplify assessment findings. Some reporters may strive to highlight a politically damaging finding instead of presenting a neutral, balanced view.

A press release should start with the date of release and the name and address of the agency (ministry of education, research institute, or other agency) responsible for the release. If possible, the agency’s logo should appear in the heading. The heading should be presented in boldface type and should be short and interesting; it may be the only chance of attracting the reader’s attention. “National Assessment Report Released” is short, but “New Report Highlights Successes in Education Sector” is both short and interesting. An initial or lead sentence to arouse the reader’s interest, followed by one or two sentences that elaborate on the lead, should be written.

The main section of a press release should be factual and should contain brief answers to the following questions:

– Who carried out the national assessment?

– Why was it carried out?

– When was it carried out?

– How was it carried it?

– What were the main findings?

– Why are they important?

Drafters of a press release should do the following:

– Have a clear idea of what they expect readers to conclude from the press release

– Confine themselves to the facts and not embellish the findings

– Avoid long sentences, technical terms, and statistical jargon

– Use active-voice verbs

– Phrase the text as they would like it to appear in the newspaper

– Limit the release to one, or at most two, double-spaced pages

– Check to see that the text is factually and grammatically correct and free from spelling errors

– Provide a contact phone number, as well as postal and e-mail addresses

– Use ### under the last line of text to signify the end of the press release.

If diagrams are used, they should be easily understood, have a clear purpose, and not attempt to present too much detail or use footnotes.

Graphs in the form of bars, rather than lines, are considered more effective (Hambleton and Slater 1997).

Box 3.3 presents an excerpt from the two-page press release on the 2007 national assessment in the United States.