Turning the tables

Tags: | | | .

School league tables in England have been amended again by the Department for Education. The changes are welcome and will help parents, but the DfE could still do better

Tomorrow the new school league tables are published, with the usual blitz of interest in the rise and fall of individual schools. The arguments for and against the publication of these tables are now so familiar as to excite little interest.

But this year there is a significant change in the content of the tables.  For the first time, GCSE results for each school will be reported for groups of pupils within the school, groups defined by their Key Stage 2 (KS2) scores. Specifically, for each school the tables will report the percentage of pupils attaining at least five A*–C grades (including English and Maths) separately for low-attaining pupils, high attaining pupils and a middle group.  This change has potentially far-reaching implications.

It is a change for the better, and one that we have proposed and supported elsewhere.  Why? In order to support parents choosing a school, league tables need to be functional, relevant and comprehensible. The last of these is straightforward (though not all league table measures in the past have been comprehensible: Contextualised Value-Added (CVA) being the perfect example). ‘Relevant’ means that a measure has some relevance to the family’s specific child. A simple school average, such as the standard whole-cohort percentage attaining five A* to C, is not very informative about how one specific pupil is likely to get on there. By ‘functional’ we mean a measure that does actually help a family to predict the likely GCSE attainment of their child in different schools. If a measure is not functional it should not be published at all.

The new group-specific component is comprehensible and is more relevant than the whole-cohort %5 A* to C measure.  In our analysis of functionality, we show that it is as good as the standard measure, and much better than CVA.

It also addresses in a very straightforward way the critique of the standard league tables that they simply reflect the ability of the intake into schools, and not the effectiveness of the school.  By reporting the attainment of specific groups of students of given ability, this measure automatically corrects for prior attainment, and in a very transparent way. This is therefore much more informative to parents about the likely outcome for their own children than a simple average.  This, of course, is what value-added measures are meant to do, but they have never really become popular, and as we show they are not very functional.

However, the details of the new measure now published are problematic in one way. The choice of groups is important. We defined groups by quite narrow ten percentile bands, the low-attaining group lying between the 20th and 30th percentiles in the KS2 distribution, the high-attaining group between the 70th and 80th percentiles, and the middle group between the 45th to 55th percentiles. While clearly there is still variation in student ability within each band, it is second order and the main differences between schools in performance for any group will come from variation in schools’ teaching effectiveness.

However, the DfE has chosen much broader bands, and have defined the groups so that they cover the entire pupil population: the low-attaining group are students below the expected level (Level 4) in the KS2 tests; the middle-attaining are those at the expected level, and the high-attaining group comprises students above the expected level.

This has one significant disadvantage, set out in detail by Rebecca Allen here. The middle group contains around 45% of all pupils, and so there is very significant variation in average ability within that group across schools. This, in turn, means that differences in league table performance between schools will reflect differences in intake as well as effectiveness, even within the group, thus partly undermining the aim of group-specific reports.

We can speculate as to why the DfE chose to have much broader groups. There may be statistical reasons, pragmatic reasons or what can be termed ‘look and feel’ reasons. Using narrow KS2 bands will correctly identify the effectiveness of the school, but will almost always be averaging over a small number of students. So the estimates will tend to be ‘noisy’, and induce more variation from year to year than averaging over bigger groups. The trade-off here is then between a noisy measure of something very useful against a more stable measure of something less useful. Our original measure was intended to balance these, the DfE have gone all the way to the latter.

A pragmatic reason is that some schools may not have any pupils in a particular narrow percentile band of the KS2 distribution. The narrower the band the more likely this is to be true. This would mean either null entries in the league tables, which might be confusing, or some complex statistical imputation procedure, which might be more confusing. The broad groups that cover the entire pupil population are likely to have very few null entries. Finally, the broad groups feel more ‘inclusive’, they report the performance of all of a school’s students. This is a red herring – the point of the tables is to inform parents in choosing a school, not to generate warm glows.

The new measures hold out the promise of improvements in two areas: choices by parents and behaviour by schools. Parents will have better information on the likely academic attainment of their child in a range of schools. Second, parents will be able to see more directly whether school choice actually matters a great deal for them: whether there are worthwhile differences in attainment within the ability group of their child.

The key point for schools is that performance measures have consequences for behaviour. If this new measure is widely used, it will give schools more of an incentive to focus across the ability distribution. It is still the %5 A* – C measure that is the focus of attention for each group, but now schools will have to pay attention to improving this metric for high and low ability groups as well as simply the marginal children with the highest chance of getting that crucial fifth C grade.

If one believes that gaming and focusing of resources within schools is a very big deal (and there is little quantitative evidence either way) then the new measures could have a major impact on such behaviour. Even if such resource focusing is second order, performance measures send signals on what is valued. These new league table measures will explicitly draw widespread media and public attention to the performance of low- and high-ability children in every school in England.

Rebecca Allen is senior lecturer in the economics of education at the Institute of Education and Simon Burgess is professor of economics and director of the Centre for Market and Public Organisation. This post first appeared on the CMPO blog

3 comments on Turning the tables

  1. Howard Clark says:

    ‘Tweaking’ league tables is what I liken to the smoking trap. It goes like this …

    Smokers smoke because it gives them a ‘hit’. The ‘hit’ stops the craving for a while then it demands another ‘hit’. Stopping smoking leaves you with the craving and desire to continue. You are in a trap.

    Many smokers think that the way out of the trap is ‘cold turkey’ or nicotine replacement tools (patches, gum, inhalers, drops, hypnotherapy). Most of these fail because the belief that we need ‘replacements’ is actually because we are still in the trap. Our heads think we still need this stuff. The only way out of the trap is to understand the trap and the causes for the trap. Then you find that the belief that you will have cravings go quickly.

    League tables (and targets and inspection against ‘standards’ are part of the same fallacy). They are based upon a number of plausible (but flawed) assumptions about the nature of control and improvement. These beliefs have led us to create policy tools that cause dysfunction.

    So we use targets and league tables and inspection against ‘standards’ because we believe that they improve education in schools and give parents choice. But what we have seen over the past decade is as exam scores increase are increased reports of gaming, stress, staff turnover, teaching to the test. Or finding easier exams, or getting tips from the exam boards (hoping for custom because ‘choice’ & ‘competition’ is ‘cheaper’). These have had harmful effects upon children’s learning. Workplaces have discovered that many young adults have qualifications but can’t spell, add-up, reason. Universities find that students with good qualifications can’t argue, reason, spell and expect to be taught only things that pass the degree exam. Students described as having had ‘all of the value sucked out of them’ without the intellectual curiosity you might expect. And last year the UK had slipped down the PISA tests.

    These signals have been treated as problems with teachers rather than problems with the control mechanisms. Inspection is being toughened-up. So now inspectors turn-up hoping to catch teachers and schools out. Teachers must be smart to address ‘slipping standards.’ Satisfactory is no longer good enough (as though you could abolish the statistical median and average). Teachers can be sacked in 6 weeks for teaching methods created to respond to the targets and league tables. We blame teachers when they spend time carrying out mock inspections to get their schools past inspection or see their careers damaged. League tables are tweaked, more comparisons added. More ‘incentives’ thrown into the system. Each time we expect a different response.

    We question the design of the tool instead of questioning the thinking behind the system. We are still caught in education’s version of the smoking trap. The harder we push, the more dysfunction we drive-in.

    …. Another cigarette anyone? … (not for me, I gave up a while back) …

  2. John Faulkner says:

    The Contextual Value Added figures were always there to see how well a school had performed, but the press always conecntrated on the raw data as that made a better story. Lets see if things improve.

  3. Howard Clark says:

    Or as John (above comment) suggests, tweak the tables and wait another 5 years.

    Here’s some food for thought from 2 things I read this week. You could try Tom Bennett’s take on CVA where he describes CVA as not ‘useful’ and ‘it’s corrosive, and actively damages education’. And he also highlights that CVA’s replacement ‘Value Added’ still ‘retains the intrinsic problems shared by CVA.’ He goes into more detail with a real flare for good writing.

    http://behaviourguru.blogspot.com/2012/01/bones-have-spoken-is-value-added.html

    Or indeed the equally impressive account of Pasi Sahlberg who describes how Finnish education is cheap, quite impressive and doesn’t have targets, league tables or indeed inspection against standards. In fact he suggests that Finnish success is in large part due to their absence.

    http://www.theatlantic.com/national/archive/2011/12/what-americans-keep-ignoring-about-finlands-school-success/250564/

Comments

Your email address will not be published. Required fields are marked *

*