16 January 2017
Who watches the watchmen?
"We ... find ourselves continually trying to accommodate new realities within inappropriate existing institutions, and trying to think about those new realities in traditional but sometimes dangerously irrelevant terms."
- Gwynne Dyer
You see, but you do not observe. The distinction is clear.'
-Sherlock Holmes, A Scandal in Bohemia
Question: how relevant and helpful are mathematics lesson observations?
The first question is how reliable lesson observation is as a tool - and perhaps the results will surprise you. Firstly, even 'experts' tend to disagree. According to Professor Rob Coe at Durham University's Centre for Evaluation and Monitoring,
"Using Ofsted's categories, if a lesson is judged 'Outstanding' by one observer, the probability that a second observer would give a different judgement is between 51% and 78%. For observations conducted by Ofsted inspectors or professional colleagues, 'training' in observation is generally not of the quality and scale used in these studies, and no evidence of reliability is available. Hence, we are probably justified in assuming that the true value will be close to the worst case. In other words, if your lesson is judged 'Outstanding', do whatever you can to avoid getting a second opinion: three times out of four you would be downgraded. If your lesson is judged 'Inadequate' there is a 90% chance that a second observer would give a different rating."
The validity of lesson observation as a proxy for measuring student progress is also questionable. One study (Strong et al, 2011) suggests the likelihood of an incorrect judgement by a teacher or headteacher observing - based on only a two-variable measurement - is around 63%.
Putting these astonishing findings to one side for a moment: if we are to take part in lesson observation (and most schools still do), what should be the focus? One particular issue seems to be that lesson observations are almost exclusively channelled through generic formats. Observers might have very little understanding of the mathematics involved (as was the case for me being observed as a Head of Mathematics in a school where only I had experience and subject knowledge enough to teach A-Level), and the form they are using might all but ignore the subject being taught in favour of the teaching methods, the objectives, and the behaviour management being employed. Pieces of paper focus attention. We cannot possibly see everything (and if you know anything about human beings, you won't be shocked by the outcomes of this experiment) - so following a format is important. The quality and focus of that format is therefore decisive.
Is it time we mathematicians demanded subject-specific lesson observation formats? If so, what is out there that will fulfil our needs?
Last month, ACME (the Advisory Committee on Mathematics Education) released, as part of a wider piece on professional development for maths teachers , a mathematics-specific lesson observation form produced by the University of Nottingham. It is simple and clear, with noticeably little focus on anything but the mathematical needs of learners and a catch-all 'Other things noticed' box.
In the US, research at Harvard University led to the MQI (Mathematical Quality of Instruction) measure, which takes the following questions and uses the answers to form a judgement of the mathematics teaching (the Common Core is the mathematics curriculum used in the US):
Common Core-Aligned Student Practices
To what extent are the students, as opposed to the teacher, doing the mathematics of the lesson-engaging in mathematical thinking and reasoning, communicating about mathematics, and solving high-cognitive demand tasks and contextualized problems?
Working with Students and Mathematics
To what extent does the teacher use student mathematical ideas or misconceptions to move the lesson forward?
Richness of the Mathematics
To what extent are teachers and students making sense of the mathematics of the lesson? Are there elements of "why" and not just how? Do the teacher and students attend to precision in their use of mathematical language?
Errors and Imprecision
Is the mathematics of the lesson clear and correct?
These two examples raise the question: just what should we be looking for in a mathematics lesson? Almost all of the other maths-specific observation formats that are widely available contain sections for items like lessons timings, girl-boy numbers, displays, feedback from exercise books, and are noticeably longer and more complex.
Some would argue that part of the problem is the need to see and capture too much information, and that lesson observations should be sharply targeted instead of an attempt to inspect every area of a teacher's practice all in one lesson. One thing that seems clear is that many forms reflect the use of observation for monitoring and not professional development or support for maths teachers.
What would your ideal maths lesson observation form look like?