I recently saw some fascinating data. Students from different schools sat two identical tests. One test was marked by a human and one test was marked by a computer. Performance in each test was then correlated i.e. did both tests rank students in the same order? The answer was, as is so often the case in education, it depends. For some schools the correlation between the two tests was high i.e. both tests ordered the students in the same way. For other schools the correlation was low i.e. the tests ranked students in different ways. Now, there’s a few reasons for why this could be, but I want to explore one possible cause: accuracy of marking. You see, we often stop when we think we have an accurate assessment, but that is only the beginning. The conditions of the assessment and how it is marked are just as important and because marking is hard, teachers need support and training to mark better.
How then can we improve our ability to mark a test? Over the last few years I have worked with many teachers to find ways to improve marking accuracy – let’s call this standardisation. Below is a standardisation protocol that works well and could be used in your school or department to help make judgements accurate and consistent.
Start with making a dummy paper and a marking commentary
The process starts with an ‘expert’ writing a dummy paper. This is essentially answering the test that students sat with mocked up answers. The best dummy papers are written in such a way that they represent common student answers, misconceptions and errors as well as modelling some exemplary student responses. It’s a good idea to have the exam board marking guidance with you when you write the dummy paper, so you can make some deliberate mistakes such as:
- phonetic spelling e.g fizzics
- formulas instead of names e.g. HCl not hydrochloric acid
- marking of lists e.g. cats have four legs and dogs have three legs scores 0
- crossed out answers that are not replaced are still marked
- using ‘it’ without referring to what ‘it’ is e.g. it has chlorophyll may score 0
- penalising an error only once e.g. 1 sheep + 3 sheep = 5 sheep (wrong); 5 sheep have 20 legs (right) so one of two marks awarded
Alongside creating your dummy paper you should also create the marking commentary. This is the expert’s view of how many marks each question is worth and the rationale as to why marks are, or are not awarded. Click here for an example of a marking commentary.
First teachers mark – what do YOU think?
Once students have sat the actual test, but before marking has begun, arrange a time to get the department together. Hand out your dummy paper along with a mark scheme and Post-it and ask teachers to mark the paper in silence. Exam conditions!
It’s important you don’t allow teachers to discuss at this stage. This can be a scary exercise so you need to make sure that you set the right tone but it’s crucial that each teacher assigns a mark to the paper. Once the paper has been marked, ask teachers to write just the total mark on a Post-it. Now collect in the Post-its and arrange these on the board in order of increasing marks.
The great debate – discussing marks awarded and agreeing a consensus
Hopefully, you will have now exposed a mark range. This makes the point that marking is hard and so standardisation is necessary. Teachers are often amazed, even in subjects such as science and maths, at how awarded marks for the same answers can differ. Now working in groups of three on separate tables, teachers must reach a consensus mark for the paper. Groups should spend time reviewing each question together, debate why they awarded each mark and then come up with an agreed ‘table mark’. It can be helpful here to provide an additional copy of the dummy paper along with general exam board guidance on marking.
The big reveal – do we agree with the expert?
Each table now shares their agreed mark with the wider department. You could write these group marks on the board. Now hand out the marking commentary on a piece of paper to each person. This is the experts view that lists the marks awarded for each question and the all important explanation. Give time for some silent reflection before opening up the table discussions. Stand back and watch the debate flare as the expert gets some ‘feedback’!
Reflection and resolution
Once teachers have had a chance to review the expert’s marking commentary, you can then take feedback to agree the final, group consensus. Undoubtedly the expert will have made some of their own judgement errors. This can be a tricky conversation to manage, so you may prefer groups to simply edit the marking commentary and hand this back in. What is important is that you allow the groups and individuals to reflect on their own marking, where were they generous and why? What new knowledge did they get about awarding marks? How has their own subject knowledge developed? Encourage teachers to think about how this new knowledge is going to ensure consistency when they mark their own papers.
So why is this a powerful standardisation approach?
I think the reason this process works is because it gives teachers feedback on their own marking and provides a strong case for why standardisation is important. Secondly, it encourages a professional dialogue between colleagues around student work, subject knowledge and mark schemes – with lots of opportunity for discussion and construction. Thirdly, this can be quite a fun and motivating exercise.
A final word of warning. Teachers are incredibly passionate about what they do and how marks are awarded. If not managed carefully, debate can snowball out of control. The key is to structure the session so that differences in opinion can be captured in tables and not managed as a whole group.
At the end of the day mark schemes are judgement calls. The consensus is not a truth, it’s just an attempt to help teachers mark more fairly so that our students are all given an equal opportunity to succeed.