Honourable Gavin Davis MP (DA) is dishonest
This is a brief response to Honourable Gavin Davis’ open letter published in the City Press on 30 December 2016 entitled; “We need to ensure that matric standards are maintained.” Umalusi would like to make it abundantly clear that Honourable Davis is being dishonest and duplicitous in his letter by not providing the real reasons why Dr Mafu Rakometsi, the CEO of Umalusi, “...declined to answer his questions” as he curtly puts it.
In a series of emails between Mr Davis and Dr Rakometsi, which started on 27 December 2016, several reasons were given to him why it was not prudent to engage in an email discussion on a matter as complex and technical as standardisation. Umalusi has therefore decided to divulge the chain of emails between Dr Rakometsi and Mr Davis so that the South African public can make its own informed judgment on the issue. The emails have been attached as part of this response.
Suffice to say that it is Umalusi’s considered view that issues raised by Honourable Davis in his letter, genuine as they may be, also expose his lack of understanding of the intricate process of standardisation. Mr Davis only had a two hour exposure to a ten day process that culminated in the meeting that he attended on 23 December 2016 in Pretoria. He attended the meeting as part of a delegation of the Portfolio Committee on Basic Education, which enjoys observer status in the meeting.
While the final stages of the standardisation process may seem highly statistical, this process of adjustment is the culmination of a long process of receiving and reflecting on qualitative and quantitative inputs. It starts with the setting of papers, then moderation, writing of exams, marking of exams, verification and only then - finally - adjustment of mark distributions.
Given the complex nature of the stages and processes followed, it can lead to misinterpretations especially if one observes any one of the stages in isolation or just the final one. The whole process of standardization is the basis for Umalusi to declare exams fair, valid and credible.
So, Umalusi still maintains the point made in Dr Rakometsi’s email response to Mr Davis that Council is prepared to conduct a workshop for the entire Portfolio Committee on Basic Education based on an invitation from the Chairperson of the committee. Therefore, to avoid creating more confusion to the public, Umalusi will not respond in the media to Mr Davis’ queries which require lengthy and technical responses. Instead, Umalusi will place a discursive article in one of the national publications on 5 January 2017. The article will outline the basic methodology and principles of standardisation in a non-technical parlance.
Umalusi is the Council for the quality assurance of general and further education and training in South Africa. According to Section 17A(4) of Umalusi’s founding Act, the General and Further Education and Training Quality Assurance (GENFETQA) Act, 2001 (as amended in 2008), “the Council may adjust raw marks during the standardisation process.”
Statement issued by Lucky Ditaunyane, Senior Manager: Public Relations and Communications, 1 January 2017
Text of the correspondence:
I
Text of email from DA MP Gavin Davis to Professor John Volmink, Umalusi chairperson, 27 December 2016
From: Gavin Davis
Date: 27 December 2016 at 9:07:11 AM SAST
To: John Volmink
Subject: Issues arising from the standardization meeting held in Pretoria 23 December 2016
Dear Professor Volmink
It was great to chat briefly after the standardisation meeting on Friday.
I am still gathering my thoughts and attempting to make sense of the data presented and the process followed.
The analysis below is based only on what I observed at the meeting and the incomplete information I have at this point on the adjustment process and outcome. My intention is only to satisfy myself that the process followed will achieve a fair and equitable outcome that is in the best interests of current and future learners.
I am going to group my observations into sections to make this correspondence easier to read and respond to. Some specific questions follow at the end.
1. Using the raw marks as a proxy for cognitive demand
As I understand it, Umalusi can only adjust the mark if there is evidence that the exam paper was more or less cognitively demanding than previous years. Indeed, the DBE in its submissions on Friday was at pains to identify particularly difficult questions in some of the papers to justify adjusting the marks upwards.
I noticed at Friday's meeting, however, that the starting point for adjusting the marks was not the papers themselves, but the results. In cases when the raw mark was worse than last year’s, the DBE went back to the paper and found difficult questions to explain the drop in the raw mark. The DBE then motivated for the raw mark to be adjusted upwards accordingly.
This methodology seems incorrect. It would seem that – as a general principle – the cognitive demand of the papers should be assessed independently of the marks. This should preferably happen before the papers are even written so that there is no need for an adjustment process afterwards.
The obvious problem of using the raw marks as proxy for cognitive demand is that there may be cases in which the paper was of the appropriate standard, but the learners (for whatever reason) were below the standard of previous years. Adjusting the marks upwards in such a case would therefore mask systemic problems that need to be addressed.
2. The tendency to adjust upwards, but not downwards
From my observations today it seemed that, in most cases when the raw mark was lower than last year, Umalusi and/or DBE pushed for the final mark to be increased using the computer adjustment. However, when the raw mark was higher than last year, the raw mark was in most cases accepted without any downward adjustment. The net effect was an overall increase in the marks.
I mentioned earlier how, when the raw mark for a subject was low, DBE identified tough questions in the paper to justify an upward adjustment. However I did not observe the same methodology being employed when the raw mark was better than last year’s. In other words, in these cases, there was little interrogation of why the raw mark was better than last year’s, and whether this could have been because the paper was ‘too easy’. Instead, the increased raw mark tended to be accepted as a welcome sign that the system is improving.
If we are adjusting upwards when the raw mark is low, and keeping the raw mark when the raw mark is high, then I fear we may end up artificially inflating the marks. This could result in a situation in which a number of matriculants have passes and bachelor passes on paper but lack the actual cognitive skills to compete in the real world of tertiary study and work.
3. The tendency to adjust marks regardless of whether the circumstances are exceptional
You made the important point at the end of the meeting that the raw mark should be the default, and that adjustments should be made only in exceptional circumstances. You added that, out of 58 subjects, 32 were adjusted and only 26 retained the raw mark. It would be interesting to hear, in the case of each adjustment, what made that case exceptional. Because it seemed to me, as an observer, that some adjustments were made as a matter of routine rather than exception.
4. The impact of progressed learners
According to a reply to a parliamentary question received on 29 November, as many as 109 391 progressed learners wrote the National Senior Certificate this year – up from 65 673 last year.
It follows, therefore, that that there was a significant increase in the number of weaker students who wrote the NSC. This raises the question of whether the inclusion of progressed learners in the standardisation process leads to certain anomalies.
If the raw scores are used as a proxy to determine whether papers were too difficult or too easy (as described in section 1 above), then it is likely that the inclusion of progressed learners will skew the data. This is because, if the raw marks are lower than previous years (when there were no progressed learners), then these low scores could be as a result of the inclusion of weaker students (i.e. progressed learners) and not because the papers were more difficult.
But, as set out earlier, the only legitimate justification for adjusting the raw scores is if the paper was more or less cognitively demanding than previous years. Could it be that the inclusion of progressed learners in the standardisation process creates additional impetus to adjust the marks upwards, for reasons not related to the cognitive demand of the papers?
* * *
In conclusion, I have a number of questions relating to my observations today that I would like to put to you and your team:
1. Am I correct that Umalusi can only adjust the raw mark if there is evidence that the exam paper was more or less cognitively demanding than previous years? Is this protocol formalised in any documents?
2. Is it fair to say that the starting point for the adjustments was not the exam papers themselves but the raw marks?
3. Is any attempt made to assess the cognitive demand of each paper before it is written? If not, why not?
4. How does Umalusi determine the cognitive demand of each paper independently of the raw marks, if at all?
5. How can we satisfy ourselves that the upward adjustment of marks does not mask systemic problems that need to be addressed?
6. What criteria and/or set of variables does the computer use to determine whether or not there should be an adjustment? On what grounds is the recommendation of the computer either accepted or rejected by Umalusi?
7. Is it fair to say that, in most cases, when the raw mark was lower than last year the mark was adjusted upwards but, when the raw mark was higher than last year, the raw mark was in most cases accepted without any downward adjustment?
8. You mentioned that 32 of the 58 subjects were adjusted. Please can Umalusi provide a list of the subjects adjusted and whether each was adjusted upwards or downwards from last year.
9. Given that adjustments can only be effected in exceptional circumstances, please can Umalusi indicate the exceptional circumstance in the case of each adjustment, including the justification for the adjustment in each case.
10. Were progressed learners included in the standardisation process? If so, how many progressed learners were included?
11. Is it fair to say that the inclusion of progressed learners in the standardisation process creates additional impetus to adjust the marks upwards, for reasons not related to the cognitive demand of the papers?
I look forward to Umalusi’s response to these questions, which are posed in the spirit of constructive engagement. Thanks again for your unwavering commitment to improving education in our country.
Kind regards
Gavin
---
Gavin Davis MP
II
Text of email from Dr Mafu Rakometsi, CEO Umalusi to Gavin Davis MP, 27 December 2016:
On Tue, Dec 27, 2016 at 3:40 PM, Dr Mafu Rakometsi wrote:
PRIVATE AND CONFIDENTIAL
Dear Honourable Davis
The email enquiry you sent to Professor John Volmink, Chairperson of Umalusi Council, on 27 December 2016 refers.
I would like to advise that it would be imprudent to attempt to explain the complex standardisation process through the exchange of emails. Arriving at decisions in a standardisation meeting involves a complex set of variables that may not be easy to cogently explain in email conversations. It is for this reason that I wish to propose that you ask the Chairperson of the Portfolio Committee on Basic Education to invite Umalusi to a workshop where the standardisation exercise could be explained and all your questions could be answered. It is my considered view that this will help answer all the important questions you raise.
Sincerely,
Dr Mafu S Rakometsi
CEO: Umalusi
III
Text of letter from Gavin Davis to Dr Mafu Rakometsi, 27 December 2016:
From: Gavin Davis
Sent: 27 December 2016 05:01 PM
To: Dr Mafu Rakometsi
Cc: Prof John Volmink
Subject: Re: FW: Issues arising from the standardization meeting held in Pretoria 23 December 2016
Dear Dr Rakometsi
Thank you very much for your reply to my email addressed to Professor Volmink.
Firstly, the reason I sent my questions in writing is because I would like a response in writing. This will give me the opportunity to study the response and to send follow-ups if the answers are not clear to me.
Secondly, Umalusi will be briefing the media on Thursday on the standardisation process. There is a possibility that I will be asked to comment on the process, and it would be very useful for me to have clear answers to the questions I have raised before I do so.
I therefore reiterate my request for a response, in writing, to the questions contained in the email addressed to Professor Volmink.
I look forward to hearing from you.
Kind regards
Gavin
---
Gavin Davis MP
IV
Text of email from Dr Mafu Rakometsi, CEO Umalusi to Gavin Davis MP, 27 December 2016
---------- Forwarded message ----------
From: Dr Mafu Rakometsi
To: Gavin Davis, Prof John Volmink
Cc:
Date: Tue, 27 Dec 2016 19:41:55 +0000
Subject: RE: FW: Issues arising from the standardization meeting held in Pretoria 23 December 2016
Dear Honourable Davis
Your wish to have all your questions answered in writing is duly noted. However, we are not able to attend to your request at this stage because of the following reasons:
- Umalusi is currently operating with a limited staff complement because of the holidays. The staff that remain at the office have specific and laborious tasks that they need to fulfil for the Council. Any attempt to dedicate staff to give insights into the intricate standardisation decisions could compromise the huge assignment at hand and the stringent deadlines ahead of us.
- I want to reiterate the point I made earlier that we are more likely to do justice to the discussion on standardisation in a workshop environment where all your concerns can be answered tête-à-tête. This will also be for the benefit of other members of the Portfolio Committee on Basic Education.
- Your request to have all your questions answered promptly seems to hinge on the assumption that you may need to deal with media queries in the aftermath of our media briefing on Thursday. My humble view is that this should not be the compelling reason for us to rush what seems to be a very important stakeholder engagement. The media has never expected any observer of the standardisation processes to give a comment on the standardisation decision making processes and outcomes. The observer status that Umalusi gives to different stakeholders is a privileged position that is given to key education stakeholders with strict confidentiality expectations.
- To this end, it would be desirable for the Chairperson of the Portfolio Committee to prioritise the proposed workshop in the new year.
In the meantime, kindly receive the attached document that explains some of the underlying principles and processes of standardisation.
Regards,
Dr Mafu S Rakometsi
CEO: Umalusi
V
Gavin Davis publishes open letter to Dr Rakometsi, 30 December 2016 (see here)
VI
Text of letter from Dr Mafu Rakometsi to Gavin Davis MP, 1 January 2016
---------- Forwarded message ----------
From: Dr Mafu Rakometsi
Date: Sunday, 1 January 2017
Subject: RESPONSE TO YOUR OPEN LETTER TO DR RAKOMETSI
To: Gavin Davis
Dear Honourable Davis
Find appended hereto Umalusi’s response to your open letter.
Season’s greetings
Dr Mafu S Rakometsi
CEO: Umalusi
ENDS
