Abstract
According to the British Dyslexia Association, it is thought that Dyslexia affects around 10% of the population. It is classed as a Specific Learning Difficulty (SpLD) and affects the way that information is processed, stored and retrieved, the results of which include problems with memory, processing speed, time perception, organisation and sequencing.
It’s important to note that dyslexia is a neurological condition occurring independently of intelligence and affecting individuals in a variety of ways to greater or lesser extent. However, significant group differences between dyslexic and non-dyslexic candidates seem to exist on many cognitive assessments which is a cause for concern given the number of organisations using cognitive ability assessments as part of their recruitment process.
In a previous paper at DOP, we showed that differences on cognitive tests used in graduate selection reduced dramatically when time was taken out of the equation, comparing fixed time, 25% extra time, and untimed scenarios.
Could the use of CAT (Computerised Adaptive Testing) help with two issues: lessening the group differences by providing all candidates with the time they require, and allowing dyslexic candidates to be included in general proctored testing sessions, due to the large decreases in test time that CAT administration requires?
Processing speed is often cited as the major reason for the differences between candidates with or without dyslexia. This suggests that the use of timed assessments should be avoided and untimed assessments used. There is still the logistics issue, for example, how to ‘time’ a proctored session where candidates have different time limits or the need to provide rooms for candidates needing extra time.
Our previous study suggests that the extra time provided does lessen the possible disadvantage that timed assessments present for candidates with dyslexia but does not remove the issue. 25% extra time reduced the group score differences between dyslexic and non-dyslexic candidates but this ‘one size fits all’ approach failed to compensate for some individuals whilst overcompensating for others.
From this, we can see that the ‘reasonable adjustment’ does have a real effect and may be sufficient in most cases but not all. Accordingly blanket policies to deal with time (i.e. give an extra 25% of the time)(Bartram & Coyne, 2003) may be insufficient.
The introduction of CAT-based systems (which generally use and should use untimed assessments) would seem to offer the best solution. As CAT systems adapt and modify themselves to the candidate, the need to make a reasonable time adjustment is unnecessary.
Stone and Davy (2011) outline the advantages of CAT in reduced seat time, more tailored assessments, removing the need to adjust time limits and the ability to modify the assessment itself if need be. This makes it a more useful tool in accommodating all candidates.
The greater measurement accuracy of CAT and the ability to reduce test times (while paradoxically having untimed assessments) are likely to increase candidate motivation and reduce test fatigue (Linacre, 2000).
The British Dyslexia Association suggests a range of ways in which candidates with specific learning difficulties may be disadvantaged including:
1. Poor short-term (working) memory.
This makes it difficult for the candidate to hold in mind both the question and possible answers which could result in the candidate needing to re-read the question and answer options a number of times.
2. Organising information in memory.
Dyslexic, and more particularly dyspraxic, candidates may find it difficult to tailor the wide knowledge they may have of a subject to the specific requirements of the question, for example, it may be difficult to distinguish between what is already known and what has just been read.
3. Slow reading.
A person with specific learning difficulties will need more time than other candidates to read through lengthy questions, or passages of text, which may precede multiple choice questions. In cases where the question or case study that they have to read is complex, their difficulties with slow reading will be compounded by their difficulties with working memory, and so they may need to re-read the text several times in order to grasp the detailed meaning.
4. Poor visual tracking / visual processing.
A candidate with these difficulties may find it hard to keep his/her place on a line of text and so a close group of four short answers could be visually confusing. If some answers are differentiated simply by the presence or absence of a single word (such as not), the candidate could easily fail to spot this. The candidate may identify the right answer but mistakenly mark an incorrect box on the answer grid.
5. Visual stress (e.g. Meares-Irlen syndrome).
This may cause the candidate to see:
· white paper as ‘glaring’;
· lines overlapping and jumbling together;
· words/letters shifting about on the page.
Visual stress is greatly exacerbated by fluorescent lighting and can result in repeated loss of place and reduced reading comprehension as well as physical symptoms such as headaches which may have an impact on cognitive performance.
6. Poor layout of papers / inconsistency in phrasing of answers.
The British Dyslexia Association’s Code of Practice includes this advice: Avoid multiple choice questions if possible. These may be discriminatory for dyslexic candidates where there are difficulties in the areas of reading comprehension, working memory and visual tracking. While CAT cannot address these issues in particular, it can create conditions which give dyslexic candidates the time to complete the given assessment. This adds to the previous suggestion that timed assessments are inappropriate.
Many of these issues are related to time limits (even if an extra time adjustment is made) or can be accommodated by appropriate test design. Administration methods such as CAT remove the time but do not address the visual issues mentioned specifically.
Questions
There are two questions that are being addressed in this paper:
· What are the changes in group difference scores between dyslexic and non-dyslexic using CAT versions and
non-CAT versions of some cognitive ability tests?
· What are the test completion times for CAT vs non-CAT for dyslexic candidates?
The general hypothesis for Question 1 is that CAT should reduce the group differences significantly.
The general hypothesis for Question 2 is that test completion times on CAT assessments for dyslexic candidates will be considerably lower than those for non-CAT assessments.
Method
A group of a minimum of 100 dyslexic and 100 non-dyslexic candidates will take up to three different cognitive ability tests commonly used in graduate recruitment and selection, under two different conditions: CAT-based versions of the tests, and non-CAT, timed (reasonable adjustment being applied for dyslexic candidates).
All of the candidates will take the assessments under both conditions so it will be possible to make direct comparisons.
There may be test-retest issues but the time between the CAT and non-CAT versions of the same assessment will be regulated to reduce the learning effect. However the learning effect will impact both dyslexic and non-dyslexic candidates alike so the conditions for both groups are the same. There may be an effect on the test completion times so the non-CAT versions will be taken second although this is likely to decrease the hypothesised 'improvement' in time taken to complete.
References
Alster, E. H. (1997). The effects of extended time on algebra test scores for college students with and without learning disabilities. Journal of Learning Disabilities, 30(2), 222–227.
Bartram, D. & Coyne, I (2003). International Guidelines on Computer-Based and Internet Delivered Testing. The International Test Commission.
Huesman, R. L., & Frisbie, D. A. (2000). The validity of ITBS reading comprehension test scores for learning disabled and non-learning disabled students under extended time conditions. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA.
Linacre, J. M. (2000). Computer-adaptive testing: a methodology whose time has come. MESA Memorandum No. 69.
Published in S. Chae, U. Kang, E. Jeon & J. M. Linacre. Development of computerised middle school achievement test (in Korean). Seoul, South Korea: Komesa Press.
Snowling, M.J. (2008). Specific disorders and broader phenotypes: The case of dyslexia. Quarterly Journal of Experimental Psychology, 61, 142–156.
Stone, E. & Davy, T. (2011). Computer-Adaptive Testing for Students with Disabilities: A Review of the Literature, ETS Research Report RR 11-32
The British Dyslexia Association (2014). Frequently Asked Questions Adults and Employment, retrieved from http://www.bdadyslexia.org.uk/files/Tackling on July 31st 2014.
The Equality Act (2010), retrieved from http://www.legislation.gov.uk/ukpga/2010/15 on July 31st, 2014.
It’s important to note that dyslexia is a neurological condition occurring independently of intelligence and affecting individuals in a variety of ways to greater or lesser extent. However, significant group differences between dyslexic and non-dyslexic candidates seem to exist on many cognitive assessments which is a cause for concern given the number of organisations using cognitive ability assessments as part of their recruitment process.
In a previous paper at DOP, we showed that differences on cognitive tests used in graduate selection reduced dramatically when time was taken out of the equation, comparing fixed time, 25% extra time, and untimed scenarios.
Could the use of CAT (Computerised Adaptive Testing) help with two issues: lessening the group differences by providing all candidates with the time they require, and allowing dyslexic candidates to be included in general proctored testing sessions, due to the large decreases in test time that CAT administration requires?
Processing speed is often cited as the major reason for the differences between candidates with or without dyslexia. This suggests that the use of timed assessments should be avoided and untimed assessments used. There is still the logistics issue, for example, how to ‘time’ a proctored session where candidates have different time limits or the need to provide rooms for candidates needing extra time.
Our previous study suggests that the extra time provided does lessen the possible disadvantage that timed assessments present for candidates with dyslexia but does not remove the issue. 25% extra time reduced the group score differences between dyslexic and non-dyslexic candidates but this ‘one size fits all’ approach failed to compensate for some individuals whilst overcompensating for others.
From this, we can see that the ‘reasonable adjustment’ does have a real effect and may be sufficient in most cases but not all. Accordingly blanket policies to deal with time (i.e. give an extra 25% of the time)(Bartram & Coyne, 2003) may be insufficient.
The introduction of CAT-based systems (which generally use and should use untimed assessments) would seem to offer the best solution. As CAT systems adapt and modify themselves to the candidate, the need to make a reasonable time adjustment is unnecessary.
Stone and Davy (2011) outline the advantages of CAT in reduced seat time, more tailored assessments, removing the need to adjust time limits and the ability to modify the assessment itself if need be. This makes it a more useful tool in accommodating all candidates.
The greater measurement accuracy of CAT and the ability to reduce test times (while paradoxically having untimed assessments) are likely to increase candidate motivation and reduce test fatigue (Linacre, 2000).
The British Dyslexia Association suggests a range of ways in which candidates with specific learning difficulties may be disadvantaged including:
1. Poor short-term (working) memory.
This makes it difficult for the candidate to hold in mind both the question and possible answers which could result in the candidate needing to re-read the question and answer options a number of times.
2. Organising information in memory.
Dyslexic, and more particularly dyspraxic, candidates may find it difficult to tailor the wide knowledge they may have of a subject to the specific requirements of the question, for example, it may be difficult to distinguish between what is already known and what has just been read.
3. Slow reading.
A person with specific learning difficulties will need more time than other candidates to read through lengthy questions, or passages of text, which may precede multiple choice questions. In cases where the question or case study that they have to read is complex, their difficulties with slow reading will be compounded by their difficulties with working memory, and so they may need to re-read the text several times in order to grasp the detailed meaning.
4. Poor visual tracking / visual processing.
A candidate with these difficulties may find it hard to keep his/her place on a line of text and so a close group of four short answers could be visually confusing. If some answers are differentiated simply by the presence or absence of a single word (such as not), the candidate could easily fail to spot this. The candidate may identify the right answer but mistakenly mark an incorrect box on the answer grid.
5. Visual stress (e.g. Meares-Irlen syndrome).
This may cause the candidate to see:
· white paper as ‘glaring’;
· lines overlapping and jumbling together;
· words/letters shifting about on the page.
Visual stress is greatly exacerbated by fluorescent lighting and can result in repeated loss of place and reduced reading comprehension as well as physical symptoms such as headaches which may have an impact on cognitive performance.
6. Poor layout of papers / inconsistency in phrasing of answers.
The British Dyslexia Association’s Code of Practice includes this advice: Avoid multiple choice questions if possible. These may be discriminatory for dyslexic candidates where there are difficulties in the areas of reading comprehension, working memory and visual tracking. While CAT cannot address these issues in particular, it can create conditions which give dyslexic candidates the time to complete the given assessment. This adds to the previous suggestion that timed assessments are inappropriate.
Many of these issues are related to time limits (even if an extra time adjustment is made) or can be accommodated by appropriate test design. Administration methods such as CAT remove the time but do not address the visual issues mentioned specifically.
Questions
There are two questions that are being addressed in this paper:
· What are the changes in group difference scores between dyslexic and non-dyslexic using CAT versions and
non-CAT versions of some cognitive ability tests?
· What are the test completion times for CAT vs non-CAT for dyslexic candidates?
The general hypothesis for Question 1 is that CAT should reduce the group differences significantly.
The general hypothesis for Question 2 is that test completion times on CAT assessments for dyslexic candidates will be considerably lower than those for non-CAT assessments.
Method
A group of a minimum of 100 dyslexic and 100 non-dyslexic candidates will take up to three different cognitive ability tests commonly used in graduate recruitment and selection, under two different conditions: CAT-based versions of the tests, and non-CAT, timed (reasonable adjustment being applied for dyslexic candidates).
All of the candidates will take the assessments under both conditions so it will be possible to make direct comparisons.
There may be test-retest issues but the time between the CAT and non-CAT versions of the same assessment will be regulated to reduce the learning effect. However the learning effect will impact both dyslexic and non-dyslexic candidates alike so the conditions for both groups are the same. There may be an effect on the test completion times so the non-CAT versions will be taken second although this is likely to decrease the hypothesised 'improvement' in time taken to complete.
References
Alster, E. H. (1997). The effects of extended time on algebra test scores for college students with and without learning disabilities. Journal of Learning Disabilities, 30(2), 222–227.
Bartram, D. & Coyne, I (2003). International Guidelines on Computer-Based and Internet Delivered Testing. The International Test Commission.
Huesman, R. L., & Frisbie, D. A. (2000). The validity of ITBS reading comprehension test scores for learning disabled and non-learning disabled students under extended time conditions. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA.
Linacre, J. M. (2000). Computer-adaptive testing: a methodology whose time has come. MESA Memorandum No. 69.
Published in S. Chae, U. Kang, E. Jeon & J. M. Linacre. Development of computerised middle school achievement test (in Korean). Seoul, South Korea: Komesa Press.
Snowling, M.J. (2008). Specific disorders and broader phenotypes: The case of dyslexia. Quarterly Journal of Experimental Psychology, 61, 142–156.
Stone, E. & Davy, T. (2011). Computer-Adaptive Testing for Students with Disabilities: A Review of the Literature, ETS Research Report RR 11-32
The British Dyslexia Association (2014). Frequently Asked Questions Adults and Employment, retrieved from http://www.bdadyslexia.org.uk/files/Tackling on July 31st 2014.
The Equality Act (2010), retrieved from http://www.legislation.gov.uk/ukpga/2010/15 on July 31st, 2014.
| Original language | English |
|---|---|
| Pages | 185 |
| Number of pages | 187 |
| Publication status | Published - Jan 2015 |
| Externally published | Yes |
| Event | The British Psychological Society Division of Occupational Psychology Conference - , United Kingdom Duration: 7 Jan 2015 → 9 Jan 2015 |
Conference
| Conference | The British Psychological Society Division of Occupational Psychology Conference |
|---|---|
| Abbreviated title | BPS DOP |
| Country/Territory | United Kingdom |
| Period | 7/01/15 → 9/01/15 |