Abstract
BACKGROUND
The demand for cybersecurity experts has quadrupled to a record high over the last year in light of data breaches at TalkTalk, Sony and Ashley Madison to name but a few. This surge in demand has left the industry actively seeking to make new hires in 2016, with the jobs market expected to be strong for the foreseeable future.
Large organisations across many industries are planning to bolster in-house security teams, as well as calling on specialist contractors for support. Businesses are now more aware than ever of the devastating effects which a hacking incident can have. TalkTalk was hit by a significant and sustained cyber-attack in October 2015, its third in the calendar year, which could cost the telecoms company up to $50 million.
The global cybersecurity market is expected to be worth $170 billion by 2020, and the cyber security market is estimated to grow at a compound annual growth rate of 9.8% from 2015 to 2020. Compounding this further is the global cybersecurity labour epidemic. More than 200,000 U.S. cybersecurity jobs are unfilled. The cybersecurity workforce shortage is expected to reach 1.5 million unfilled positions by 2019. Demand for cybersecurity professionals over the past five years grew 3.5 times faster than demand for other IT jobs and about 12 times faster than for all other jobs.
It is within this environment that the MoD is in competition with the private sector for recruiting and retaining cyber skilled individuals. Having far less flexibility than their private sector counterparts, the MoD is reliant on identifying individuals both capable of acquiring the required skills and motivated to serve their country, thus being more likely to remain in service.
Developing a solution
Research
We conducted job analysis in order to identify what existing skills, abilities, aptitude, and behaviours would be required of an individual suitable for this career. 27 1:1 interviews and 8 focus groups were conducted with a representative sample from across the division. All individuals involved also completed existing cognitive ability and personality assessments.
The sensitive nature of the roles seriously hampered typical job analysis approaches as we were not allowed physical access to the working environment, let alone conduct observations and the people we interviewed were not at liberty to explain in much detail what they did or how they went about doing it. As such, we were almost completely reliant on the use of analogies to get insights into what would be required to successfully perform in the role.
During this time, we identified a paradox between generic concepts of military ‘good’ and the reality of being good within the cyber team, this included working in a rank agnostic and more sedentary environment. To summarise, we identified that the major role requirements for a Cyber Specialist are individuals who are:
• ‘Intellectually bright’ but not necessarily academically qualified. There are significant and surprising amounts of material to learn, which is continuous and builds on previous learning.
• Very resilient due to the confidential and sometimes disturbing nature of materials that may be involved.
• Tenacious and interested/happy working with lots of detail, being willing to probe again and again.
• IT Literate with a good knowledge of how IT systems work. This however was a skill identified as something that could be acquired in role rather than exist at time of commencement.
• Independent and comfortable working in an environment which is based on technical skills and low on structure. They have a lot of latitude to determine what they work on and how they do it.
• Confident dealing with individuals and staff at all levels within an organization.
• Effective and credible communicators capable of flexing various communication styles and making clear recommendations that will enable sometimes less ‘able’ ‘clients’ to make informed decisions.
• Proactive in seeking opportunities to enhance and share knowledge.
Whilst there are already a number of commercial cyber solutions, existing assessments tend to concentrate more on current cyber technical knowledge, ability and/or sills rather than potential aptitude, our findings made it clear that what was required was an assessment tool which could test underlying cognitive ability, personality behaviours and innate aptitude for cyber roles.
Design
The premise for the assessment design is that as the technologies are changing rapidly, it is more appropriate to identify underlying cognitive abilities, skills and behaviours aligned to ‘learning and adapting’ continuously as new technologies and challenges emerge in this fast-paced and challenging environment.
Our final solution was a suite of assessments referred to as the Defence Cyber Aptitude Test (DCAT) that comprises of:
1. Computerised Adaptive Test of Generic Cognitive Ability
Constructs Assessed:
• Logical Reasoning
• Numerical Reasoning
• Verbal Reasoning
Number of questions: Varialbe
Alpha: minimum of 0.815
Estimated completion time: 15 minutes
Rationale: Cognitive ability is the best predictor of performance across jobs. It is also the best predictor of training or knowledge acquisition success.
The Cyber roles require the evaluation of critical information to come to a conclusion:
• Do I pursue this further?
• What’s the best way to make my argument to supervisors or co-workers?
• How do I make sense of the information gathered about these 3 people?
The Cyber roles also require the ability to learn new concepts and skills:
• What’s this new social app and how can I leverage this information?
• How will I do in this class?
• How does this new programming language compare with what I already know?
Note, this assessment was previously created by our team for the British Army to use in recruitment.
2. Fault Finding Assessment
Constructs Assessed:
• Logical Reasoning
• Fault Finding
• Decoding
Alpha: minimum of 0.87
Final number of questions: 10 (From first iteration where 24 items were trialed)
Time Limit: 6 minutes
Rationale: The fault finding test is designed to provide a fair, objective, rapid and practical measure of skills in diagnosing faults in a system. It does this by presenting an input, a series of functions and a subsequent output and requires that candidates identify an faults in the coding system.
A pictorial code was created to ensure no benefits would be given to any individuals with previous coding experience were we to use one already in existence. This will help to identify individuals who have or are capable of attaining the skills required to identify errors in processes as well as the ability to translate a form of code.
3. Checking Assessment
Constructs Assessed:
• Checking Skills
• Accuracy
• Systematic Approach
Alpha: minimum of 0.73
Final number of questions: 13 (From first iteration where 24 items were trialed)
Time Limit: 12 minutes
Rationale: Accuracy is typically assessed under time constraints but this would muddy the construct with an irrelevant requirement. As such, the Bridge Checking Test was designed to provide a fair, objective, rapid and practical measure of skills in applying a systematic approach to identify errors rather than using time as a differentiator.
It does this by presenting a grid of ‘islands’ interconnected with a series of ‘bridges’. Candidates are required to identify if the ‘islands’ have been labelled correctly based on the number of ‘bridges’ connected to them. This will help to indicate if individuals have or are capable of attaining the skills required to accurately analyse data and unearth anomalies.
4. Personality Preference Assessment
Key Personality Traits Assessed:
• Tolerance
• Perseverance
• Team Player
• Sociability
• Dependability
• Conformity
• Self Confidence
• Social Assuredness
• Energy
Number of questions: 82 (From first iteration where 120 items across 13 traits were trialed)
Estimated completion time: 10 minutes
Rationale: The ‘Interplay’ of many of the personality traits ran counter-intuitive and sat outside of typical ‘service’ ethos. For example, individuals with strong preference for persevering with tasks until completion might find it challenging to cease work on or hand over a task with no completion point.
Data analysis & results
Analysis was conducted on a much larger form (with an average seat time of 85 minutes). Norms (n=262) were based on the initial trial data although it was anticipated that maximum performance assessment means would increase, given that the trail population would be less motivated to perform well (no consequences) and more likely to experience test fatigue (average seat times for the first iteration was 85 minutes with some individuals taking nearly twice that).
Re-norming of the final form took place early this year (n=150). Most individuals are now completing the assessment suite in less than 50 minutes (significantly less if not taking the personality preference questionnaire).
The DCAT is still in the nascent stages of its lifecycle and as such a concurrent validation has not yet been conducted. An independent validation exercise is scheduled to be completed by the time of the conference.
Next steps
Given the dynamic nature of this test, we would expect test upgrades to continue for the next few years on an annual or bi-annual basis. It is also entirely possible that the next 12-18 months will see MOD UK looking to refine and hybridise the core tool to include gamification and an adaption of the tool to enable it to be applied to all prospective entry-level applicants to the UK Armed Forces (116,200 applicants from July 2014 to June 2015).
Discussion points
• Conducting robust Job Analysis when you can’t ask someone what they do, how they do it or why
• Assessing for aptitude rather than ability and keeping construct assessment as ‘pure’ as possible
• The benefits of working with a military organization when meeting aggressive timelines
The demand for cybersecurity experts has quadrupled to a record high over the last year in light of data breaches at TalkTalk, Sony and Ashley Madison to name but a few. This surge in demand has left the industry actively seeking to make new hires in 2016, with the jobs market expected to be strong for the foreseeable future.
Large organisations across many industries are planning to bolster in-house security teams, as well as calling on specialist contractors for support. Businesses are now more aware than ever of the devastating effects which a hacking incident can have. TalkTalk was hit by a significant and sustained cyber-attack in October 2015, its third in the calendar year, which could cost the telecoms company up to $50 million.
The global cybersecurity market is expected to be worth $170 billion by 2020, and the cyber security market is estimated to grow at a compound annual growth rate of 9.8% from 2015 to 2020. Compounding this further is the global cybersecurity labour epidemic. More than 200,000 U.S. cybersecurity jobs are unfilled. The cybersecurity workforce shortage is expected to reach 1.5 million unfilled positions by 2019. Demand for cybersecurity professionals over the past five years grew 3.5 times faster than demand for other IT jobs and about 12 times faster than for all other jobs.
It is within this environment that the MoD is in competition with the private sector for recruiting and retaining cyber skilled individuals. Having far less flexibility than their private sector counterparts, the MoD is reliant on identifying individuals both capable of acquiring the required skills and motivated to serve their country, thus being more likely to remain in service.
Developing a solution
Research
We conducted job analysis in order to identify what existing skills, abilities, aptitude, and behaviours would be required of an individual suitable for this career. 27 1:1 interviews and 8 focus groups were conducted with a representative sample from across the division. All individuals involved also completed existing cognitive ability and personality assessments.
The sensitive nature of the roles seriously hampered typical job analysis approaches as we were not allowed physical access to the working environment, let alone conduct observations and the people we interviewed were not at liberty to explain in much detail what they did or how they went about doing it. As such, we were almost completely reliant on the use of analogies to get insights into what would be required to successfully perform in the role.
During this time, we identified a paradox between generic concepts of military ‘good’ and the reality of being good within the cyber team, this included working in a rank agnostic and more sedentary environment. To summarise, we identified that the major role requirements for a Cyber Specialist are individuals who are:
• ‘Intellectually bright’ but not necessarily academically qualified. There are significant and surprising amounts of material to learn, which is continuous and builds on previous learning.
• Very resilient due to the confidential and sometimes disturbing nature of materials that may be involved.
• Tenacious and interested/happy working with lots of detail, being willing to probe again and again.
• IT Literate with a good knowledge of how IT systems work. This however was a skill identified as something that could be acquired in role rather than exist at time of commencement.
• Independent and comfortable working in an environment which is based on technical skills and low on structure. They have a lot of latitude to determine what they work on and how they do it.
• Confident dealing with individuals and staff at all levels within an organization.
• Effective and credible communicators capable of flexing various communication styles and making clear recommendations that will enable sometimes less ‘able’ ‘clients’ to make informed decisions.
• Proactive in seeking opportunities to enhance and share knowledge.
Whilst there are already a number of commercial cyber solutions, existing assessments tend to concentrate more on current cyber technical knowledge, ability and/or sills rather than potential aptitude, our findings made it clear that what was required was an assessment tool which could test underlying cognitive ability, personality behaviours and innate aptitude for cyber roles.
Design
The premise for the assessment design is that as the technologies are changing rapidly, it is more appropriate to identify underlying cognitive abilities, skills and behaviours aligned to ‘learning and adapting’ continuously as new technologies and challenges emerge in this fast-paced and challenging environment.
Our final solution was a suite of assessments referred to as the Defence Cyber Aptitude Test (DCAT) that comprises of:
1. Computerised Adaptive Test of Generic Cognitive Ability
Constructs Assessed:
• Logical Reasoning
• Numerical Reasoning
• Verbal Reasoning
Number of questions: Varialbe
Alpha: minimum of 0.815
Estimated completion time: 15 minutes
Rationale: Cognitive ability is the best predictor of performance across jobs. It is also the best predictor of training or knowledge acquisition success.
The Cyber roles require the evaluation of critical information to come to a conclusion:
• Do I pursue this further?
• What’s the best way to make my argument to supervisors or co-workers?
• How do I make sense of the information gathered about these 3 people?
The Cyber roles also require the ability to learn new concepts and skills:
• What’s this new social app and how can I leverage this information?
• How will I do in this class?
• How does this new programming language compare with what I already know?
Note, this assessment was previously created by our team for the British Army to use in recruitment.
2. Fault Finding Assessment
Constructs Assessed:
• Logical Reasoning
• Fault Finding
• Decoding
Alpha: minimum of 0.87
Final number of questions: 10 (From first iteration where 24 items were trialed)
Time Limit: 6 minutes
Rationale: The fault finding test is designed to provide a fair, objective, rapid and practical measure of skills in diagnosing faults in a system. It does this by presenting an input, a series of functions and a subsequent output and requires that candidates identify an faults in the coding system.
A pictorial code was created to ensure no benefits would be given to any individuals with previous coding experience were we to use one already in existence. This will help to identify individuals who have or are capable of attaining the skills required to identify errors in processes as well as the ability to translate a form of code.
3. Checking Assessment
Constructs Assessed:
• Checking Skills
• Accuracy
• Systematic Approach
Alpha: minimum of 0.73
Final number of questions: 13 (From first iteration where 24 items were trialed)
Time Limit: 12 minutes
Rationale: Accuracy is typically assessed under time constraints but this would muddy the construct with an irrelevant requirement. As such, the Bridge Checking Test was designed to provide a fair, objective, rapid and practical measure of skills in applying a systematic approach to identify errors rather than using time as a differentiator.
It does this by presenting a grid of ‘islands’ interconnected with a series of ‘bridges’. Candidates are required to identify if the ‘islands’ have been labelled correctly based on the number of ‘bridges’ connected to them. This will help to indicate if individuals have or are capable of attaining the skills required to accurately analyse data and unearth anomalies.
4. Personality Preference Assessment
Key Personality Traits Assessed:
• Tolerance
• Perseverance
• Team Player
• Sociability
• Dependability
• Conformity
• Self Confidence
• Social Assuredness
• Energy
Number of questions: 82 (From first iteration where 120 items across 13 traits were trialed)
Estimated completion time: 10 minutes
Rationale: The ‘Interplay’ of many of the personality traits ran counter-intuitive and sat outside of typical ‘service’ ethos. For example, individuals with strong preference for persevering with tasks until completion might find it challenging to cease work on or hand over a task with no completion point.
Data analysis & results
Analysis was conducted on a much larger form (with an average seat time of 85 minutes). Norms (n=262) were based on the initial trial data although it was anticipated that maximum performance assessment means would increase, given that the trail population would be less motivated to perform well (no consequences) and more likely to experience test fatigue (average seat times for the first iteration was 85 minutes with some individuals taking nearly twice that).
Re-norming of the final form took place early this year (n=150). Most individuals are now completing the assessment suite in less than 50 minutes (significantly less if not taking the personality preference questionnaire).
The DCAT is still in the nascent stages of its lifecycle and as such a concurrent validation has not yet been conducted. An independent validation exercise is scheduled to be completed by the time of the conference.
Next steps
Given the dynamic nature of this test, we would expect test upgrades to continue for the next few years on an annual or bi-annual basis. It is also entirely possible that the next 12-18 months will see MOD UK looking to refine and hybridise the core tool to include gamification and an adaption of the tool to enable it to be applied to all prospective entry-level applicants to the UK Armed Forces (116,200 applicants from July 2014 to June 2015).
Discussion points
• Conducting robust Job Analysis when you can’t ask someone what they do, how they do it or why
• Assessing for aptitude rather than ability and keeping construct assessment as ‘pure’ as possible
• The benefits of working with a military organization when meeting aggressive timelines
| Original language | English |
|---|---|
| Pages | 178-182 |
| Number of pages | 5 |
| Publication status | Published - Jan 2017 |
| Externally published | Yes |
| Event | The British Psychological Society Division of Occupational Psychology Conference - , United Kingdom Duration: 4 Jan 2017 → 6 Jan 2017 |
Conference
| Conference | The British Psychological Society Division of Occupational Psychology Conference |
|---|---|
| Abbreviated title | BPS DOP |
| Country/Territory | United Kingdom |
| Period | 4/01/17 → 6/01/17 |