What AI/DX Adaptive Leader®︎ Systems Thinking shows and does not show is set out on the Certification page. Here we add the research behind it, how the pass mark is set, and the data we will collect.
Overlap with general reasoning ability: what is known so far
For a neighboring construct, complex problem solving (solving tasks while operating a dynamic system), measurement research has accumulated. Within what we have been able to confirm, the following is known.
|
Question
|
What has been confirmed
|
Scale of the source
|
|
Correlation with general intelligence
|
Mean correlation r = .43 (shared variance of roughly 19%). There is overlap, but they are not the same
|
Meta-analysis of 47 studies and 13,740 participants (2015)
|
|
What happens when intelligence is measured broadly
|
With a broad test battery, the correlation rises to about r = .70, and apart from a mathematics achievement test, almost nothing remained that it explained independently beyond intelligence
|
Follow-up study by the same research group (2016)
|
|
Does it predict occupational outcomes
|
Among 245 employees of one manufacturer, it had very weak additional explanatory power for the number of days of training attended (β = .14)
|
A single study (2015)
|
|
Does thinking training transfer beyond the training
|
Effects appear on measures close to the trained tasks. On measures far from them, the effect converged on zero once publication bias and similar factors were controlled
|
Second-order meta-analysis integrating several meta-analyses (2019)
|
We draw one conclusion from this table.
We do not say that AI/DX Adaptive Leader®︎ Systems Thinking measures "a separate ability independent of general reasoning". We say that it measures demonstrated ability, overlapping to a degree with general reasoning, within the knowledge scope we publish and the task formats we define.
This is a modest claim. It is the level of claim we adopt.
For reference: the OECD's PISA 2012 assessment of problem solving (44 countries and economies, about 85,000 students) is a large-scale standardized measurement of a neighboring construct, using simulation tasks run on a computer. We cite it as a precedent in measurement science, not for comparison. It was designed to estimate the level of populations, not to decide whether an individual passes. The fact that this could be measured at scale cannot be used as it stands as grounds for the feasibility of a certification examination for individuals. In the 2015 cycle, the assessment switched to a different construct (collaborative problem solving).
How the pass mark (cut score) is set, and what we will publish
How it is set
-
For each level, we first fix and publish what a candidate must be able to do to pass (Can-Do).
-
Several raters judge, for each question, whether a candidate who meets the Can-Do would answer it correctly, and the pass mark is set by aggregating those judgments.
-
The raters include at least one external person (from the first sitting; the target is 30% of all raters within three years).
-
After each sitting, we check the setting against the actual score distribution.
This procedure has not yet been made into formal rules, and it has never been carried out.
What we will publish
|
What we publish
|
Level of detail
|
|
How the pass mark is set
|
The full procedure
|
|
Grounds for the setting
|
The kinds of material used in the judgment, and the aggregated result
|
|
Number of raters, and the names and affiliations of external raters
|
With each person's consent
|
|
Statistics for each sitting
|
Number of candidates, number passing, and pass rate (with the year and definitions)
|
|
Individual questions and answers
|
Not published, to protect the security of the examination
|
We will publish the pass rate, but we do not use it as a target. Setting a pass rate as a target creates an incentive to adjust the questions or the scoring to meet it. The figures we publish are results, not aims.
Data we will collect, and when we will publish it
|
What we collect
|
What it tells us
|
When we plan to publish
|
|
Correct-answer rate and discrimination for each question
|
Whether the questions are working
|
Together with the results of the first sitting
|
|
Distribution of levels and scores
|
Whether the pass mark was appropriate
|
Same as above
|
|
Results when the same candidate sits again
|
Stability of the measurement
|
Once the second and later sittings are available
|
|
Before-and-after comparison in organizations that adopt it
|
Whether the relationship with job performance can be answered
|
When the data is available. There is no guarantee that it will be
|
If data emerges that can answer the relationship with job performance, we will publish the result even if it is unfavorable to us.