Blog

IRT (Item Response Theory) enables fair measurement, something that was not possible with conventional classical test theory, and is becoming increasingly popular in the fields of qualification exams and computer-based testing. A distinctive feature of IRT is that it makes it possible to compare Candidate' abilities on the same scale even when different question sets are used.

In this article, we will explain the differences between IRT and classical test theory, the benefits of IRT, points to note when implementing it, and provide examples of tests in which IRT is used.

So, what are the differences between IRT and classical test theory? There are three main differences:

IRT and classical test theory have fundamentally different approaches to measuring ability.

Classical test theory is a simple method of calculating scores by dividing the number of correct answers by the total number of questions, and does not take into account the quality of the questions. All questions are treated as having the same weight, so answering a difficult question correctly has the same impact on the score as answering an easy question correctly.

On the other hand, IRT quantifies the characteristics of each question (difficulty and discrimination ability) and estimates ability scores based on which questions are answered correctly. Even if two Candidate who answered more difficult questions correctly is considered to have a higher ability score. For example, if one of two Candidate answered many difficult questions correctly and the other answered many easy questions correctly, the former would have a higher ability score.

Furthermore, IRT allows for more precise assessments by taking into account that standard errors of measurement vary across ability levels.

There are also differences between IRT and classical test theory in how questions are assessed.

In classical test theory, the quality of a question is judged by the "correct answer rate" and "correlation with score," so questions with too high a correct answer rate are rated as too easy, and questions with too low a correct answer rate are rated as too difficult. Furthermore, questions with a low correlation with the total score are judged to not be able to adequately measure the ability of Candidate.

IRT uses the Item Characteristic Curve (ICC) to visualize the rate of correct answers at each ability level. It is characterized by its ability to statistically detect cases where questions have low discriminatory power, making it easier to identify low-quality questions. Questions with low discriminatory power are targets for improvement or elimination, as they can reduce the measurement accuracy of the entire test.

By utilizing IRT, quality control of item banks (question banks) can be carried out scientifically, making it possible to maintain test quality over the long term.

IRT and classical test theory also differ in their approaches to test design.

In classical test theory, everyone must solve the same questions, making it difficult to adjust the difficulty level. To ensure the fairness of the test, the same question set must be given to all Candidate. However, this method increases the risk of questions being leaked, and new question sets must be created from scratch when retaking the test.

On the other hand, IRT makes it easy to create multiple forms because it can measure ability scores on a common scale even with different question sets. It also makes it possible to implement CAT (Computerized Adaptive Testing), which changes the questions presented according to Candidate 's ability. CAT allows for efficient measurement because the next question is selected in real time based on Candidate 's response status.

Maintaining fairness even when different questions are asked during retakes is a major advantage in administering exams.

One of the benefits of introducing IRT is that it enables fair and precise measurements.

Even if Candidate solve different problem sets, their ability scores can be compared on the same scale, allowing for fair comparisons between Candidate even if the test dates or Test Center are different.

Furthermore, the evaluation takes into account the difficulty of the questions, allowing true ability to be measured that cannot be seen simply by the number of correct answers. Candidate answer difficult questions correctly are given a high ability score, while Candidate can only answer easy questions are given a low ability score. Another benefit is that the reliability of the pass/fail judgment can be numerically shown because measurement error can be calculated statistically.

By introducing IRT, quality control of item banks (question banks) can be carried out scientifically.

Because the characteristics of each question are quantified, it is possible to see at a glance which questions are high quality and which need improvement, and this can be used as feedback to question creators.In addition, when adding new questions, it is possible to adjust the difficulty level compared to existing questions.Based on the data accumulated in the item bank, it is possible to estimate the difficulty of new questions in advance and create questions that take into account the balance with existing questions.

Furthermore, data accumulated over a long period of use allows for early detection of problems such as deterioration of questions or deficiencies. The introduction of IRT will ensure both confidentiality of questions and test quality, and will also strengthen measures against fraud.

One of the benefits of IRT is that it makes it possible to realize CAT.

By optimizing the questions presented in real time according to Candidate 's answer patterns and selecting questions that best suit Candidate 's ability level, it becomes possible to measure efficiently, as it is no longer necessary for everyone to answer the same number of questions.By allowing Candidate Candidate high and low abilities to answer questions that suit their respective ability levels, it is possible to achieve the same measurement accuracy with fewer questions.

This simultaneous achievement of shorter test times and improved measurement accuracy is a major benefit for test administrators. Candidate will also be able to focus on questions that are appropriate for their level, reducing stress.

Introducing IRT requires considerable cost and time.

Implementing IRT requires expertise in statistics and psychometrics, and collaboration with external experts is essential. If your company does not have in-house IRT experts, you will need to work with consultants or research institutions. Implementing IRT without expert knowledge runs the risk of misuse and misinterpretation. Furthermore, estimating problem parameters requires data from hundreds of individuals, which requires a large initial investment.

On the other hand, in the long term, the return on investment can be expected to be sufficient due to the improved efficiency and quality of test administration. The ease of reusing questions and the resulting labor-saving nature of test administration will lead to medium- to long-term cost reductions.

To ensure the accuracy of IRT, a sufficient number of questions and Candidate are required.

Accurate parameter estimation requires test data from approximately 300 to 500 people per question ^*1, and using fewer people than this may affect measurement accuracy. For small-scale tests or tests with a small number Candidate, the accuracy of IRT may not be fully demonstrated.

A more effective approach would be to introduce IRT in stages, first testing it in some subjects or questions. Rather than introducing IRT to all exams at once, it would be more realistic to gradually introduce it in subjects with a large number of Candidate.

Therefore, it is important to have a long-term plan to gradually improve measurement accuracy through continuous data accumulation.

*1 The number Candidate required will vary depending on the IRT model used and the number of questions.

When introducing IRT, it is necessary to maintain consistency with the evaluation criteria used in traditional classical test theory. There must be no discrepancy between the ability levels of past successful candidates and the passing standards after the introduction of IRT.

To ensure fairness with past successful candidates, careful consideration is required when setting standards during the transition period. During the transition period, it may be necessary to use both the previous standards and the IRT standards.

Therefore, when introducing IRT, it is important to provide sufficient explanation to Candidate and other related parties and promote their understanding. It is also recommended to take a gradual transition plan and establish a verification period to minimize risk.

IRT is used in a variety of tests both in Japan and overseas. International tests such as TOEFL have adopted IRT from an early stage. In Japan, IRT has also been introduced in the Fundamental Information Technology Engineer Examination.

Guaranteeing fair evaluation regardless of the year or Test Center will improve the exam's social credibility. Candidate will be able to take the exam with the peace of mind that they will be evaluated according to the same standards no matter when or where they take it.

IRT is a theory that measures question characteristics and test-Candidate ability separately, and offers many advantages over traditional classical test theories. Implementation requires specialized knowledge and costs, but in the long run, it leads to improved test quality and operational efficiency. By hiring a company with the necessary expertise, you can expect smooth implementation and operation. For information on implementing CBT tests using IRT, please contact Prometric, which has a team of test development experts.