Test Development Service
Test Development Process
Designing an appropriate “measure” to serve as a standard for validity and reliability
Prometric 's rigorous process for exam development emphasizes developing valid, reliable, and fair test.
A rigorous test development process is applied to all tests to determine, for example, whether the doctor is qualified, the architect has the appropriate knowledge, and the engineer is sufficiently skilled.
An exam is a promise and guarantee by Test sponsor that the person being granted a license or certification meets important criteria.
There are no shortcuts to creating valid, reliable, and fair tests. And that's what the exam writing process is all about.
Step1: Job Analysis /test specification design
The first step in developing and operating Test that maintains fairness, validity, and reliability.
Job Analysis
The first step in the exam writing process is a thorough and complete Job Analysis. For example, before we can test who is a good nurse or not, we must first identify what it takes to be a good nurse. This determines the knowledge, skills, and abilities required to effectively perform a specific job.
Test Blueprint Design
When designing Test blueprint, we decide on the format of Test item (multiple choice format, descriptive format, etc.), difficulty level, knowledge areas to be asked, and number of questions to be asked. This ensures that all Candidate are consistently measured on common knowledge, skills and abilities.
Solve challenges such as:
- How to decide the duration and the item format of the exam?
- What is the first step in launching a company's internal exam?
- What is necessary to design a test that measure the ability we want to assess?
- What is the critical point when transferring the exam from PBT to CBT?
Deliverables
- test definition document
- blue print
Case Study
Step2: Test question creation/verification
Support for creating Test items by sharing correct question writing process/rules
We support the creation of Test items that take into account the quality of Test.
Writing questions that do not follow the correct process or rules tends to result in Test are not of sufficient quality.
One of the most important factors in creating Test is leveraging the knowledge possessed by subject matter experts (SMEs).
Analysis of existing Test question groups (item bank)
If there are existing problems, we will analyze them and consider whether to continue using them.
We also analyze the number of Test items added for each knowledge area and provide feedback to the next Test question creation process.
Test question creation and verification support by a knowledge domain expert (SME) group
One of the most important factors in creating valid, reliable, and fair Test is leveraging the knowledge of subject matter experts.
Subject matter experts (SMEs) provide industry-specific knowledge and field experience on what Candidate "need to know" to function effectively in practice.
Our company can not only collaborate with SMEs designated by Test sponsor and conduct workshops, but also hire qualified experts with the approval of Test sponsor.
Solve challenges such as:
- I don't know how to write Test items
- I want to create Test items with an appropriate level of difficulty.
- How many questions do I need to create?
- I want to know the quality standards for Test items.
- I don't know if my abilities are being measured correctly.
- It is difficult to create questions every time.
Deliverables
- Test question creation rules
- Test items
Step3: Create exam form
Building a system that simultaneously provides flexible examination opportunities and ensures fairness through CBT
Building a system to appropriately present Test items
An appropriately equated Test form is created from the created Test question group (item bank).
Test form refers to the set of Test items that are asked to each Candidate.
In order to provide flexible testing opportunities through CBT services, we need a system that allows different questions to be asked for Candidate and at the same time ensures fairness.
Test In order to avoid overexposure of questions and at the same time to avoid creating unfair situations between Candidate, we will consider the most appropriate mechanism to present questions to Candidate from the form method, LOFT (Linear-on-the-fly Testing) method, and other question submission methods.
Here, psychometricians will ensure fairness by designing (equating) each form to be equivalent in terms of scoring and scoring criteria, based on scientific evidence.
Solve challenges such as:
We propose the optimal question structure based on Test specification design.
- Form method: A system in which a set of questions is prepared in advance and asked.
- LOFT method: A system in which a form is created with a different combination of questions for each Candidate when taking a test.
Deliverables
- Test form review results
- Test form
Step4: Setting Passing Score
Setting pass/fail criteria based on scientific evidence
Our highly skilled psychometric experts use scientific evidence to set passing scores in a way that is appropriate for your situation.
The pass/fail criteria thus established are guaranteed to be highly reliable in determining the appropriate cut-off score for Test.
- Modified Angoff method
- Bookmark method
- Equating methods besed on Classical Test Theory (CTT)
- Item Response Theory (IRT) scale scores (1PL model, 2PL model, 3PL model, etc.)
Step5: Test Result Analysis
Verification and continuous Test management to maintain and improve Test quality
To maintain and improve Test quality
This result analysis process is the most important for accurately understanding Test quality.
Confirming the validity and reliability of the test through the psychometric analysis (scientific analysis method based on the test theory, etc.) and feeding the results back into the test development process enables us to maintain and improve the test quality continuously.
Prometric employs a strong staff of psychometric analysis experts, whose core responsibility is to ensure that exams ultimately meet the design goals, market requirements, and legal standards of Test sponsor It is to guarantee.
We perform various analyzes to determine the quality of the measurement for each exam question.
We use professional industry-standard modeling for all exams, as well as criterion validity, simulation studies, custom research, and utilize industry-standard methodologies.
Solve challenges such as:
- I want to keep the passing rate constant.
- What data other than Test scores can be used?
- I have a question with a low percentage of correct answers, what should I do?
- The abilities of successful candidates vary depending on Test held.
- I want to use Test results for education.
- How can we use data?
Deliverables
- Item analysis report
- Psychometric analysis report