Prometric Blog

  1. TOP
  2. Prometric Blog
  3. Measuring questioning ability and the potential use of AI generation and CBT

Measuring questioning ability and the potential use of AI generation and CBT

June 2, 2025
  • CBT
  • IBT

author

Professor and Presidential Advisor, Osaka Municipal University
International Core Education Institute/Graduate School of Modern Systems Science
Doctor of Engineering Fumihito Ikeda

Professor and Assistant to the President (Admissions) at OMU (Osaka Public University) from April 2024. Born in Chiba Prefecture. Graduated from the Faculty of Science, Kyoto University, and completed a PhD in Engineering at NAIST. Worked at NTT Data (1996-2001) and Hokkaido University (2001-2024) before assuming his current position. At Hokkaido University, he was involved in the development and improvement of AO entrance exams, the introduction and evaluation of comprehensive entrance exams, and the development and introduction of competency evaluation in comprehensive selection. At OMU, he is working on entrance exam reform and high school-university-society connections through the development and evaluation of Qi (Question-insight). His hobbies are playing musical instruments (Vn, Cb, Piano) and badminton.

Declining birthrate and the popularization of universities

According to future population estimates by the National Institute of Population and Social Security Research, the population of 18-year-olds is expected to fall to about 80% of the current population (about 1.1 million) by 2035 (10 years from now), to less than 900,000. This rapid decline in the birthrate has direct implications for university management, and the risk of under-enrollment is rapidly increasing, especially at universities in rural areas.

On the other hand, the university enrollment rate for 2024 is expected to exceed 59%, a new record high, and we are entering an era in which "everyone can go to university." However, the actual number of students entering university has decreased by about 4,000 from the previous year, indicating a rapid decline in the birthrate.

This trend overlaps with the "popularization of higher education" pointed out by sociologist of education Michael Trow (Trow, 1974). In other words, universities are changing their nature from places for a small elite to educational institutions open to the general public.

The decline in academic ability and motivation as universities become more popular

The broadening of the base of universities is a welcome change that raises the knowledge base, but it also leads to a widespread lack of purpose or motivation to go to college to conform to others, and a decline in motivation to learn has long been a problem (Furuichi, 1993).Recent surveys have shown that the diversification of academic abilities of new students has led to an increase in the number of students who cannot keep up with classes and who lack basic literacy such as writing ability and logical thinking skills (National Federation of University Cooperative Associations, 2023).

This trend is likely to lower the quality of university education and have a negative impact on the social standing of graduates. Furthermore, a decline in the motivation and academic ability of new students will lead to a future decline in research capabilities, which is one of the important functions of universities.

In other words, the declining birthrate and the popularization of universities are an opportunity to reconsider university management, as well as to fundamentally reexamine the very nature of universities.

Three pillars of high school-university transition system reform

To solve these problems, it is necessary to firmly connect primary and secondary education with university, i.e., higher education, which is the tertiary education following primary and secondary education.

The Ministry of Education, Culture, Sports, Science and Technology (MEXT) established the Council for Reforming the High School-University Transition System in 2014, and set out the three pillars below, highlighting the need for triune reform (Council for Reforming the High School-University Transition System, 2016). The three elements of academic ability are: 1) knowledge and skills, 2) ability to think, make decisions, and express themselves, and 3) initiative, diversity, and collaboration.

  • High school education reform: Improving the quality of learning and comprehensively cultivating the three elements of academic ability
  • University education reform: ensuring quality of education and clarifying the exit path
  • University Entrance Examination Reform: Multifaceted and Comprehensive Evaluation of the Three Elements of Academic Ability

The reasons why these reforms are necessary include: 1) high school education is overly tied to university entrance examinations; 2) it is unclear what type of students universities are looking for (admissions policy) and what type of talent they want to develop (diploma policy); and 3) university entrance examinations that place too much emphasis on knowledge are preventing the comprehensive development of the three elements of academic ability.

High school education reform: inquiry-based learning

The core of the high school education reform is the "Integrated Inquiry Time" (Ministry of Education, Culture, Sports, Science and Technology, 2023). This is the so-called "integrated learning," which will be made compulsory in high schools from 2022, following on from elementary and junior high schools. The purpose is for students to set their own problems, collect, organize and analyze information, think logically, collaborate with others and acquire the ability to express themselves.

Because knowledge and skills are acquired naturally through this type of inquiry-based learning, inquiry-based learning is essential for comprehensively cultivating the three elements of academic ability. In other words, the high school education reform places emphasis on proactive, inquiry-based learning aimed at problem discovery and problem solving.

However, in actual educational settings, as inquiry-based learning becomes more diverse, there are many challenges, such as the fact that inquiry-based learning is limited to research learning, it is difficult to teach students how to discover problems, which is the starting point of an inquiry, or how to pose scientific questions, and there is no way to assess the abilities and qualities cultivated through inquiry-based learning (Yamashita and Ikeda, 2025).

University Education Reform: Quality Assurance and Three Policies

There is a strong demand for "quality assurance of education" in university education reform. The quality of university education needs to be assured at the following three stages (Scheerens, et al., 2011). These three elements are influenced by the context, that is, the social context in which the university is located. The three elements need to be designed and operated with an eye on the ideal future and current state of the university.

  • Input: Guarantee the quality of students through university entrance examinations based on admission policies.
  • Process: Ensure quality education through lesson design and learning support systems based on curriculum policy.
  • Output: Based on the diploma policy, ensure the quality of graduates through social contribution and diverse career paths

In this way, the quality of university education is assured based on these three policies. Therefore, it is necessary to clarify these three policies and evaluate and assure the quality of each in accordance with these policies.

University entrance reform: Comprehensive selection and the acceptance of diverse talent

The key to this transition from high school to university is the university entrance exam. In other words, in primary and secondary education, a "comprehensive selection" (formerly known as the AO entrance exam) is being promoted that can comprehensively and comprehensively evaluate the three elements of academic ability cultivated mainly through inquiry learning (Innovation Design & Technologies, 2023). It is quite new that Tohoku University announced in January this year that it will switch all of its admissions to the comprehensive selection by 2050.

In the comprehensive selection process, in order to look at the three elements of academic ability from multiple angles and in a comprehensive manner, Candidate are evaluated not only through academic tests, but also through application letters, interviews, essays, presentations, portfolios, etc. Among the selection processes for the Spiral inquiry entrance examination at J. F. Oberlin University, there is a long-term selection process that requires applicants to participate in an inquiry program run by the university (J. F. Oberlin University, 2021).

Issues with comprehensive selection

Although the comprehensive selection system is ideal for evaluating the three elements of academic ability in a multifaceted and comprehensive manner, it faces many challenges, such as the difficulty of designing the evaluation criteria, the heavy workload, and the difficulty of converting the evaluation results into scores (Innovation Design & Technologies, 2023). In particular, the increased workload of teachers takes away their research time, which leads to a decline in their research capabilities (Ministry of Education, Culture, Sports, Science and Technology, 2019).

Furthermore, comprehensive selection also poses a problem in terms of addressing the declining birthrate at universities. In other words, to maintain the number Candidate in the future, it is necessary to actively accept international students and working adults, but it is extremely difficult to evaluate Candidate with such diverse backgrounds using the same standards as Japanese high school students.

The current comprehensive selection process, which is mainly aimed at Japanese high school students, is already problematic in terms of fairness and impartiality, but if it also targets international students and working adults, it is expected to become even more problematic, as it will be difficult to ensure that applicants are assessed using the same standards (fairness), and it will be even more difficult to provide equal opportunities to take the exam in terms of geography and time (fairness).

The key to the solution: Questioning skills and generative AI x CBT

The key to solving these problems is to measure questioning ability using generative AI and CBT(Computer Based Testing). In other words, it is about using generative AI to efficiently create test questions that measure the ability to question, which encompasses the three elements of academic ability, and then using those questions to create a CBT that can be taken anywhere in the world at any time.

"The ability to ask questions" encompasses the three elements of academic ability

The ability to ask questions is an ability that encompasses the three elements of academic ability (Ikeda, 2020). In order to ask questions, it is necessary to recognize the gap between the subject and oneself (the knowledge and information one possesses). If one's knowledge and information are advanced, one can ask advanced questions. Therefore, scientific knowledge and information are required for scientific questions, and creative knowledge and information are required for creative questions.

The questions asked in scientific inquiry and research need to be scientific. Science is based on logic, and finding gaps in logic is a scientific question. Therefore, the ability to ask questions requires logical and critical thinking skills.

Furthermore, in order to carry out research, one must have the judgment to select from among the many questions one has identified which ones should be researched from the perspective of creating new knowledge or contributing to society. One must also be able to express the questions appropriately, as well as the process and results of the research based on those questions.

Furthermore, questions have the characteristic of motivating the person to whom they are directed (Senay, et al., 2010). If a question is directed at oneself, it motivates oneself, i.e., it encourages initiative. If a question is directed at others, it motivates them, i.e., it encourages collaboration.

In this way, developing the ability to question leads to the comprehensive development of the three elements of academic ability. And because logic is a universal skill, the ability to appropriately question logic is a universal ability that can be used anywhere in the world.

CBT paves the way for the future of university entrance exams

How wonderful it would be if people not only from all over Japan but from all over the world could take university entrance exams without being bound by time or place. If people with diverse backgrounds, knowledge and abilities could study together at university, they would be able to form a truly comprehensive body of knowledge.

CBT is a method that makes such a bright future possible. CBT here does not mean that questions are simply displayed on a computer and Candidate input their answers into the computer, but rather that Candidate can take the test at any time and place they like at test centers covering the world. This ensures fairness and impartiality in the opportunity to take the test.

This type of CBT eliminates the need for exam Test Center, exam supervisors, and the management and administration of these, as well as the distribution, collection, and checking of exam questions, all of which were previously necessary for university entrance exams. This makes it possible to avoid various human risks, such as information leaks and checking errors.

To realize this ideal CBT, test development based on IRT (Item Response Theory) is necessary, because it is necessary to properly evaluate the questioning ability of each Candidate, regardless of whether the questions or Candidate groups are different (see https://www.prometric-jp.com/column/archives/2).

Ideal CBT made possible by generative AI

Creating test questions to measure a new construct (such as the ability to measure) like questioning ability and grading the answers requires a lot of manpower, time, and money. Furthermore, there is an inevitable risk of human error, such as mistakes in setting questions and grading.

We believe that generative AI is effective in reducing these costs and risks. The reason is that in order to create good questions, it is necessary to create a large number of questions and then select good questions from them, but by using generative AI, it is possible to efficiently generate a large number of questions based on constructs. Generative AI can also generate rubrics (scoring criteria) based on constructs and grade them. Of course, the final decision is still made by a human, but this can reduce a significant amount of costs and risks.

Furthermore, creating test questions based on IRT generally requires conducting preliminary tests to adjust the level of difficulty, etc.; however, by using generative AI, it may be possible to automatically verify questions suitable for IRT by simulating test-taking by an anticipated group of test-takers while ensuring fairness and impartiality.

Conclusion: Towards the final stage of the evolution of universities

M. Trow, mentioned at the beginning of this article, predicted that universities would move away from elite education and towards the popularization of education, and further foresee that the final form of this process would be universalization. By utilizing generative AI to create test questions suitable for IRT, which ensures universal fairness and impartiality, and then implementing these in CBT, which can be taken universally, it is expected that we will pave the way to true universalization.

References

Questions about introducing CBT, consultation on operation, fees, etc.
If you have any questions, please feel free to contact us.

LANGUAGE