Psychometric Analysis of High School Mathematics Ability Test
DOI:
https://doi.org/10.15503/jecs2025.2.503.519Słowa kluczowe:
difficulty level, discrimination level, MEASURE, reliability, validityAbstrakt
Aim. Mathematics is essential for developing higher-order thinking skills. However, Indonesian students' performance in PISA 2022 and TIMSS 2015 shows a lack of proficiency in these skills. One contributing factor is school assessments that do not adequately foster higher-order thinking. This study evaluates the quality of a high school mathematics proficiency test designed to assess higher cognitive levels, focusing on validity, reliability, discrimination, difficulty, and variation in student responses.
Methods. This quantitative study used the MEASURE approach with five essay problems on exponents, logarithms, geometric sequences, arithmetic series, and trigonometry, involving three validators and 102 high school students from three district schools in Banjarmasin.
Results. The I-CVI values for questions 1, 2, 3, and 5 met the required criteria, requiring no revision. Question 4, with an I-CVI of 0.90, needed revision. All kappa values (κ) were ≥ 0.9, confirming reliability. Empirical data showed all questions had significance values below 0.05, and Cronbach’s alpha was 0.72. Discrimination analysis categorized 60% of questions as excellent and 40% as good. Difficulty levels indicated that 80% were moderate and 20% difficult. The instrument generated diverse student responses, reflecting varying ability levels.
Conclusion. The high school mathematics proficiency test, designed for cognitive levels C4 and C5, met validity and reliability criteria, demonstrated good discrimination, and had varied difficulty levels. While emphasizing higher-order thinking, a more comprehensive assessment could integrate multiple-choice questions for C1-C3 and essays for C4-C5.
Pobrania
Bibliografia
Abed, R. K., & Hassan, A. K. (2021). Multiple mathematical representations according to the (lesh) model of high school students and its relationship to their mathematical ability. Journal of Contemporary Issues in Business and Government, 27(3), 2200–2211. https://cibgp.com/au/index.php/1323-6903/article/view/1830/1800.
Almanasreh, E., Moles, R., & Chen, T. F. (2019). Evaluation of methods used for estimating content validity. Research in Social and Administrative Pharmacy, 15(2), 214–221. https://doi.org/10.1016/j.sapharm.2018.03.066
Anderson, L. W., & Krathwohl, D. R. (2001). A taxonomy for learning, teaching, and assessing. a revision of Bloom’s taxonomy of educational objectives. Addison Wesley Longman, Inc.
ArIsoy, B., & Aybek, B. (2021). The effects of subject-based critical thinking education in mathematics on students’ critical thinking skills and virtues. Eurasian Journal of Educational Research, 21(92), 99-119. https://doi.org/10.14689/ejer.2021.92.6
Azaryahu, L., Broza, O., Cohen, S., Hershkovitz, S., & Adi-Japha, E. (2023). Development of creative thinking patterns via math and music. Thinking Skills and Creativity, 47, Article 101196. https://doi.org/10.1016/j.tsc.2022.101196
Boopathiraj, C., & Chellamani, D. K. (2013). Analysis of test items on difficulty level and discrimination index in the test for research in education. International Journal of Social Science & Interdisciplinary Research, 2(2), 189-193.
Brame, C. J. (2019). Spotlight 1. Writing Learning Objectives Using Bloom’s Taxonomy. In C. J. Brame (Ed.), Science Teaching Essentials (pp. 29–34). Elsevier. https://doi.org/10.1016/B978-0-12-814702-3.00025-1
Davis, L. L. (1992). Instrument review: Getting the most from a panel of experts. Applied Nursing Research, 5(4), 194–197. https://doi.org/10.1016/S0897-1897(05)80008-4
Diki, D., & Yuliastuti, E. (2018). Discrepancy of difficulty level based on item analysis and test developers’ judgment: Department of Biology at Universitas Terbuka, Indonesia. In K. Persichitte, A., Suparman, M., Spector (Eds.), Educational Technology to Improve Quality and Access on a Global Scale (pp. 215–225). https://doi.org/10.1007/978-3-319-66227-5_17
Dunn, L., Morgan, C., O’Reilly, M., & Parry, S. (2003). The Student Assessment Handbook. Routledge. https://doi.org/10.4324/9780203416518
Etang, M. A. G., & Regidor, R. M. (2022). Students’ mathematical beliefs and attitudes as predictors to students’ mathematical ability. International Journal of Education and Social Science Research, 5(3), 23–60. https://doi.org/10.37500/IJESSR.2022.5303
Finch, W. Holmes., & French, B. F. (2019). Educational and Psychological Measurement. Routledge.
Furlan, R. M. M. M., Santana, G. A., Motta, A. R., & Casas, E. B. de Las. (2021). An instrument for tongue performance assessment in activities associated with digital games: Content and construct validity. Revista CEFAC, 23(5). https://doi.org/10.1590/1982-0216/20212359621
Furr, R. M. (2021). Psychometrics: An introduction (4th ed.). SAGE publications. (Original work published 2007)
Giani, Zulkardi, & Hiltrimartin, C. (2015). Analisis tingkat kognitif soal-soal buku teks matematika kelas VII berdasarkan taksonomi Bloom [Analysis of cognitive level of questions in grade VII mathematics textbooks based on Bloom's taxonomy]. Jurnal Pendidikan Matematika, 9(2), 78–98.
Gradini, E., Khairunnisak, C., & Noviani, J. (2022). Development of higher-order thinking skill (hots) test on mathematics in secondary school. AKSIOMA: Jurnal Program Studi Pendidikan Matematika, 11(1), 319-330. https://doi.org/10.24127/ajpm.v11i1.4649
Grégoire, J. (2016). Understanding creativity in mathematics for improving mathematical education. Journal of Cognitive Education and Psychology, 15(1), 24–36. https://doi.org/10.1891/1945-8959.15.1.24
Heong, Y. M., Othman, W. B., Yunos, J. B. M., Kiong, T. T., Hassan, R. B., & Mohamad, M. M. B. (2011). The level of Marzano Higher Order Thinking Skills among technical education students. International Journal of Social Science and Humanity, 1(2), 121–125.
Kalkbrenner, M. T. (2021). A practical guide to instrument development and score validation in the social sciences: The measure approach. Practical Assesment, Research & Evaluation, 26(1), Article 1. https://doi.org/10.7275/svg4-e671
Karsenty, R. (2020). Mathematical ability. In S. Lerman (Ed.), Encyclopedia of mathematics education (pp. 494–497, 2nd ed.).
Kemendikbud. (2022). Capaian Pembelajaran pada Pendidikan Anak Usia Dini, Jenjang Pendidikan Dasar, dan Jenjang Pendidikan Menengah pada Kurikulum Merdeka [Learning Outcomes in Early Childhood Education, Elementary Education Level, and Secondary Education Level in the Independent Curriculum]. https://kurikulum.kemdikbud.go.id/wp-content/unduhan/CP_2022.pdf
Krutetskii, V. A. (1976). The Psychology of Mathematical Abilities in School Children. University of Chicago Press.
Kusuma, M. D., Rosidin, U., Abdurrahman, & Suyatna, A. (2017). The development of higher order thinking skill (hots) instrument assessment in physics study. IOSR Journal of Research & Method in Education, 7(1), 26-32. https://doi.org/10.9790/7388-070103XXXX
Lacave, C., Molina, A. I., & Redondo, M. A. (2018). A preliminary instrument for measuring students’ subjective perceptions of difficulties in learning recursion. IEEE Transactions on Education, 61(2), 119–126. https://doi.org/10.1109/TE.2017.2758346
Lewy, L., Zulkardi, & Aisyah, N. (2009). Pengembangan soal untuk mengukur kemampuan berpikir tingkat tinggi pokok bahasan barisan dan deret bilangan di kelas IX akselerasi SMP Xaverius Maria Palembang [Development of questions to measure high-level thinking skills on the topic of number sequences and series in class IX acceleration at SMP Xaverius Maria Palembang]. Jurnal Pendidikan Matematika, 3(2), 15–28.
Lynn, M. R. (1986). Determination and quantification of content validity. Nursing Research, 35(6), 382–385.
Muhammad, I., & Angraini, L. M. (2023). Research on students’ mathematical ability in learning mathematics in the last decade: A bibliometric review. JOHME: Journal of Holistic Mathematics Education, 7(1), 108-122. https://doi.org/10.19166/johme.v7i1.6867
Mullis, I. V. S., Martin, M. O., Foy, P., & Hooper, M. (2015). TIMSS 2015 International Result in Mathematics. International Study Center, Lynch School of Education, Boston College.
Nababan, S. A., & Tanjung, H. S. (2020). Development of learning instrument based on realistic mathematics approach to improve mathematical disposition ability. Advances in Mathematics: Scientific Journal, 9(12), 10325–10333. https://doi.org/10.37418/amsj.9.12.24
NCTM. (2000). Principles and standards for school mathematics. NCTM: Reston VA.
Obeng-Denteh, W., & Amoah-Mensah, J. (2011). Pure mathematicians’ and applied mathematicians’ saga: but one family! a mathematical panacea. Continental Journal of Education Research, 4(2), 1–10. https://www.researchgate.net/publication/326232786
OECD. (2023). PISA 2022 Results (Volume I). OECD. https://doi.org/10.1787/53f23881-en
Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Focus on research methods: Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing and Health, 30(4), 459–467. https://doi.org/10.1002/nur.20199
Radmehr, F., & Drake, M. (2018). An assessment-based model for exploring the solving of mathematical problems: Utilising revised Bloom’s taxonomy and facets of metacognition. Studies in Educational Evaluation, 59, 41–51. https://doi.org/10.1016/j.stueduc.2018.02.004
Rahayu, W., Sinaga, O., Oktaviani, M., & Zakiah, R. (2019). Analysis of mathematical ability of high school students based on item identification of national examination set. In Jufrizal, Y. Rozimela, Atmazaki, A. Fauzan, R. Syahrul, Hamzah, R. Refnaldi, & Yerizon (Eds.), Proceedings of the 1st International Conference on Innovation in Education (ICoIE 2018) (pp. 412-416). Atlantis Press. http://dx.doi.org/10.2991/icoie-18.2019.88
Ramdhani, S. S., Susanti, R., & Meilinda. (2024). Cognitive level of Program for International Student Assessment (PISA) questions based on the revised Bloom’s taxonomy. European Journal of Education and Pedagogy, 5(2), 104–112.
Raykov, T., & Marcoulides, G. A. (2011). Introduction to Psychometric Theory. Routledge.
Reiner, C. M., Bothell, T. W., & Sudweeks, R. R. (2002). Preparing effective questions a Self-directed workbook for educators. New Forums Pres.
Sari, R. N., & Juandi, D. (2023). Improving Student’s Critical Thinking Skills in Mathematics Education: A Systematic Literature Review. Jurnal Cendekia : Jurnal Pendidikan Matematika, 7(1), 845–861. https://doi.org/10.31004/cendekia.v7i1.2091
Sarnoko, S., Asrowi, A., Gunarhadi, G., & Usodo, B. (2024). An analysis of the application of problembased learning (PBL) model in mathematics for elementary school students. Jurnal Ilmiah Ilmu Terapan Universitas Jambi, 8(1), 188–202. https://doi.org/10.22437/jiituj.v8i1.32057
Sürücü, L., & Maslakçı, A. (2020). Validity and reliability in quantitative research. Business & Management Studies: An International Journal, 8(3), 2694–2726. https://doi.org/10.15295/bmij.v8i3.1540
Tanujaya, B. (2016). Development of an instrument to measure higher order thinking skills in senior high school mathematics instruction. Journal of Education and Practice, 7(21), 144–148.
Tobón, S., & Luna‐Nemecio, J. (2021). Complex thinking and sustainable social development: Validity and reliability of the complex‐21 scale. Sustainability (Switzerland), 13(12), Article 6591. https://doi.org/10.3390/su13126591
Wieczerkowski, W., Cropley, A. J., & Prado, T. M. (2000). Nurturing talents/gifts in mathematics. In K. A. Heller, F. J. Mönks, R. J. Sternberg, & R. F. Subotnik (Eds.), International handbook of giftedness and talent (pp. 413–425). http://dx.doi.org/10.1016/B978-008043796-5/50029-2
Wijaya, M. C., & Kloping, Y. P. (2021). Validity and reliability testing of the Indonesian version of the eHealth Literacy Scale during the COVID-19 pandemic. Health Informatics Journal, 27(1). https://doi.org/10.1177/1460458220975466
Yang, E., Halpin, P., & Handy, D. (2022). Using psychometric analysis to improve soft-skill assessments. MDRC.
Yee, M. H., Yunos, J. Md., Othman, W., Hassan, R., Tee, T. K., & Mohamad, M. M. (2015). Disparity of learning styles and higher order thinking skills among technical students. Procedia – Social and Behavioral Sciences, 204, 143–152. https://doi.org/10.1016/j.sbspro.2015.08.127
Yusoff, M. S. B. (2019). ABC of content validation and content validity index calculation. Education in Medicine Journal, 11(2), 49–54. https://doi.org/10.21315/eimj2019.11.2.6
Zamanzadeh, V., Ghahramanian, A., Rassouli, M., Abbaszadeh, A., Alavi-Majd, H., & Nikanfar, A.-R. (2015). Design and implementation content validity study: Development of an instrument for measuring patient-centered communication. Journal of Caring Sciences, 4(2), 165–178. https://doi.org/10.15171/jcs.2015.017
Zulkifli, I. Z., Mohamad Razi, N. F., & Mohammad, N. H. (2021). Cognitive levels towards performance of mathematics score in secondary school. Mathematical Sciences and Informatics Journal, 2(1), 70–78. https://doi.org/10.24191/mij.v2i1.13042
Pobrania
Opublikowane
Numer
Dział
Licencja
Prawa autorskie (c) 2025 Aminah Ekawati, Tatag Yuli Eko Siswono, Agung Lukito
Utwór dostępny jest na licencji Creative Commons Uznanie autorstwa 4.0 Międzynarodowe.
CC-BY
Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal. All authors agree for publishing their email adresses, affiliations and short bio statements with their articles during the submission process.
