engleski

Psychometric validation of an objective measure of STEM school achievement in primary school

General description on research questions, objectives and theoretical Framework: Many national governments and organizations around the world (e.g. American National Science Foundation, European Union countries etc.) within their educational policies recognized the problem of lower interest for careers in science, technology, engineering, mathematics and related fields (STEM). Therefore, a big effort is made in order to encourage the interest for these fields among students (Honey, Pearson & Schweingruber, 2014). Some of the current international projects have focus on the students’ achievement in the fields of science and mathematics, such as PISA or TIMSS (OECD, 2003 ; Braš Roth, Gregurović, Markočić Dekanić & Markuš, 2008 ; Mullis & Martin, 2013). Their frameworks influenced changes in many national science or math curricula (e.g. KMK, 2003). PISA examines mathematical and science literacy on 15-year old students, while the TIMSS study encompasses students at the age of 10 and 14. This age interval is generally taken as an important for forming attitudes about future interests and careers (Lindahl, 2007) as well as acquiring knowledge and competences necessary for following educational levels (OECD, 2003). Recent studies on STEM achievement refer to the problem of traditionally separated school subjects involved in STEM areas (mathematics, science, biology, physics, chemistry etc). “Most studies of STEM learning consider each discipline singly and do not measure students’ ability to make connections across disciplines” (Honey et al, 2014, p. 52). Also, “assessment instruments on integration in STEM are rare because theories and tests have generally focused on content area–specific concepts and procedures” (Honey et al, 2014, p. 52). Therefore Xie, Fang and Shauman (2015) discuss the term 'STEM education' which emphasizes logical and conceptual connections across different STEM fields. In this view, the STEM education is considered as a whole, not as a set of separated subjects. However, the idea of simply integrating STEM subjects in school practice is complex and should be considered carefully. According to meta-analysis about integrated mathematics and science instruction conducted by Hurley (2001), there are fewer positive benefits of integration for mathematics outcomes compared to science outcomes. The results indicate that it is difficult to enhance good mathematics achievement just by integrating the math into another discipline. These findings imply that we should take the integration into account carefully, focusing on logical and conceptual connections across different STEM subjects. Cognitive abilities which connect STEM disciplines are problem-solving, spatial ability, numeracy, comprehension, reasoning, knowledge acquisition, abstract thought, and connection-making (Xie et al, 2015). Cognitive ability is strongly associated with students’ school performance, particularly in math and science tests (Deary, Strand, Smith & Fernandez, 2007). The results of Croatian students on PISA and TIMSS assessments show the low or average results in science and mathematics (Braš Roth et al, 2008 ; NCVVO, 2012a ; 2012b). Also, the achievement of Croatian students in science and mathematics is low in comparison to their achievement in other school subjects (Burušić, Babarović & Šakić, 2008). These findings motivated us for examining the STEM achievement of Croatian 10, 11 and 12-year-old students. This is the period when the Croatian students change from lower to middle compulsory education. Valid STEM achievement instruments and especially psychometric validated integrated STEM knowledge tests are still missing in Croatia. With the purpose to adequately measure STEM knowledge in Croatian primary schools, we designed three STEM tests, following the above-mentioned outlines for STEM education and the analysis results of current curriculum resources (MZOS, 2006 ; 2010). The aim of this research is psychometric validation of this newly designed STEM tests for grades 4, 5 and 6. Methods/Methodology: • Participants and procedure: The participants in this study were 1798 primary school students attending grades 4 to 6 (age 10 - 12). They are equally distributed by gender (49.8%girls) and grades (N4th grade586, N5th grade=580, N6th grade=632). This study is a part of a broader research project JOBSTEM - STEM career aspirations during primary schooling (www.jobstem.eu). The study is conducted in 16 schools in the city of Zagreb and its surroundings. Students from sixteen primary schools participated in the study. Within each school, two classes of students within one generation were sampled by random, and joined the survey. Data used in this study was collected within regular classes in school. In addition, student school achievement data (school grades) was collected from schools. • Instruments Three tests were designed to measure knowledge in STEM area for grades 4, 5 and 6, following the outlines for STEM education and for task design (Sullivan, Clarke & Clarke, 2013 ; IDM, 2007 ; Smith & Stein, 1998) and current curriculum resources (MZOS, 2006 ; 2010). The final versions of the three knowledge tests were pretested with a larger number of items in pre-test versions (25 test items). The items with the best psychometric properties were retained in the final tests versions. The final tests for each grade contained 20 items. Each item in the test had only one correct answer and it was coded as 1 or 0. The test total score was calculated as sum of the item points ranging from 0 to 20, with higher scores indicating greater knowledge in STEM area. School achievement was measured by school grades (marks) scaling from1 to 5 (5 - Excellent, best possible grade ; 4 - Very Good, next highest grade ; 3 - Good, indicates average performance ; 2 - Sufficient, lowest passing grade ; 1 - Insufficient, failing grade). In the Croatian school system and school curriculum, the STEM area in grades 4 to 6 is covered by the following school subjects: mathematics, nature, geography and technical education. STEM school achievement in our study was measured as an average of school marks in all STEM school subjects within different grades (4, 5, 6). Expected outcomes/results: The mean score in 4th grade was M=11.81, in 5th grade M=9.34 and in 6th grade M=10.93. The average item difficulty was 61% for 4th grade, 42% for 5th and 58% for 6th grade. Somewhat higher standard deviation (SD = 4.32) in 4th grade indicates slightly higher variations between students’ knowledge in 4th grade compared to 5th (SD = 3.66) and 6th grade (SD = 3.97). The distribution of scores in all three tests is symmetric but with higher tails. Cronbach’s alpha coefficients indicate adequate internal reliability, α4th grade = 0.78 ; α5th grade = 0.70 ; α6th grade = 0.79. To examine construct validity, principal component analysis (PCA) was performed. As expected, Scree test criterion revealed a single factor structure in all three tests, where all items were highly saturated with general factor. Single factor structures were confirmed by confirmatory factor analysis (CFA) for test in 4th grade (χ² (170) = 239 ; p<0.05 ; χ²/df = 1.41 ; CFI = 0.951 ; TLI = 0.945 ; RMSEA = 0.026), 5th grade (χ² (170) = 313.15 ; p<0.05 ; χ²/df = 1.84 ; CFI = 0.941 ; TLI = 0.935 ; RMSEA = 0.035) and 6th grade (χ² (170) = 248.8 ; p<0.05 ; χ²/df = 1.46 ; CFI = 0.938 ; TLI = 0.931 ; RMSEA = 0.027). Correlations (r) between test scores and achievement in STEM school subjects were all positive and moderate whereby all three test scores had highest correlations with achievement in math and slightly lower correlation with other STEM school subjects (geography, nature and technical education). The STEM knowledge tests proved to be reliable and valid measures of knowledge in STEM area in primary school. It seems that knowledge in STEM area, taught within distinctive school subjects, can be integrated in one comprehensive but concise measure. These tests could be used as objective, short and valid assessment of STEM knowledge in Croatian primary schools.

STEM (science, technology, engineering, mathematics) ; STEM achievement ; measuring achievement,

nije evidentirano