Robin H. Kay
Liesel Knaack
Authors
Dr. Robin H. Kay is an assistant professor with the Faculty of Education, at the University Of Ontario Institute Of Technology in Oshawa, Ontario. Correspondence regarding this article can be sent to: robin.kay@uoit.ca
Dr. Liesel Knaack is an assistant professor with the Faculty of Education, at the University Of Ontario Institute Of Technology in Oshawa, Ontario.
Abstract: The current study offers a formative analysis of the impact of learning objects in middle school mathematics and science classrooms. Five reliable and valid measure of effectiveness were used to examine the impact of learning objects from the perspective of 262 students and 8 teachers (14 classrooms) in science or mathematics. The results indicate that teachers typically spend 1-2 hours finding and preparing for learning-object based lesson plans that focus on the review of previous concepts. Both teachers and students are positive about the learning benefits, quality, and engagement value of learning objects, although teachers are more positive than students. Student performance increased significantly, over 40%, when learning objects were used in conjunction with a variety of teaching strategies. It is reasonable to conclude that learning objects have potential as a teaching tool in a middle school environment.
L’impacte des objets d’apprentissage dans les classes de mathématique et de sciences à l’école intermédiaire : une analyse formative
Résumé : Cette étude présente une analyse formative de l’impacte des objets d’apprentissage dans les classes de mathématique et de sciences à l’école intermédiaire. Cinq mesures de rendement fiables et valides ont été exploitées pour examiner l’effet des objets d’apprentissage selon 262 élèves et 8 enseignants (414 classes) en science ou mathématiques. Les résultats indiquent que les enseignants passent typiquement 1-2 heures pour trouver des objets d’apprentissage et préparer les leçons associées qui seraient centrées sur la revue de concepts déjà vus en classe. Quoique les enseignants aient répondu de façon plus positive que les élèves, les deux groupes ont répondu positivement quant aux avantages au niveau de l’apprentissage, à la qualité ainsi qu’à la valeur motivationnelle des objets d’apprentissage. Le rendement des élèves aurait aussi augmenté de façon significative, plus de 40%, quand les objets d’apprentissage ont été exploités avec une variété de stratégies d’enseignement. Il serait donc raisonnable de conclure que les objets d’apprentissage ont un potentiel comme outils d’enseignement à l’école intermédiaire.
Until recently, the vast majority of use and research with respect to learning objects has been in the domain of higher education (e.g., Haughey & Muirhead, 2005; Kay & Knaack, in press). Increased presence of learning objects in K-6 (e.g., Bower, 2005; Brush & Saye, 2001; Clarke & Bowe, 2006a, 2006b; Freebody, Muspratt, & McRae, 2007; Kay & Knaack, 2005; Liu & Bera, 2005; Nurmi & Jaakkola, 2006; Reimer & Moyer, 2005) indicates that there is a need for more research on learning objects in middle school environments. Research on the impact, effectiveness, and usefulness of learning objects in the K-6 domain is limited, partially because comprehensive, theoretically-based, reliable, and valid evaluation tools are scarce. The purpose of the current study is to examine the impact of learning objects in middle school classrooms.
Learning objects (LO) are defined in this paper as “interactive web-based tools that support learning by enhancing, amplifying, and guiding the cognitive processes of learners”. This definition was arrived at based on a aggregate of previous definitions (Agostinho, Bennett, Lockyer & Harper, 2004; Butson, 2003; McGreal, 2004; Parrish, 2004; Wiley, et al. 2004).
Learning objects offer a number of advantages for educators and students including accessibility (Wiley, 2000), ease of use (e.g., Gadanidis, Gadanidis, & Schindler, 2003; Sedig & Liang, 2006), reusability (e.g., Agostinho et al., 2004; Duval, Hodgins, Rehak & Robson, 2004; Rehak & Mason, 2003), interactivity (e.g., Gadanidis, et al., 2003; Sedig & Liang, 2006) and visual supports (e.g., Gadanidis, et al., 2003; Sedig & Liang, 2006).
With respect to enhancing learning, many learning objects are interactive tools that support exploration, investigation, constructing solutions, and manipulating parameters instead of memorizing and retaining a series of facts. The success of this constructivist based model is well documented (e.g., Albanese & Mitchell, 1993; Bruner, 1983, 1986; Carroll, 1990; Collins, Brown, & Newman, 1989; Vygotsky, 1978). In addition, a number of learning objects have a graphical component that helps make abstract concepts more concrete (Gadanidis et al., 2003). Furthermore, certain learning objects allow students to explore higher level concepts by reducing cognitive load. They act as perceptual and cognitive supports, permitting students to examine more complex and interesting relationships (Sedig & Liang, 2006). Finally, learning objects are adaptive, allowing users to have a certain degree of control over their learning environments, particularly when they are learning and for how long.
In spite of this list of potential benefits, relatively little systematic research has been done examining the actual use and impact of learning objects in middle school classrooms.
An extensive review of the literature on learning objects in the past 10 years uncovered nine studies focussing on elementary (Bower, 2005; Clarke & Bowe, 2006a, 2006b; Reimer & Moyer, 2005) and middle school (Akpinar & Bal, 2006; Bower, 2005; Freebody et al., 2007; Kong & Kwok, 2005; Liu & Bera, 2005) grade levels. Note that with the exception of three detailed, comprehensive reports from the Learning Federation (Clarke & Bowe, 2006a, 2006b; Freebody et al., 2007) these studies were selected from peer-reviewed journals. The papers go back no further than 2005, indicating that research in this domain is fairly new.
Context. The context of use varied with respect to investigations of learning objects. In terms of time spent using learning objects, some studies observed students use these tools for 40-60 minutes (Akinpar & Bal, 2006; Nurmi & Jaakkola, 2006; Reimer & Moyer, 2005). Other studies, investigated the use of learning objects over several weeks (Clarke & Bowe, 2006a, 2006b; Freebody et al., 2007; Liu & Bera, 2005). With respect to the number of learning objects evaluated, some papers focussed on a single learning object, while others looked at multiple learning objects (Clarke & Bowe, 2006a, 2006b; Freebody et al., 2007; Nurmi & Jaakkola, 2006). Finally, regarding implementation strategies, a majority of teachers played a facilitating role, allowing students to explore the learning object on their own (Liu & Bera, 2005; Nurmi & Jaakola, 2006; Reimer & Moyer, 2005), although other researchers (Clarke & Bowe , 2006a, 2006b; Freebody et al., 2007) reported the use of numerous strategies over an extended period of time including large group discussions, worksheets, collaborative learning, writing reflective comments, and integration with other subject areas.
Teacher perspective. Three reports looked specifically at teacher attitudes toward the use of learning objects in the classroom (Clarke & Bowe, 2006a, 2006b; Freebody et al., 2007). Qualitative and survey data collected suggested that instructors valued the following characteristics: immediate feedback, the ability to replay and redo tasks for both enjoyment and mastery, and the motivational qualities. To date, no peer-reviewed journal articles could be found that examined middle school teacher’s attitudes toward of learning objects.
Student perspective. Four studies examined student attitudes toward learning objects (Clarke & Bowe, 2006a, 2006b; Freebody et al., 2007; Reimer & Moyer, 2005). Students reported liking learning objects because they were (a) fun and enjoyable, (b) easy to control with respect to the pace of learning, (c) provided timely feedback, (d) consisted of a number multimedia tools, and (e) helped them learn (Clarke & Bowe, 2006a, 2006b; Freebody et al., 2007; Reimer & Moyer, 2005).
Student performance. In contrast to research presented for secondary and higher education domains (Kay & Knaack, 2005, 2007a, 2007b), the majority of studies done on learning objects and K-6 classrooms have focussed on some form of student performance (Akpinar & Bal, 2006; Bower, 2005; Freebody et al., 2007; Kong & Kwok, 2005, Liu & Bera, 2005; Nurmi & Jaakkola, 2006; Reimer & Moyer, 2005). In all seven studies, students who used learning objects showed significant improvement on various performance measures. Several contextual details, though, are worth noting.
Nurmi & Jaakkola (2006) noted that learning performance was dependent on the type of learning object and how it was used. Students working with drill-and-practice learning objects tended to be less focussed on learning and more focussed on competing with their peers. Students involved in a mixed learning object / laboratory lesson, performed significantly better than in other learning scenarios used with learning objects. Freebody et al. (2007) reported significantly higher performance gains in middle school classrooms where learning objects were used to teach mathematics over a six week period versus “business as usual” classrooms over the same time period. Reimer & Moyer (2005) noted that students using virtual manipulatives in their study improved significantly in conceptual knowledge, but not in procedural knowledge. Finally, Bower (2005) observed that performance gains were contingent upon receiving feedback that compared personal results with the group.
This study reviewed eight articles looking at the use of learning objects in either elementary or middle school classrooms. It is commendable that most of these studies emphasized student performance, a pattern that has not been observed at higher levels of education. That said, a number of challenges remain with respect to improving the analysis of learning objects, particularly in middle schools.
First, while there is good data on how teacher use learning objects, limited information is provided on teachers’ attitudes toward learning objects. Only three studies examined the teacher’s perspective (Clarke & Bowe, 2006a, 2006b; Freebody et al., 2007).
Second, even though a wide range of learning objects exist, the majority of papers focus on a single learning object. It is difficult to determine whether the evaluation tools used in one study generalize to the full range of learning objects that are available.
Third, sample populations tested in many studies are relatively small and poorly described making it challenging to extend any conclusions to a larger population. All but two studies (Bower, 2005; Freebody et al., 2007) examined less than 70 students.
Fourth, triangulation of data collection is somewhat limited with only two papers using more than two data collections procedures (Freebody et al., 2007; Reimer & Moyer, 2005). Furthermore, only one unpublished report (Freebody et al., 2007) looked at student attitudes, teacher attitudes, and student performance simultaneously.
Finally, while most evaluation studies reported that students benefited from using learning objects, the evidence is based on loosely designed assessment tools with no validity or reliability. Only two studies reviewed offered estimates of reliability (Kong & Kwok, 2005; Liu & Bera, 2005) and no studies provided validity data. The lack of reliability and validity of evaluation tools reduces confidence in the results presented to date.
In summary, previous methods used to evaluate learning are limited with respect to examining multiple perspectives of learning objects, particularly those of teachers, number of learning objects assessed, sample size, reliability, and validity.
The purpose of this study was to examine the impact of learning objects in middle school classrooms from the perspective of both teachers and students.
In order to examine the impact of learning objects on middle school students, the following questions were addressed in the data analysis:
In order to address the methodological challenges noted in the literature review, the following steps were taken:
Teachers. The teacher sample consisted of 8 teachers (3 males, 5 females; Grades 4 to 8) and 14 classrooms (a number of teacher used learning objects more than once). Teaching experience ranged from 0.5 to 22 years with a mean of 6.6 (SD = 7.9). Subject areas taught were math (n=12 classrooms) and science (n=2 classrooms). A majority of the teachers rated their ability to use computers as strong or very strong (n=5) and their attitude toward using computers as positive or very positive (n=7). In spite of the high ability and positive attitude, none of teachers used computers in their classrooms more than once a month.
Students. The student sample consisted of 262 middle school students (143 males, 119 females), 10 to 15 years of age (M = 12.5, SD = 1.3). Eighty-four percent (n=231) of the student reported being good at working with computers, 90% (n=235) liked working with computers at school, and 87% (n=229) noted that they felt comfortable working with computers in school.
The population base spanned two separate boards of education, seven middle schools, and eight different classrooms. The students were selected through convenience sampling and had to obtain signed parental permission to participate.
Learning Objects. A total of 10 unique learning objects were selected by middle school teachers in this study (see http://faculty.uoit.ca/kay/papers/ MSLO.html for a complete set of links to the learning objects used). Most of the learning objects covered topics in mathematics including factoring, estimating, triangular pyramids, multiplying fractions, geometric facts, probability, and measuring similar shapes. Two science learning objects covered rocks and soil and volume and displacement.
Teachers from three boards of education in Ontario, Canada were contacted via email and informed of the learning object study by an educational coordinator. Participation was voluntary and a participant could withdraw from the study at any time. Each teacher received a half day of training in November on how to choose, use, and assess learning objects (see http://www.education.uoit.ca/lordec/lo_use.html for more details on the training provided). They were then asked to use at least one learning object in their classrooms by April of the following year. Email support was available throughout the duration of the study. All students in a given teacher’s class used the learning object that the teacher selected. However, only those students with signed parental permission forms were permitted to fill in an anonymous, online survey about their use of the learning object. In addition, students completed a pre- and post-test based on the content of the learning object.
All teachers in this study used learning objects in a lab setting. In order to simulate real middle school environment as much as possible, teachers were given full control over the learning object they selected, the purpose for using the learning object, how long the learning object was used, teaching strategies for using the learning objects, and the design of the pre- and post-tests.
Teacher Use. Teachers were asked (a) how much preparation was involved when using learning objects (e.g., how long it took them to find and integrate learning objects into their classroom), (b) their purpose for using the learning object (e.g., motivate students, teach a new concept, review, supplementing a lesson), (c) strategies they used to integrate learning objects (e.g., demonstration, providing a set of guiding questions, let student explore, discussion after learning object) and (d) how long the learning object was used in their classroom.
Teacher survey. After using a learning object, each teacher completed the Learning Object Evaluation Scale for Teachers (LOES-T) to determine their perception of (a) how much their students learned (learning construct), (b) the quality of the learning object (quality construct), and (c) how much their students were engaged with the learning object (engagement construct). Data from the LOES-T showed low to moderate reliability (0.63 for learning construct, 0.69 for learning object quality construct, 0.84 for engagement construct), and good construct validity using a principal components factor analysis. See Kay & Knaack (2007c) for a detailed of the teacher-based learning object scale.
Teacher comments. Finally, teachers were asked to comment on the overall impact that the learning object had on learning (Q9, Appendix A).
Table 1. Coding Scheme to Categorize Student Comments about Learning Objects
Student comments. Students were asked to comment on what they liked and disliked about the learning object (Appendix B – questions 13 and 14). These qualitative items were organized according to the three main constructs identified in the literature review (learning, quality, and engagement) and analysed using the coding scheme provided in Table 1. This coding scheme (Kay & Knaack, 2007a) was used to categorize 596 student comments. Each comment was then rated on a five-point Likert scale (-2 = very negative, -1 = negative, 0 = neutral, 1 = positive, 2 = very positive). Two raters independently assessed all comments made by students based on categories listed in Table 1 and the 5 point rating system. The approximate inter-rated reliability achieved was 75% for categories and 80% for ratings. The two raters then met to discuss any discrepancies. Inter-rater reliability increased to 99 % for categories and 100% for ratings.
Note that the total impact of any one category was determined by multiplying the mean rating by the total number of students who made a comment. For example, from Table 4, the impact of visual supports on learning was calculated by multiplying the mean which was 1.00 by the number of student s who commented about visual supports (15) for a total of 15.0.
Student performance. Students completed a pre-test and post-test created by each teacher based on the content of the learning object used in class. Questions for pre- and post-test were identical in form, but differed in the raw numbers used. The type of questions asked varied according to the goal of the specific learning objects. Some tests focussed primarily on factual knowledge while others assess higher order thinking focussing on “what-if” scenarios. The measure was used to determine student performance. Because of the wide range of learning objects used, it was not possible to assess the validity of this test data.
Finding a learning object. Thirty-six percent (n=5) of the teachers reported that finding an appropriate learning object took them less than 30 minutes. Forty-three percent (n=6) took 30 to 60 minutes. And the remaining 21% (n=3) took over an hour.
Preparing a learning object lesson. With respect to preparation for using the learning object in class, 14 % of the teachers (n=2) spent little or no time, 36% (n=5) spent less than 30 minutes, 36% (n=5) spent 30 to 60 minutes, and the remaining 14% (n=2) spent over an hour.
Using a learning object. On average, teachers used learning objects for 26.1 minutes (SD 11.5) with a range of 15 to 60 minutes. Students worked on their own on computers in all classrooms.
Reason for using a learning object. The top three reasons cited by teachers for using learning objects were to motivate students about a topic (64%, n=9), provide another way to look at a concept (57%, n=8), and review a previous concept (50%, n=7). About one third of the time, teachers used learning objects to teach a new concept (36%, n=5) or introduce a new topic (29%, n=4). Teachers rarely or never used learning objects for homework or extending a concept.
Strategies for using learning objects. Teachers in this study typically provided a brief introduction to a learning object then let the students start exploring on their own (74%, n=10). Only three teachers (21%) offered a formal demonstration of the learning object before the class used it. Forty-three percent of teachers (n=6) chose to have a class discussion about the learning object after it was used by students. Only one teacher prepared a formal handout with guiding questions to be use by students while using the learning object.
Learning. The mean rating for impact on learning (Items 1 and 2 – Appendix A) was 12.1 (SD = 2.2) or 6.1 on a 7-point scale. This indicates that most teachers agreed that the learning object had a positive impact on student learning (Table 2).
Table 2. Teacher Rating of Learning, Quality, and Engagement for Learning Objects
Quality of Learning Object. The mean rating of learning object quality (Items 3 to 5 – Appendix A) was 17.3 (SD = 1.9) or 5.8 on a 7-point scale. Most teachers agreed or strongly agreed the learning object was of good quality (Table 2).
Engagement. Teachers also rated engagement of learning objects (Items 6 to 8 – Appendix A) quite high with a mean score of 19.6 (SD = 1.2) or 6.5 on a 7-point scale. A majority of teachers, then, felt students were engaged while using learning objects. The range of learning object engagement scores was quite narrow (18 to 21) providing further support for the conclusion that teachers felt that the learning objects they selected were engaging (Table 2).
Teacher comments about learning objects. Two main themes emerged from the 22 comments teachers made about the overall impact of learning objects: learning and engagement. With respect to learning, 10 teachers commented on how learning objects kept students on task, helped review for a topic, or provided visual cues to help students learn.
All students were on task during the 90 minute lesson (note that math is usually only a 60 minute class!It provided immediate feedback to the students and offered excellent re-enforcement of the lesson.
I believe [the learning object] helped my visual-spatial and hands-on learners.
Eight teachers commented that the learning objects were motivating or engaging for their students.
The learning objects added a motivating tool, and offered a novel way of introducing a concept.
It not only helped review concepts, but it also motivated students about probability (the topic of our new unit).
The learning object helped to motivate students and get them excited about learning math.
Learning. Students rated learning objects lower than teachers with respect to learning (Items 1 to 5 – Appendix B) (M=16.9, SD = 4.9) with a mean item rating of 3.4 out 5 (or 4.8 out of 7). Students were between neutral and agree with respect to how much they felt the learning objects contributed to their learning. The range of scores was broad (5 to 25) indicating that there was considerable variability with this construct (Table 3). The mean range was 15 (SD = 3.6) out of a possible 20 point spread.
Quality of Learning Objects. Students rated the quality of learning objects (Items 6 to 9 – Appendix B) higher than the learning value, although the mean item rating was still lower than that of the teachers. The mean item rating was 3.9 out of 5 (5.4 out of 7) indicating that most students agreed that the learning objects they used were of good quality. The range of learning object quality scores (4 to 20) showed considerable variability (Table 3). The mean range was 9 (SD = 3.6) out of a possible 16 point spread.
Engagement. Ratings of learning object engagement (Items 10 to 12 – Appendix B) were moderate (M=10.8, SD = 3.3) with a mean item rating of 3.6 out or 5 (or 5.0 out of 7). In other words, as was the case with the learning construct, students were somewhere in between neutral and agree when assessing the engagement value of the learning objects they used. High variability among student engagement ratings is supported by the wide range of scores reported (3 to 15). The mean range was 9 (SD = 1.9) out of a possible 12 point spread.
Table 3. Description of Student Learning Object Evaluation Scales (LOES-S)
Student comments about learning objects. Nearly 600 student comments are summarized in Table 4. With respect to learning, the visual support that a learning object offered toward understanding a concept was rated the highest, whereas the pedagogical challenge of the learning object was rated quite low. In other words, some students liked the visual benefits of learning objects, however, more students thought the learning object they used was not challenging enough.
With respect to rating the quality of learning objects, ease of use was the highest rated feature, followed by graphics and animation. On the other hand, the quality of help and the excessive amount of text were rated the lowest.
Regarding engagement, the motivational or engaging features of learning objects were rated the highest, followed a general positive attitude toward using technology and a preference for using learning objects over other teaching methods.
Differences between pre- and post-test scores were calculated for classes where the learning object was not used for review. This yielded a total of 89 students. Student performance scores improved by an average of 40.9% from 30.5% to 71.4%. This change was significant (t = -10.59, df = 88, p < .001). The effect size (based on Cohen’s d) of 1.55 is considered “large” according to Thalheimer & Cook (2002).
Table 4. Summary of Student Comments about Learning Objects 
The purpose of this study was to examine the impact of learning objects in middle school classrooms. Six measures of use and/or effectiveness were used including teacher use, teacher ratings, teacher comments about overall impact, student ratings, student comments about what they liked and did not like, and student performance. The results from each of these measures will be discussed in turn.
Previous research is relatively silent with respect to data on finding and preparing to use learning objects; this study provides new information in this area. It appears that searching for and planning to use a learning object does not take excessive time – roughly an hour on average. Teachers did not comment that time was a problem, so we can assume that this aspect of learning object use was not a prominent issue.
It is interesting that 50 to 65% of teachers used learning objects to supplement concepts that they had already taught. Only one-third used learning objects to help introduce a new topic before a formal lesson or to teach a new concept on its own. One could speculate that teachers were being somewhat cautious with a new teaching tool in their classroom.
With respect to using learning objects, most teachers offered a brief introduction and let the students explore on their own. This finding is consistent with the behaviour of middle school teachers observed in previous research (Liu & Bera, 2005; Nurmi & Jaakola, 2006; Reimer & Moyer, 2005). In addition, 40% of teachers chose to have a class discussion after students used learning objects. Finally, most teachers did not prepare a formal handout – students were left to investigate and draw conclusions on their own. Perhaps teachers felt that, since the concept being covered was a review, there was no need to provide additional support, especially when a formal discussion was common practice after students use. It might also mean that teachers felt that the learning objects they selected should stand on their own in terms of teaching.
It is reasonable to conclude that teachers felt that the learning objects they selected were good quality, engaging tools that supported learning. Ratings were very high, often 6 on a 7-point Likert scale. On the one hand, it is somewhat predictable that teachers would rate learning objects high – after all they did select them. On the other hand, teachers rated these learning objects after they watched them being used by students in their classroom. Positive reaction from teachers in this study is consistent with previous findings for middle school teachers using learning objects (Clarke & Bowe, 2006a, 2006b).
With respect to learning, teacher comments supported the survey results. Many teachers felt their students were on task and that the learning objects offered a good review of concepts. In addition, learning objects were thought to be engaging. These comments reflect results reported in previous research (Clarke & Bowe, 2006a, 2006b). It should be noted that these comments reflect the opinions of a relatively small number of middle school teachers and that more research is needed to build and extend upon or replicate these findings.
Students, on average, “slightly agreed” that the learning objects they used were good quality, engaging learning tools that helped them learn. The students in this study appreciated learning objects less than in other reports (Clarke & Bowe, 2006a, 2006b; Freebody et al., 2007). Part of the difference might be attributed to short- versus long-term use. Learning objects in the current study were used less than 30 minutes on average compared to extensive six weeks of use reported by the Learning Federation (Clarke & Bowe, 2006a, 2006b; Freebody et al., 2007). Presumably, students can experience a wider range of learning objects and strategies over a longer time period and teachers can become more proficient at integrating them into the curriculum. It could also be speculated that the reason why student ratings were lower in this study was that learning objects developed by the Learning Federation were of higher quality. An argument could be made that a profitable next step in evaluating learning objects is to examine the specific features of learning objects that students like, coupled with the pedagogical strategies used.
It is worthwhile noting that the range of student scores was broad for all three constructs. In addition, the mean rating of student comments on “Overall Learning” was essentially zero. For any given learning object, some students liked it a lot; others disliked it a lot, even when it was the same learning object and the same teacher. Learning objects, in spite of the many potential benefits, may not be an effective learning medium for some students. The challenge that remains is to understand the nature of individual differences that exist.
Finally, teachers rated learning, quality, and engagement higher than students. This discrepancy has not been observed in previous research, although it could be a potential problem. What middle school teachers think is an effective, high quality, motivating teaching tool, may not correspond with what their students actually experience. It may also reflect the fact that teachers are so busy trying to organize and run a class that is difficult for them to accurately assess specific and overall impact with respect to learning objects. One suggestion for future research is to include observational data by a third party.
Student comments offer a more detailed look at what students liked and did not like when using learning objects. Students preferred learning objects that were easy to use and had good interactivity, visual supports, animations, and graphics. They did not like learning objects that were not challenging enough, nor did they like inadequate help features and excessive amounts of text. These comments are similar to those made by students in previous studies (Clarke & Bowe, 2006a, 2006b; Freebody et al., 2007; Reimer & Moyer, 2005).
It is clear that learning performance increased when learning objects were used. A mean jump of over 40% resulted in a large effect size. This result was observed repeatedly in previous research (Akpinar & Bal, 2006; Bower, 2005; Kong & Kwok, 2005, Liu & Bera, 2005; Nurmi & Jaakkola, 2006; Reimer & Moyer, 2005). The data also suggest that teachers’ assessment of learning impact in this study is more closely aligned to performance that than students assessment of learning. Recall that students were relatively moderate when rating the learning impact of learning objects. These modest ratings did not match significant jumps in performance.
The inconsistency between increases in student performance and essentially neutral or slightly positive student attitudes is somewhat of a mystery. One would predict that attitude and performance would be more closely aligned as in the Freebody et al. report (2007). One might expect significant gains in performance for most teaching methods, simply because the concepts being taught are new. Pre-tests scores are predictably low because students have little or no knowledge of the concepts to be taught. It is not clear from the design of this study, whether similar gains would have occurred with another teaching approach. An experimental design, like the one used by Freebody et al. (2007) is needed to establish a stronger causal link.
Another issue worth noting is the possibility of a Hawthorne effect. Students may have improved in performance due to the fact that “something new” was introduced in the classroom. However, this explanation is somewhat unlikely for two reasons. First, only 22 out of 596 comments made about learning objects referred to learning objects perceived as something different. Second, roughly 90% of the sample population reported that they felt comfortable with computers. Students in this study, then, may not have seen the use of learning objects as “something new”. Investigation of long term learning object use is necessary to establish the validity of a Hawthorne effect.
It is also important to note that improvement in student performance does not mean that learning objects were solely responsible for gains observed, regardless of a possible Hawthorne effect. As in previous studies (Clarke & Bowe, 2006a, 2006b), a number of teachers used these tools in combination with a more formal lessons or class discussion, so the influence of learning objects is partially confounded by additional teaching techniques. It is reasonable, and perhaps desirable, that learning objects are integrated within a full classroom lesson that involves multiple teaching strategies.
There are several implications for middle school educators who intend to use learning objects in their classrooms. First, it will take on average between one and two hours to find and prepare to use learning objects. Second, both teachers and students are positive about the use of learning objects in the classroom, but the impact of learning objects may vary greatly within the same classroom. Accommodations may have to be made for students with different ability and interest levels. Third, based on the behaviour of most teachers in this study, combining the use of learning objects with a formal lesson using a brief introduction and or class discussion appears to work well.
Considerable effort was made to ensure the reliability and validity of the results reported in this study, yet several limitations exist which provide opportunities for future research endeavours. First, even though the student population was relatively large and balanced in terms of gender, topics covered were restricted to mathematics and science. Different results may be observed with other subject areas. In addition, only eight teachers were involved in this study – a future larger scale investigation is warranted to increase generalizability.
Second, while the overall impact of learning-object lessons on student performance was large, the impact of specific instructional strategies was not examined. In other words, we don’t know which teaching strategies work best with learning objects. Perhaps different strategies work with different kinds of learning objects.
Third, short term use of learning objects was assessed in this study. It is reasonable to follow the pathway of the researchers (Clarke & Bowe, 2006a, 2006b; Freebody, et al., 2007) investigating learning objects created by the Learning Federation (see http://www.thelearningfederation.edu.au/default.asp). However, access to technology and learning objects is far more limited in many countries where well developed, concentrated collections of learning objects are not readily available.
Finally, the study was designed to look at the overall impact of learning objects – the impact of specific kinds of learning objects was not looked at. It is possible that certain categories of learning objects may have decidedly different impacts on learning. For example, a tool-based learning object (e.g., http://www.article19.com/shockwave/oz.htm) may be good for exploring a new concept where as a question-and-answer learning object (e.g., http://amazing-space.stsci.edu/resources/explorations/trading/directions.html ) may be best for review. Developing and evaluating a classification system for learning objects could be an important next step in learning object research.
The results of this study support and add new information to the relatively small knowledge base that exists with respect to the use of learning objects in middle school. Perhaps the biggest contribution was methodological rigour. Three forms of data collection were combined to analyse the impact of learning objects: teacher attitude, student attitude, and student performance. In addition a relatively large, diverse sample of students and multiple teachers were assessed using a number of different learning objects. Finally, reliable and valid measures were used. This comprehensive approach to evaluation is rarely observed in learning object research. The positive results reported are consistent with past studies examining student attitudes, teacher attitudes, and student performance with respect to learning objects.
Previous researchers have focussed on student performance and, to a lesser extent, student attitudes about learning objects. Relatively little is known about the teacher’s viewpoint. This paper looked at eight teachers who use learning objects in 14 middle school classrooms. They were very positive about the learning value, quality, and engagement of learning objects and these attitudes were consistent with significant gains in learning performance. In fact, the teacher impressions were more closely aligned with student performance than those of students.
This study also provides some insight into how teachers prepare for and use learning objects, an area that has not been examined before. Most middle school teachers take anywhere from one to two hours to produce a lesson that is often focussed on review, and to a lesser extent introducing new concepts.
Agostinho, S., Bennett, S., Lockyear, L., & Harper, B. (2004). Developing a learning object metadata application profile based on LOM suitable for the Australian higher education market. Australasian Journal of Educational Technology, 20(2), 191-208.
Akpinar, Y. & Bal, V. (2006). Student tools supported by collaboratively authored tasks: The case of work learning unit. Journal of Interactive Learning Research, 17 (2), 101-119.
Albanese, M. A., & Mitchell, S. A. (1993). Problem-based learning: A review of the literature on its outcomes and implementation issues. Academic Medicine, 68, 52-81.
Bower, M. (2005). Online assessment feedback: Competitive, individualistic, or…preferred form! Journal of Computers in Mathematics and Science Teaching, 24(2), 121-147.
Bruner, J. (1983). Child's talk. Learning to use language, Toronto, Canada: George J. McLeod Ltd.
Bruner, J. (1986). Actual minds, possible worlds. Cambridge, MA: Harvard University Press.
Brush, T. & Saye, J. (2001). The use of embedded scaffolds with hypermedia-supported student-centered learning. Journal of Educational Multimedia and Hypermedia, 10 (4), 333-356.
Butson, R. (2003). Learning objects: weapons of mass instruction. British Journal of Educational Technology, 34(5), 667-669.
Carroll, J. B. (1990). The Nurnberg funnel. Cambridge, MA: MIT Press.
Clarke, O., & Bowe, L. (2006a). The learning federation and the Victorian department of education and training trial of online curriculum content with Indigenous students. 1-14. Retrieved May 31, 2008 from http://www.thelearningfederation.edu.au/verve/_resources/tlf_detvic_indig_trial_mar06.pdf
Clarke, O., & Bowe, L. (2006b). The learning federation and the Victorian department of education and training trial of online curriculum content with ESL students. 1-16. Retrieved May 31, 2008 from http://www.thelearningfederation.edu.au/verve/_resources/freebody_final_report_2007.pdf
Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. In L. B. Resnick (Ed.), Knowing, learning, and instruction (pp. 453-494). Hillsdale, NJ: Erlbaum Associates.
Duval, E., Hodgins, W., Rehak, D., & Robson, R. (2004). Learning objects symposium special issue guest editorial. Journal of Educational Multimedia and Hypermedia, 13(4), 331-342.
Freebody, P., Muspratt, S., & McRae, D. (2007). Evaluating the learning federation’s online curriculum content initiative. 1-144. Retrieved May 31, 2008 from http://www.thelearningfederation.edu.au/verve/_resources/freebody_final_report_2007.pdf
Gadanidis, G., Gadanidis, J., & Schindler, K. (2003). Factors mediating the use of online applets in the lesson planning of pre-service mathematics teachers. Journal of Computers in Mathematics and Science Teaching, 22(4), 323-344.
Haughey, M., & Muirhead, B. (2005). Evaluating learning objects for schools. E-Journal of Instructional Sciences and Technology, 8(1). Retrieved June 1, 2007 from http://www.usq.edu.au/electpub/e-jist/docs/vol8_no1/fullpapers/eval_learnobjects_school.htm
Kay, R., & Knaack, L. (2005). Developing learning objects for secondary school students: A multi-component model. Interdisciplinary Journal of Knowledge and Learning Objects, 2005(1), 229-254.
Kay, R. H., & Knaack, L. (2007a). A systematic evaluation of learning objects for secondary school students. Journal of Educational Technology Systems, 35(4), 411-448.
Kay, R. H. & Knaack, L. (2007b). Evaluating the learning in learning objects. Open Learning, 22(1), 5-28.
Kay, R. H., & Knaack, L. (2007c). Teacher evaluation of learning objects in middle and secondary school classrooms. Manuscript submitted for publication. Retrieved Feb 1, 2008 from http://faculty.uoit.ca/kay/papers/LOES_Teacher_2007.pdf
Kay, R. H., & Knaack, L. (in press). Assessing learning, quality and engagement in learning objects: the learning object evaluation scale for students (LOES-S). ETR&D. Retrieved May 31, 2008 from http://faculty.uoit.ca/kay/papers/LOES_Student_2007.pdf
Kong, S. C., & Kwok, L. F. (2005). A cognitive tool for teaching the addition/subtraction of common fractions: A model of affordances. Computers and Education, 45(2), 245-265.
Liu, M., & Bera, S. (2005). An analysis of cognitive tool use patterns in a hypermedia learning environment. Educational Technology, Research and Development, 53(1), 5-21.
McGreal, R. (2004). Learning objects: A practical definition. International Journal of Instructional Technology and Distance Learning, 1(9). Retrieved August 5, 2005 from http://www.itdl.org/Journal/Sep_04/article02.htm
Nurmi, S., & Jaakkola, T. (2006). Effectiveness of learning objects in various instructional settings. Learning, Media, and Technology, 31(3), 233-247.
Parrish, P. E. (2004). The trouble with learning objects. Educational Technology Research & Development, 52(1), 49-67.
Rehak, D., & Mason, R. (2003). Chapter 3: Keeping the Learning in Learning Objects. Journal of Interactive Media in Education, 2003(1). Retrieved July 1, 2005 from http://www-jime.open.ac.uk/2003/1/reuse-05.html
Reimer, K. & Moyer, P.S. (2005). Third-graders learning about fractions using virtual manipulatives: A classroom study. Journal of Computers in Mathematics and Science Teaching, 24(1), 5-25.
Sedig, K., & Liang, H (2006). Interactivity of visual mathematical representations: Factors affecting learning and cognitive processes. Journal of Interactive Learning Research. 17(2), 179-212.
Thalheimer, W., & Cook, S. (2002). How to calculate effect sizes from published research articles: A simplified methodology. Retrieved July 14, 2007 from http://work-learning.com/effect_sizes.htm
Vygotsky, L. S. (1978). Mind in society. Cambridge, M.A.: Harvard University Press.
Wiley, D. A. (2000). Connecting learning objects to instructional design theory: A definition, a metaphor, and a taxonomy. In D. A. Wiley (Ed.), The Instructional Use of Learning Objects: Online Version. Retrieved July, 1, 2005, from http://reusability.org/read/chapters/wiley.doc
Wiley, D., Waters, S., Dawson, D., Lambert, B., Barclay, M., & Wade, D. (2004). Overcoming the limitations of learning objects. Journal of Educational Multimedia and Hypermedia, 13(4), 507-521.
1 – Strongly Disagree / 2 – Disagree / 3 – Slightly Disagree / 4 – Neutral /
5 – Slightly Agree / 6 – Agree / 7 – Strongly Agree
What was the overall impact of the learning object on your lesson?
1 – Strongly Disagree / 2 – Disagree / 3 – Neutral / 4 – Agree / 5 – Strongly Agree
What, if anything, did you LIKE about the learning object?
What, if anything, did you NOT LIKE about the learning object?