Validation of a Questionnaire Assessing Students’ Self-Directed and Collaborative Learning With and Without Technology in Canadian Middle School Classrooms Validation d’un questionnaire évaluant l’apprentissage autonome et collaboratif des élèves, avec et sans technologie, dans les salles de clas

In the current study, the researchers examine the validity of a questionnaire assessing students’ perceptions of their self-directed and collaborative learning with and without technology with a group of Canadian middle school students. Lee and colleagues (2014) developed the 18-item questionnaire to assess high school students’ perceptions of their learning. Three hundred and twenty middle school students from across Alberta, Canada completed the questionnaire. The results of a confirmatory factor analysis revealed that the questionnaire did not have sufficient model fit. The researchers used a jackknifing procedure to systematically remove four items in order to achieve a psychometrically sound questionnaire. The results suggest that the reduced questionnaire is a useful self-report instrument for assessing Canadian middle school students’ perceptions of their learning.


Introduction
Among other 21st century skills, educators have identified collaborative learning (CL) and self-directed learning (SDL) as important skills for success in our current global knowledge society (Henry, 2015;Partnership for 21st Century Skills, 2011). SDL is defined as "any increase in knowledge, skill, accomplishment, or personal development that an individual selects and brings about by his or her own efforts using any method in any circumstances at any time" (Gibbons, 2002;p. 2). SDL is associated with numerous meaningful learning outcomes and skills, such as critical thinking and logical thinking skills (Willett, Yamashita, & Anderson, 1983), and has long been recognized as an essential component for educational growth and lifelong learning (Garrison, 1997;Gibbons, 2002). A complementary skill set, CL occurs when two or more students work together towards a common learning goal (Dillenbourg, 1999). As students engage in CL, they serve each other as a source of support, motivation and learning (Bolhuis, 2003;Schwartz, Tsang, & Blair, 2016). CL provides students with the chance to analyze and exchange information, and engage in the learning material in a way not offered during individual learning, fostering students' critical thinking skills, and greater conceptual understanding of course material (Hussain, Anwar, & Majoka, 2011;Schwartz, Tsang, & Blair, 2016;Terenzini, Cabrera, Colbeck, Parente, & Bjorklund, 2001).
Information and communication technology (ICT) is used in educational settings for a variety of purposes. The use of ICT can facilitate access to both information and online expertise for the purpose of pursuing learning goals and interests (Teo, et al., 2010). Current technology supported learning practices include technology-rich, network-enhanced, blended, and virtual learning environments (Resta & Laferriere, 2007). Activities within technology-supported classrooms usually include using the Internet to find information and participating in group discussions about learning tasks with peers in the classroom and in online environments, as well as traditional non-technological activities, such as accessing information through textbooks and direct instruction (Lee, Tsai, Chai, & Koh, 2014). Open education websites and learning portals provide learners with a plethora of informational material and opportunities to learn. Wikipedia, Youtube, and Khan Academy have all been identified as examples of websites used by selfdirected learners (Bonk, Lee, Kou, Xu, & Sheu, 2015). Tools such as email, online chat forums, blogs, wikis, videoconferencing systems, and course management systems have all been used to support online CL (Resta & Laferriere, 2007). Other tools such as Wikispaces, Moodle, and social media platforms also have built-in features to facilitate both CL and SDL (Domalewska, 2014).
Research has demonstrated that ICT-supported CL (i.e., completing a task together using a shared computer) positively impacts learning outcomes. Blaye and colleagues (1991) examined the CL of 11-year-old students (N=39) during user-user interaction while completing a timed computer-based problem-solving game. Children either worked independently or in pairs, before playing the game independently. The students who first worked collaboratively were found to be twice as likely to complete the task successfully compared to children who worked on the task alone. Importantly, students who had previously worked in pairs were twice as likely to complete the task successfully when later working alone, compared to children who had only worked alone, despite having the same amount of exposure to the task (Blaye et al., 1991).
While ICT-enhanced CL and SDL appears to impact student learning, it has also been proposed that students may require SDL and CL skills before being introduced to ICT. Lee and colleagues (2014) found a positive relationship between SDL and CL skills when applied with and without technology. They concluded that students' ability to engage in SDL and CL without technology predicted their ability to engage in SDL and CL with technology. Accordingly, these findings suggest that teachers, researchers, and educational technologists should be concerned about their students' SDL and CL skills in both technological and non-technological contexts so that students may be able to take full advantage of the benefits of ICT.
As ICTs are increasingly being used within classrooms globally, there is a need for measures to assess the extent of students' CL and SDL. Students' perceptions of their learning ability also need to be considered. Students' perceptions and beliefs about their efficacy to learn affect student motivation and ability to succeed academically (Schwartz, Tsang, & Blair, 2016). Furthermore, by examining the relationship between students' perceptions of their SDL and CL with and without technology, teachers can effectively adapt pedagogical practices and integrate ICT within their classrooms. With this in mind, Lee and colleagues (2014) created a questionnaire designed to assess student's perceptions of their SDL and CL with and without technology. Lee and colleagues (2014) constructed the survey questions from existing instruments, designed by Shell et al. (2005), Goh et al. (2013), and Teo et al. (2010). The authors pilot-tested for face and content validity by consulting two professors in educational technology and five high school teachers from Singapore. The authors then validated the four-factor structure of the 18 item 7 point Likert scale questionnaire with a group of high school students in Singapore (Lee et al., 2014). The findings revealed that the four factors had sufficient internal consistency with an overall coefficient alpha of 0.95. Coefficients above 0.7 are acceptable estimates of internal consistency for research purposes (Kaplan & Saccuzzo, 2013). The results of a confirmatory factor analysis revealed the questionnaire to have good construct validity, convergent validity, discriminant validity, and reliability with high school students in Singapore (Lee et al., 2014).
The use and interpretation of any questionnaire is only valid for the purposes and populations for which there are evidence of validity (Downing, 2003). Lee and colleagues (2014) provided validity evidence for the interpretation of the questionnaire results for the perceptions of SDL and CL with and without technology of Singapore high school students. Additional evidence is needed to support its use for interpretations with other populations. Within a Canadian context, educational policies simultaneously support both CL and SDL learning processes and the use of technology (Alberta Education, 2013). Yet, there is no evidence of validity to suggest that Lee and colleagues measure is appropriate to assess Canadian students' perceptions of their CL and SDL in ICT-supported learning environments.
In the present study, the researchers examine the validity and reliability of the instrument created by Lee and colleagues (2014) for use with Canadian middle school students. The researchers were guided by the following question: Does the Lee and colleagues (2014) questionnaire have adequate evidence of reliability and validity for use with middle students in Alberta, Canada?

Participants
Three hundred and twenty-five students (52.6% male) from eight schools across five school jurisdictions were asked to complete the questionnaire as part of a larger study. Of these students, a small number (N=5) did not complete all the items in the questionnaire. Little's (1988) missing completely at random (MCAR) test was non-significant (chi-square = 9.892, df = 26, sig. = 0.998), revealing that the values were missing at random. As only a few cases have missing values and they are missing at random, all five cases with missing values were deleted from the data set (Tabachnick & Fidell, 2014). A total of 320 participants remained.
Students were enrolled in grades 5 to 9, with the majority of students in Grade 8, (3.1% Grade 5, 2.8% Grade 6, 16.3% Grade 7, 55.3% Grade 8, 22.2% Grade 9) inclusive (i.e., students with and without special education needs) ICT-supported classrooms. Teachers reported that the inclusive classroom environments included many students with mild to moderate learning needs, ranging from 10% to 85% per classroom depending on the school. Participating schools were located in both rural and urban population centers across various regions of the province. The majority of the students (64.4%, n = 206) came from English speaking families.

Procedure
The questionnaire was administered to students mid school year as part of a larger battery of questionnaires. Students completed all questionnaires within their respective classrooms using the REDCap online survey administration and data capture program (Harris et al., 2009). To complete the questionnaires, a research assistant provided instruction to students via video embedded within the online platform. Every item was read aloud via the computer monitor to the student.

Measure
The 18-item questionnaire, created by Lee and colleagues (2014), aims to assess students' perceptions of their ability to engage in SDL and CL with and without technology within their ICT-supported classroom environments. Each scale contains items that prompt students to rate their learning skills on a 7-point Likert scale (i.e., 1-strongly disagree, 7-strongly agree). Scale scores are calculated by taking the average of the ratings for items included within the scale (Lee et al., 2014). The four scales are described below: 1. Self-directed learning without technology (SDL) scale. This 4-item scale assesses students' perceptions of the extent to which they take an active role in their learning in face-to-face non-technological settings. 2. Collaborative learning without technology (CL) scale. This 5-item scale assesses students' perceptions of the extent to which they participate in group discussions and learning within face-to-face non-technological settings. 3. Self-directed learning with technology (SDLT) scale. This 5-item scale assesses student's perceptions of the extent to which they take an active role in their learning in ICTsupported classroom settings.
4. Collaborative learning with technology (CLT) scale. This 4-item scale assesses student's perceptions of the extent to which they use ICTs to participate in group discussions and learning

Results
Statistical analyses were conducted using the Statistical Package for the Social Sciences (SPSS for Mac: Version 24) software and Analysis of Moment Structures (AMOS) add-on for SPSS for Windows (Version 24).

Confirmatory Factor Analysis
Confirmatory factor analysis (CFA) with maximum likelihood estimates was used to establish the validity of the 4-factor measurement model described by Lee and colleagues (2014). The maximum likelihood method was appropriate, as there was no evidence of non-normality within the data. No items or subscales were found to have a skewness value larger than an absolute value of 3.0 or kurtosis value with an absolute value larger than 8.0 (see Table 1), which are the cut-off values, recommended by Kline (2011). Model fit was assessed using a range of indices representing different aspects of model fit (Hooper, Coughlan, & Mullen, 2008). Model chi-square (X 2 ) value was calculated. While this is the traditional measure of model fit, chi-square is sensitive to sample size, assumes multivariate normality and almost always rejecting models with larger sample sizes (Hooper, Coughlan, & Mullen, 2008). As such, the normed chi-square, chi-square/degrees of freedom (X 2 /df), will also be used. A range of no more than 3.0 is often suggested to indicate an acceptable fit between the hypothesized model and the sample data, although recommended ranges vary from as high as 5.0 to as low as 2.0 in the literature (Hooper, Coughlan, & Mullen, 2008;Teo, et al., 2010). Root mean square error of approximation (RMSEA), normed-fit index (NFI), and comparative-fit index (CFI) statistics were also calculated. The RMSEA is one of the most informative statistics, as it is sensitive to the number of estimated parameters within the model. RMSEA values of less than 0.08 reflect good model fit. The NFI ranges from 0 to 1, with a cut-off point of 0.9 and greater, indicating acceptable model fit. The CFI statistic also ranges from 0 to 1 with a cut-off point of 0.95 and greater, indicating good model fit (Hooper, Coughlan, & Mullen, 2008).
The initial analysis of the measurement model as proposed by Lee and colleagues (2014) did not suggest acceptable model fit. Inspection of the modification indices indicated that model fit might be improved by correlating four sets of error variances. The model fit, improved through the use of correlation of error terms, was assessed using a range of indices: Χ 2 = 323.02, df=125, p< 0.01, Χ 2 /df= 2.58, NFI=0.87, CFI=0.92, and RMSEA=0.07 (90% CI of 0.06-0.08). The results of some indices suggest a mediocre model fit, while others failed to find an acceptable fit.
To evaluate the validity and reliability of the structure, the composite reliability (CR) and the average variance extracted (AVE) were calculated. CR is a measure of internal consistency of the latent constructs (Fornell & Larcker, 1981). A CR value of greater than 0.7 indicates good internal consistency (Hair, Black, Babin, & Anderson, 2010). The AVE indicates the average percentage of variation explained by items for each latent construct and provides us with an indictor of convergent validity (Fornell & Larcker, 1981). An AVE value greater than 0.5 indicates good convergent validity. An AVE score that is less than 0.5 indicates that the variance due to error is larger than the variance explained by the construct, suggesting concerns with convergent validity. Discriminant validity is achieved when the items within a factor account for more variance than the factor shares with other constructs in the model. To establish discriminant validity, the square root of the AVE for each factor is compared with the factor's inter-construct correlations. The square root of the AVE, which is based on the standardized loadings between an item and its factor, should be larger than the factor's correlations with other factors. This comparison reveals whether the factor is sufficiently distinct from the other factors (Fornell & Larcker, 1981;Henseler, Ringle, & Sarstedlt, 2015). Table 2, suggest good internal consistency, but indicate that the individual SDL and SDLT items do not correlate well with each other within their factor, suggesting poor convergent validity. SDL and SDLT factors are not well explained by their observed items. Table 3 contains the inter-correlation between factors with the square root of the AVE value along the diagonal. Examination of this table reveals discriminant validity concerns for the SDL and SDLT scales. The square root of the AVE for SDL is smaller than the correlations between SDL and CL, and the square root of the AVE for SDLT is smaller than the correlations between SDLT and CLT. The SDL scale is not sufficiently distinct from the CL scale, whereas the SDLT scale is not sufficiently distinct from the CLT scale. Overall, the results suggest that the measurement model has mediocre model fit but only when correlations of error terms are added to the model. The proposed questionnaire structure was found to have good composite reliability. Concerns were found with the convergent and discriminant validity of the SDL and SDLT scales.

Jackknifing Procedure
In an effort to achieve validity for the questionnaire, a jackknife approach, as described by Larwin and Harvey (2012), was performed to systematically reduce the number of items until the resulting model displayed good model fit, and adequate composite reliability, convergent validity, and discriminant validity, as measured by the CR and AVE. Following the systematic removal of items from the original model, the removal of item CLT4, resulted in a model that demonstrated the best model fit. After the removal of item CLT4, the model fit was assessed using a range of indices: Χ 2 = 2368.35, df=113, p< 0.01, Χ 2 /df= 2.11, NFI=0.89, CFI=0.94, and RMSEA=0.06 (90% CI of 0.05-0.07). The revised model has acceptable fit with three of the indices (Χ 2 , Χ 2 /df and RMSEA) revealing good model fit. A summary of the model fit indices is displayed in Table 4. To evaluate the validity and reliability of the structure, the CR and AVE were calculated. The results, displayed in Table 5 suggest that all scales have good composite reliability but that the convergent validity concerns remain present for SDL and SDLT. This suggests that the SDL and SDLT factors are not well explained by the items that were intended to measure these constructs. The inter-construct correlations found in Figure 1 reveal that concerns with discriminant validity of the SDLT scale were resolved. However, discriminant validity with the SDL scale remains. The SDL scale continues to be not sufficiently distinct from the CL scale.  Figure 1. Inter-factor correlations and the square root of the AVE for each resulting model.
Following the same approach, the jackknifing procedure was repeated several times until a model with a good model fit and acceptable reliability and validity was achieved. After each item removal, the removal of another item was tested systematically, and the item resulting in the best model fit was removed. The model fit indices for each model are located in Table 4, and the CR and AVE results are in Table 5. Figure 1 contains inter-construct correlation and square root of the AVE information for each model. After the removal of item CLT4 from the model, item SDL3 was removed and the model fit and validity was assessed. The model had adequate fit and sufficient composite reliability. As it still contained the same convergent and discriminant validity concerns, the jackknife procedure was repeated, and item SDLT4 was removed. This resulting model had good model fit and sufficient composite reliability, and the AVE indicated a resolution of the concerns with convergent validity for the SDLT scale. However, concerns with convergent and discriminant validity remained for the SDL scale. The jackknife procedure then resulted in the removal of item CL5. With the removal of item CL5, the SDL factor displayed adequate convergent validity and concerns with discriminant validity were resolved. This model also had good model fit, and good composite reliability. The jackknifing procedure was not continued as a model with good model fit, and adequate reliability and validity was reached. With the removal of items CLT4, SDL3, SDLT4, and CL5, the questionnaire reached an acceptable level of reliability and validity for use with Canadian middle school students. The reduced questionnaire is found in Appendix A and the resulting measurement model is found in Appendix B. Descriptive statistics for each of the new reduced scales is found in Table 6.

Discussion
The development of SDL and CL has been identified as a major educational goal for 21 st century learners (Henry, 2015;Partnership for 21st Century Skills, 2011;Voogt & Roblin, 2012). Students are expected to develop these learning processes in traditional face-to-face classroom settings and in technology-supported learning environments. Lee and colleagues (2014) developed a questionnaire to assess students' perceptions of their learning processes across both face-to-face and ICT-supported learning environments. The four factors of the questionnare (CL, SDL, CLT, SDLT) can help educators' understand how students view learning with ICT and allow for better support for learning across all environments (Lee, Tsai, Chai, & Koh, 2014).
The principal aim of the present study was to examine the validity of Lee and colleagues' (2014) questionnaire for use with Canadian middle school students. Confirmatory factor analysis was used to test the validity and the reliability of the factorial structure of the questionnaire. By testing the factorial structure of each questionnaire item, the extent to which each items measure the specfic factor they were designed to measure can be determined (Byrne, 2010). Test of the fit between the factorial model and the data indicate the degree to which the items measure their respective constructs, and provides an indication of how well the instrument supports the underlying theory (Hooper, Coughlan, & Mullen, 2008). Based on the model fit indices and measures of reliability and validity, the factorial model did not fit with our data, which raised concerns about validity of the original questionnaire for use with our sample of Canadian middle school students.
In order to establish a valid and reliable questionnaire for use with our sample of Canadian middle school students, four items (CLT4, SDL3, SDLT4, and CL5) were removed through a jackknifing procedure. The jackknifing procedure involves removing one item at a time and estimating model fit for each resulting model. The selection of the item to be removed from the questionnaire is based on which model produces the best model fit estimations. The process of removing items one by one continues until several conditions are met: the original factors continue to be explained by three observed items, the reduced model maintains structural integrity, the reduced model correlates with the primary factor model at a level greater or equal to 0.95, and the resulting reduced model has good model fit (Larwin & Harvey, 2012). The CLT 4 item, In this class, my classmates and I actively work together to construct ICT-based documents (e.g. presentation slides, web pages), initially loaded on the SDLT scale. Thus, its removal allowed for a parsimonious model fit and resolved the discriminant validity concerns with SDLT. Removing the CLT4 item reduced the correlation between the CLT and SDLT scales, allowing for the SDLT scale to be sufficiently distinct from the CLT scale. Removing the SDL3 item ("In this class, I make plans for how I will study.") did improve model fit, but did not resolve any of the validity concerns within the model. However, removing the SDLT4 item, In this class, I find out more information on the Internet to help me understand my lessons better, improved the convergent validity of the SDLT scale, implying that the SDLT4 item was not sufficiently related to the other items and was not adequately measuring the same construct. Last, by removing the CL5 item, In this class, my classmates and I actively talk about what to do during group work, resolved concerns with convergent and discriminant validity of the SDL scale. It appears that the CL5 item was related to the SDL scale.
The results from the reduced questionnaire suggest that Canadian middle school students are easily engaging in SDL, CL, and SDLT within their classroom, as indicated with means over 5 on each scale. Students reported less perceived ability to engage in CLT within their classroom. Students and educators may require additional support and resources in order to engage in ICT-supported CL.

Implications for Practice
SDL and CL are important 21 st century learning skills that are increasingly being used in ICT-supported classrooms. Students are required to apply SDL and CL in context that include educational technology. As a result, educational practitioners such as teachers, educational researchers and educational technologists need instruments to assess these skills. The reduced questionnaire can be used by practitioners to collect self-report information from their middle school students. The questionnaire was initially developed by Lee and colleagues (2014) through a theory-driven approach and found to be valid and reliable with high school students in Singapore. The reduced questionnaire presented in this study can be implemented with middle school students in a Canadian context as the psychometric jackknifing procedure allows researchers to produce a reduce questionnaire that is valid and reliable.

Limitations and Future Research
This study did not collect sufficient information regarding student learning and classroom practices in order to provide anything more than speculation as to why the original questionnaire did not have sufficient reliability and validity for use with middle students in Canada. Differences between the pedagogical practices in Singapore and Canada, and across high school and middle school classrooms may provide a hypothesis for why some items were retained while others removed in order to reach adequate evidence of validity with our sample. A comparison of the questionnaire results between the samples of Canadian middle students and the Singapore high school students from the Lee and colleagues (2014) study reveal some differences and commonalities. Both the Singapore high school students and the Canadian middle school students describe their SDL and CL without technology as slightly agreeable with means above 5 across both scales for both samples. Additionally, both Singapore high school students (mean=4.15, SD=1.52) and Canadian middle school students (mean=4.27, SD-1.24) indicate that they are less readily engaged in collaborative activities without technology, with means corresponding best with the 'agree nor disagree' options of the 7-point Likert scale. The Singapore high school students report that they do not readily engage in SDL with technology in their classrooms (mean=3.98, SD=1.51). However, Canadian middle school students report that they do engage in self-directed activities with technology (mean=5.21, SD=1.24).
Further research may wish to consult with teachers and students to better understand how self-directed learning and collaborative learning with and without technology is facilitated in middle and high school classrooms in both Canada and Singapore. Consultation with teachers may provide valuable information as to how CL and SDL items may or may not align with their teaching practices in middle school classrooms. The reduced questionnaire is a useful self-report instrument for assessing Canadian middle school students' perceptions of their learning in ICTsupported classrooms. Caution should be applied to its use with a variety of samples as the original questionnaire was not valid with our sample and the reduced questionnaire requires further validation across an array of samples. Future research is needed to understand the validity of the questionnaire across both different cultural groups and age groups. Lee and colleagues (2014) were driven by theory when they created their self-report questionnaire for use with high school students in Singapore. However, the questionnaire itself does not assess all 21 st century learning competencies (Lee et al., 2014). The Partnership for 21 st Century Learning's framework includes collaborative and self-directed learning among other skills such as creativity, critical thinking and problem solving, and communication (Partnership for 21st Century Skills, 2011). While the reduced questionnaire was shown to be a useful tool to measure SDL and CL, a comprehensive self-evaluation of Canadian students' 21 st century learning competencies requires more than just the reduced questionnaire. Future research should investigate a broad range of 21 st century learning skills.