Author
Terry Anderson is a professor and Canada Research chair in Distance Education at Athabasca University – Canada’s Open University. He teaches in the Masters of Distance Education program. Correspondance regarding this commentary may be sent by e-mail: Terry@athabascau.ca
Philip Abrami and his colleagues at Concordia University have done an admirable job of over viewing the complex state of research and related perceptions of e-learning in Canada in the early years of the 21 st Century. The work of these scholars at quite effectively aggregating and synthesizing is commendable given the emergent, regionally disparate, complex and most critically, under researched domain that they have reviewed. Unfortunately, what stands out most readily for me (and echoed by the authors) is the poverty of data, sound research, and systematic analysis of the considerable social, economic and educational impact that has accompanied the rapid integration of e-learning technologies into both formal and informal learning institutions and organizations. This dearth of data is especially ironical and perplexing given the positive views of almost all respondents (from educators to policy makers, from students to the general public) identified in the review. Canadians seem to very clearly see e-learning as a critically important component of life and learning. However that interest has not evolved into an active e-learning research and development and knowledge dissemination program
The report begins with a definition of e-learning—one developed by the Canadian Council for Learning (CCL) who are sponsors of the study. Their definition seems to adequately define the diverse set of practices and technologies in actual use. “E-learning is the development of knowledge and skills through the use of information and communication technologies (ICTs) to support interactions for learning—interactions with content, with learning activities and tools, and with other people” (Rossiter, 2002). I was especially pleased to note the focus on interaction—with both humans, activities and content that is included in the definition. In the tradition of John Dewey, Vygotsky and Paiget, I have argued and tried to further delineate the various types of interactions and noted its effect on learning motivation, achievement and persistence in educational contexts (Anderson, 2003) including those created on campus and in online environments. I also appreciated the acknowledgement by the authors and by CCL of the convergence of tools and practice between formerly distinct modes of distance and campus based education into models in which e-learning plays significant, if not dominant importance. I doubt that I could craft a better definition, but I note that the breadth and extensiveness of the definition, allows inclusion of very diverse forms of instruction and learning ranging from the use of a computer in a classroom for drill and practice to completion of complete graduate degrees online, with no face-to-face meetings. Both of these are examples of e-learning but it stretches my mind past the point of credibility to think how “what works” in one context applies to “what works” in the other.
The review attempts to answer a quixotic question that plagues every intervention in formal and informal learning. What practices and procedures relating to e-learning use are most effective in promoting learning? Given the pragmatic nature of many Canadian educators, politicians and parents, this is an important question. It follows that a thorough reading of a broad range of research, professional and popular writing should provide both specific and broad policy guidelines, best practices and specific activities that work effectively with e-learning technologies and techniques. Unfortunately, the authors admit in the executive summary that “it seems that studies that can help us understand ‘what works’ in e-learning are underrepresented in Canadian research literature.” Before reviewing the attributes of the study it is worth a short digression into the nature of the research question.
“What works” are American code words for a particular type of epistemological worldview, with strong roots in the empirical and positivist traditions of scientific research. Of course, many forms of knowledge can aide teachers, learners and policymakers in determining what works in their context; however, in recent years these words have been high-jacked to mean a particular type of evidence—thereby excluding other types. A quick visit to the American “what works” web site (http://www.whatworks.ed.gov/) reveals that evidence is only considered useful in determining “what works” if it clearly exhibits “a causal effect on student outcomes. In research that focuses on causal effects, experimental research designs provide the strongest evidence of what works, followed by certain types of quasi-experimental research designs.” I have two major concerns with this approach. The first is the misappropriation of the terms “what works” to exclude other forms of knowledge that may provide equal or even greater insight into determining the effect or impact of an intervention. For example the contextual influences of an individual or group’s state of health, motivation, access, time availability and other internal or external conditions may have tremendous effect on “what works”, yet if the subjects were not subjected to control (an almost impossible task in real world contexts), these large contextual variables are often ignored. Second, the absolute sense of “what works” implies that the particular intervention will work regardless of the context in which it is used. Education and learning happen in wide variety of contexts determined by a very wide range of cultural, economic, individual, discipline and other variables ( Figueiredo , 2005) . Neither lifelong learning nor formal educational schooling take place in controlled laboratories with identical subjects; to suggest that there are interventions that “work” regardless of the state of these variables, is to disregard the emergent complexity of any learning context.
Fortunately, the authors are veterans of this ongoing “paradigm war” and so have wisely included reviews of qualitative, narrative and opinion pieces. They have made a valiant and productive attempt to shoehorn these diverse artefacts into an understandable whole, by applying a series of tags and structured vocabularies that allow broader trends, effects and conclusions to be extracted from the whole than from any individual article or study. Of course the more personal, naturalistic and qualitative the article is the harder it is to stuff into a set of common criteria—and especially criteria that permit mathematical manipulation to determine combined effect sizes.
The authors follow two methodological approaches. The first allows use of the qualitative pieces by having coders classify each study bilaterally among a series of broad classifications. For example, was e-learning use positively, negatively or neutrally associated with a diverse set of educational outcomes including motivation, achievement , interactivity, attrition, and so on. The means of these classifications were calculated, averaged to produce a general effect of e-learning and then the correlations of the studies were cal cul ated based upon the media used, sector, locations, designs and technology types and other variables asso ciated with each study. Many of the variables did not produce significant differences on the overall effect of e-l earning, but some did. As expected, studies that noted infrastructure needs, professional development, and were dependent upon e-learning (rather than being an optional add- on) had significant positive effect on the combined e-learni ng outcome. I was also pleased to note that use of e-learning for communication applications and for student (as opposed to teacher) use of the technol ogies were associated with higher positive effect.
The second more classic multi-source technique is a meta-analysis in which an overall effect size is calculated by combining the effect size of numerous, similar studies. One can sense the frustration of the authors as 2,042 documents scanned are reduced to 726 of real value to the question from a Canadian context, then reduced to 152 with empirical data and finally to only 17 studies of an experimental or quasi-experimental design that provided an effect size and sample size n’s—critical factors in allowing entry into a traditional meta-analysis. As expected, even here heterogeneity among studies was significant, which illustrates that the variation amongst contexts and applications of e-learning is very high. As a result, one can not place a great deal of faith in the very small positive effect size that was calculated from the data from these 17 studies. As is now traditional in all educational meta-analysis studies that I have seen, the authors complain bitterly about the absence of quantity of useable studies and a host of methodological deficiencies. What is often left un-discussed is the validity of combining studies from very disparate contexts; each with only the common characteristic that e-learning is used in some way.
The report synthesizes five consensus positions amongst targeted groups of researchers, reviewers, policy makers and practitioners. The first is the focus on individual readiness, which underlines the need for individuals (and I would argue institutions) to have access, skill and attitude conducive to availing themselves of the affordances of this new net-enabled context. This conclusion underlines the need for e-learning to meet the high expectations of Net generation youngsters as well as for those who are immigrants to this very foreign, confusing and sometimes frightening new context for learning. The second conclusion echoes decades of instructional design research and underscores that it isn’t the tool, but how the tool is conceived of and used that makes the difference for learning. E-learning is much more than “putting the lectures online” and requires both pedagogical and epistemological adaptation to maximize its effectiveness. From this need for change follows the third conclusion noting that teachers (and others involved in the education business) require focused professional development. The fourth consensus notes the expansion of communicative options afforded by the Net and the potential for increases in collaborative and cooperative work leading (hopefully) to development of higher-order thinking skills. While this is certainly true and the mantra of E-learning 2.0, the second consensus reminds us that creating opportunity for interaction is no guarantee of its incidence. The fifth consensus points to impressive increases in access, with a special nod to that provided to disabled learners. Although I certainly concur with the tremendous increase in access afforded through this technology, I would not limit this to disabled students (after all, we are all disabled to some degree). But most importantly, the net provides students with freedom in multiple dimensions including control of space and pace, content, relationship, media and in some cases to cost (Paulsen, 1993) . In sum, the consensus conclusions accurately reflect the major challenges and the major affordances of the Net and it is reassuring to see these reflected in the Canadian data reported in this study.
In the introduction, the authors claim that in e-learning “there is sufficient optimism for technology’s positive impact that governments have established committees, formed task forces, and dedicated substantial funds to the delivery or enhancement of technology-based instruction” (p. 10). I argue that in fact Canada very significantly trails behind comparative countries in the size and quality of investment in e-learning policy making, implementation and especially in systematic research. For example, the governments of Australia, (http://www.flexiblelearning.net.au); Ireland (www.ncirl.ie/downloads/research _and_innovation/SFI_reportFINAL.pdf) and the United Kingdom (www.dfes. gov.uk/publications/e-strategy/), amongst many others, have articulated strategies by which they support and accelerate the use of e-learning throughout their national jurisdictions . This type of national or pan-Canadian (to be politically correct) concern and investment in e-learning and leadership is sorely missing in Canada. Ironically, the reference the authors use to defend this claim of support is from the Council of Ministers of Education, Canada. In fact, this reference is a whining press release by the provincial education ministers that urges the federal government to invest in connectivity for e-learning. But there was no evidence then, nor is there any today, that demonstrates these same provincial Ministers’ willingness to invest funds in this provincially mandated activity!
One of the impressive components of the report, and one that we have become accustomed to in work from the Centre for the Study of Learning and Performance at Concordia University, is the transparency and thus potential replication of the study methodology. Details with regard to choice of documents to be included, the Argument Catalogue used for analysis and the methods of that analysis are clear and forthcoming. This is especially important when dealing with content that is as challenging to categorize and conceptually deal with as the diversity of forms and functions of e-learning research and practice. The work also provides a foundation, model and motivation for further systemic review.
I particularly liked the brief summations of the non-research publications in which the authors’ attempt to summarize prevailing public, policy maker and practitioner attitudes towards e-learning are presented. These synopses reveal critical attitudes that both propel and confound efforts by participants to develop and exploit e-learning technology and techniques. I also commend the authors for accurately exposing some of the concerns of critics (Oliver & Conole, 2003) that are similar to my own concerns noted earlier with regard to methodological hegemony. Indeed, as Cantoni and Rega (2004) noted, there is no consensus amongst researchers on means, methods or epistemology that should guide e-learning research—thus necessitating an eclectic approach that attempts to winnow the best from both complementary and contradictory sources of evidence. Canadian researchers have, for the most part, stood on the side (not even acting as UN Peacekeepers!) as paradigm wars rage in the USA and other countries. Perhaps this reflects our national reluctance to converge our thinking on any single solution and we are destined to creating a low-energy, multi-cultural and multilinguistic e-learning research agenda in this, as in many other avenues of Canadian life.
I was also not surprised to find that 20% of the articles in the sample studied by the reviewers looked at technology adoption and integration. This very significant focus, along with a slightly smaller focus on professional development, indicates the critical change and innovation management issues that are integral to the study of e-learning in real contexts. E-learning is a disruptive technology (Anderson, 1999; Archer, Garrison, & Anderson, 1999; Norman, 1999) and one that poses considerable threat to the way in which our mostly conservative institutions (universities, colleges and schools) operate. The large number of articles focused on these change processes is heartening and shows that researchers are at least aware of the problems associated with adoption of e-learning technologies. In a listing of eight requirements for adoption of technology-based innovation in education, Ely (1999) lists a “ Dissatisfaction with the status quo” as the first necessary condition. In an era of growing student enrolments, teacher shortages and increasing emphasis on commercialization ( Bok , 2003) , there seems little or at best sporadic effort at reform and innovation needed to propel long term rethinking of teaching and learning (Pocklington & Tupper, 2002) . This is especially sad given the opportunity and demand for learning created by the affordances of the information rich and connected society ( Downes, 2005; Siemens, 2005; Willinsky, 2005) .
The report does a nice job of categorizing the evidence based upon the key target audiences of the Council for Learning (adult learning, K–12, and so on) The overall impact score of e-learning is positive for all sectors, but it is interesting to note that over half the studies were situated in post secondary contexts. This is not because Canadians spend more money or time in postsecondary as opposed to K–12 or adult education/training but rather because the vast majority of the small number of active researchers in e-learning are employed at post secondary institutions and thus the convenience samples obtained in their own institutions are most studied. A more strategic research agenda would focus e-learning research in areas where we currently have the most expense or in which we have a sense of greatest potential benefit. My own bet is that continuing lifelong and professional education offers the greatest potential yet totalling adult, health and the ‘not specified’ yields only 18% of the e-learning studies.
I was somewhat surprised to have the author’s conclude with a worrying note about the future of e-learning “given the magnitude of the investment required to develop it, sustain it and cause it to grow” (p. 45). 2005 data from Statistics Canada showing that 67.9% of Canadians access the internet regularly (http://www40.statcan.ca/l01/cst01/comm14a.htm) and 2004 data from the US showing that the number of learners enrolled in online courses is growing by over 18% per year (Sloan, 2005), indicate that e-learning is very likely here to stay. What is in question is Canada’s role in designing, distributing, researching and leading in this domain. The distributed and dispersed nature of our major education, commercial and social institutions and population woul d make one think that Canadians and their governments would place high priority on being not just consumers but world class leaders and producers of e-learning.
The report concludes with a now traditional call for greater support and more strategic funding on the most important components of the educational system. I could not agree with more enthusiasm with the key recommendations that conclude the report and these are worth repeating here:
Anderson, T. (1999). Using disruptive technologies in the Universities: Confessions of an Agent Provocateur. In Ed-Media 1999 World Conference on Educational Multimedia, Hypermedia and Telecommunications: AACE.
Anderson, T. (2003). Modes of interaction in distance education: Recent developments and research questions. In M. Moore & W. Anderson (Eds.), Handbook of Distance Education. (pp. 129–144). Mahwah, NJ: Erlbaum.
Archer, W., Garrison, D. R., & Anderson, T. (1999). Adopting disruptive technologies in traditional universities: Continuing education as an incubator for innovation. Canadian Journal for University Continuing Education, 25(1), 13–30. Retrieved September 2006 from http://www.extension.usask.ca/CJUCE/eng/Ab251.html
Bok, D. (2003). Universities in the marketplace: The commercialization of higher education. Princeton, NJ: Princeton University Press.
Cantoni, L., & Rega, I. (2004). Looking for fixed stars in the elearning community: A research on referenced lit. in SITE proceeding book from 1994–2001. Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications (Vol. 2004, pp. 4697–4704). Retrieved September 2006 from www.newmine.org/newmine_cantoni_alii_EDMEDIA04.pdf
Downes, S. (2005). E-Learning 2.0. ELearn Magazine, Retrieved Dec. 2005 from http://elearnmag.org/subpage.cfm?section=articles&article=29-1
Ely, D. P. (1999). Conditions that facilitate the implementation of educational technology innovations. Educational Technology, (39), 23–37.
Figueiredo, A. (2005). Learning contexts: A blueprint for research. Interactive Educational Multimedia, (11), 127–139. Retrieved July 2006 from http://www.ub.es/multimedia/iem/down/c11/Learning_Contexts.pdf
Norman, D. (1998). T he invisible computer: Why good products can fail, the personal computer is so complex and information appliances are the solution. Cambridge, MA: MIT Press
Oliver, M., & Conole, G. (2003). Evidence-based practice and e-learning in higher education: Can we and should we? Research Papers in Education, 18 (4), 385–397.
Paulsen, M. (1993). The hexagon of cooperative freedom: A distance education theory attuned to computer conferencing. DEOS, 3(2) Retrieved May 28, 2004 from http://www.nettskolen.com/forskning/21/hexagon.html
Rossiter, J. (2002). An e-learning vision: Towards a Pan-Canadian strategy and action plan. (CANARIE Discussion paper). Ottawa, ON: CANARIE Inc. Retrieved August 2005, from http://www.canarie.ca/funding/elearning/elearningvision.pdf
Pocklington, T., & Tupper, A. (2002). No place to learn: Why universities aren't working. Vancouver: UBC Press.
Siemens, G. (2005). A learning theory for the digital age. Instructional Technology and Distance Education, 2(1), 3–10. Retrieved October 2005 from http://www.elearnspace.org/Articles/connectivism.htm
Sloan Foundation (2005) Growing by degrees: Online education in the United States, 2005. Retrieved October 2006 from http://www.sloan-c.org/publications/survey/pdf/growing_by_degrees.pdf
Willinsky, J. (2005). The unacknowledged convergence of op en source, open access and open science. First Monday, 10(8) Retrieved August 2006 from www.firstmonday.org/issues/issue10_8/willinsky/
© Canadian Journal of Learning and Technology