Figure 1 : Criteria for Evaluating a Senior Practitioner’s Portfolio for the Canadian Society for Training and Development’s certification of Training and Development Professionals
Authors
Saul Carliner, is an Assistant Professor in the Department of Education, Centre for the Study of Learning and Performance (CSLP), Concordia University, Montreal, Quebec. Correspondence regarding this article can be sent to: scarliner@education.concordia.ca
Portfolios are central to the Bachelor of Science program in scientific and technical communication at one large state university in the midwestern United States. At the beginning of the program, students who want to gain admission must submit a portfolio that demonstrates both their ability to communicate in written form and an interest in the field. At the end of the program, students at some of the participating campuses (the program is offered in partnership between the main campus of the University and several of its sister campuses and other public institutions in the state) prepare an electronic portfolio that showcases their best work and that students use while searching for jobs.
Throughout the program, the faculty supports students in the development of their portfolios. To help students prepare their initial portfolio, the University offers a 1-credit course for pre-majors that helps students learn what scientific and technical communication really is, then explains how to prepare the portfolio for admission. Students learn how to select items for the portfolio, how to describe those items to people who will be assessing their portfolios, and how to accurately represent their work. For example, students are encouraged to include works that they contributed to but not necessarily in a lead role, such as manuals for which they handled the layout and production, but not necessarily the writing.
The portfolio serves as one of the primary pieces of evidence for acceptance into the program. During the program, students are reminded on a regular basis to add other pieces to their portfolios and prospective employers actively seek to view student work when interviewing students for jobs upon graduation. Although no one has publicly declared this portfolio program to be a success, that the academic program has used portfolios for so many years and that both students and faculty actively support it, the program can be considered one.
At about the same time that the University was devising the portfolio program for its program in scientific and technical communication, the state’s Board of Education tried to redesign the entire high school curriculum around the concept of portfolios. Readiness for graduation would involve a portfolio review. Despite support from the governor, influential state legislators, leading teachers, and the business community, the program was declared an unmitigated disaster almost as quickly as it was launched, and the state abandoned the program within a year. According to newspaper accounts at the time:
Such terms as "invigilators," "performance packages," "rubric" and "authentic assessments" have been used to explain the state's ambitious overhaul of what students should know.
The fact that many people may have no idea what they mean explains why the profile is in danger of flunking (Draper, 1999).
As these two instances show, although portfolios have great potential as a tool for collecting and gauging long-term learning and the competencies that someone brings to a situation, their long-term success as a teaching and assessment tool in education is no sure thing. Nor should people assume that, because portfolios are widely used in some workplace situations that they can be used in all such situations.
In this commentary, I reflect on the potentials and pitfalls of the e-portfolio concepts, using the other articles in this special issue as a springboard for discussion. Specifically, I explore issues associated with the literature that we consult when communicating about electronic portfolios, the conception of the roles of creators, facilitators and evaluators of the portfolio process, and the research on electronic portfolios.
In reviewing this discussion about the use of portfolios, as important as it is to notice what is discussed in the review of the literature, it’s probably more useful to notice what was left out. Two important issues regarding portfolios as a vital teaching and assessment tool are missing from these discussions: a substantive discussion of the use of paper portfolios as teaching and assessment tools, and a discussion of the use of portfolios beyond the formal educational system.
Consider, first, the issue of paper portfolios. Abrami and Barrett (2005) note in their introduction to this special issue,
EPs are the Information Age’s version of the artist’s portfolio in the sense that they not only summarize an artist’s creative achievements but also illustrate those achievements. An artist, architect, or engineer who displays her portfolio of work allows the viewer to form a direct impression of that work without having to rely on the judgments of others. EPs tell a story both literally and figuratively by keeping a temporal and structural record of events.
Although most of the articles in this special issue document the ongoing conversation about the use of electronic portfolios, noticeably absent from most of these articles are research and other discussions about the use of traditional paper portfolios in individual courses, curricula, and workplace settings. At most, just a few citations are made. Although the “electronic” aspect of portfolios is relatively new (about the last ten years), the portfolio itself has had a much longer life. Indeed, a meeting at which experienced professionals review the portfolios of their colleagues is a popular type of monthly meeting for chapters of the American Institute of Graphic Arts (AIGA, 2001).
Even though electronic portfolios significantly simplify the capture of samples, the provision of commentary, and transfer the portfolio materials among users over their paper counterparts, the general process of developing portfolios is not unique to the information age and, as Abrami and Barrett note, has a long history of use in the fine and applied arts. In academic programs in those fields, the use of portfolios has been integrated into the curriculum for decades, long before electronic portfolios were even feasible. Similarly, the use of portfolios in the job search process in those fields has been widely used for decades, too. As other disciplines consider ways to integrate portfolios into both the classroom and the job search process, we might spare ourselves some poor choices by learning from our colleagues’ experience.
As much of the conversation about the use of paper portfolios does not seem to be represented in this discussion, so is the discussion about the use of portfolios beyond the formal educational system. None of the articles discusses focuses on uses of electronic portfolios for work. That does not mean that such uses are not widely discussed in Canada. One key use of portfolios beyond the academy is as a tool for assessing prior learning. Such tools provide authentic assessments that help immigrants establish their professional credentials in a system that does not allow these people to transfer their certifications and licenses, and work in the fields for which they have been trained.
Although most discussions of using portfolios beyond the academy focus on assessment and showcase uses, potential also exists in the use of process portfolios for the workplace. As careers become obsolete before workers reach the end of their careers, job placement specialists increasingly face the challenge of placing skilled workers in new careers. Because few mechanisms exist for helping workers leverage existing skills and gain advanced placement in new careers, most face the challenge of starting all over again, with no recognition of the skills or experience they have. Portfolios provide a means for assessing such skills and experience, and re-packaging them to workers’ benefits.
Although presented as corollary discussions, most of the models and research presented in this issue eventually raise questions about the roles and responsibilities of creators, facilitators, and evaluators of electronic portfolios. One point of confusion pertains to the role and responsibility of portfolio creators in designing their electronic portfolios. On the one hand, Challis (2005) proposes that a mature user of electronic portfolios designs the navigation and visual design of the portfolio. Her model suggests that the navigation should be “clear and intuitive, let users select their own pathway(s), and fully hyperlinked” (2005). In terms of visual design, it should be “uncluttered and elegant, [use graphics] in accord with the portfolio’s purpose and its creator, [avoid] distracting elements, [be] well organized and coherent, and [have] connections [that] are readily made” (2005). This sounds like advice that a professor offers to students who are majoring in graphic design and professional communication (Kostelnick & Roberts, 1998); is this reasonable to expect of students and professionals working in other fields?
On the other hand, emphasizing the visual aspects of the electronic portfolio often pre-empts attention to its substance, as one of the students in Tosh, Light, Fleming, and Haywood’s (2005) notes. The resolution is not simple. Students in the same study expressed concern that template-based tools, which seem to be the dominant form of software for electronic portfolios, lacked the flexibility students felt they needed to accurately represent their work.
In the end, the question of how much creators should be involved in designing the navigation and appearance of their electronic portfolios is unresolved. Perhaps software for electronic portfolios could be designed to allow for more flexibility, learning a lesson from the layered user assistance provided for other types of software. Layered assistance provides people with increasing levels of flexibility and freedom as they reach more experienced levels of use (Mobley, Knight, & Meserth, 2003). For example, electronic portfolio software might provide novice users with structured templates for preparing their materials while letting experienced users bypass the templates. It seems that few systems are designed for multiple levels of users.
If the roles of creators are not well-defined, neither are those facilitators and evaluators. Facilitators are the people who instruct people on how to create electronic portfolios and support these people in creating their portfolios. The nature of the instruction and advice they offer is likely to change at various phases of a creator’s academic and professional careers. For example, the guidance needed to help a seven-year-old prepare a process portfolio substantially differs from the guidance needed to help a 47-year-old prepare a showcase portfolio for a creative director’s job.
The role of the evaluator is similarly unclear. Immediate issues facing evaluators of electronic portfolios include acceptance and evaluation rubrics. Consider the challenges of acceptance. In certain situations when electronic portfolios might make excellent sense, evaluators do not seem comfortable with electronic portfolios. For example, promotion and tenure—for which candidates often need to provide 2 three-inch binders of documentation—seems like a natural candidate for electronic portfolios. Instead of placing these oversized binders in an inconvenient but centralized location for committee members to read, candidates can place provide committee members with individual, portable CDs and DVDs, which can be viewed anywhere. But as Challis (2005) reports, committee members’ first question is, “How do I print this?” suggesting that they may not yet be ready for electronic portfolios.
Similarly, once evaluators receive portfolios, they often receive little or no guidance in evaluating them. This is not a promotion and tenure issue; this seems to be an ongoing issue with all types of portfolios. For example, although all of the articles in this issue consider evaluation, none provides an example of the specific rubrics for evaluating a portfolio. In fact, one of the concerns expressed by students in the Tosh, Light, Fleming, and Haywood (2005) article is that the assessment criteria are not provided to them, especially for such subjective assignments as writing personal reflections.
What these articles do suggest is that evaluation must be a tedious task, because several authors mention the amount of time that evaluation of an individual portfolio takes, and this may place a practical limit on the extent to which portfolios may be used.
But what do evaluation rubrics actually look like? Perhaps the rubrics for evaluating the portfolios of senior (experienced) practitioners who apply to the Certified Training and Development Professional program offered by the Canadian Society for Training and Development provide an insight into what evaluation criteria might look like.
Figure 1 : Criteria for Evaluating a Senior Practitioner’s Portfolio for the Canadian Society for Training and Development’s certification of Training and Development Professionals
But these are only rubrics for evaluating an assessment portfolio of an experienced professional. These evaluation criteria are prescriptive, representing what experts believe are the competencies of a senior training and development practitioner, rather than ones emerging directly from observation of the work in the field. These evaluation criteria are also rooted in a behaviourist approach, focusing primarily on accomplishments rather than the processes followed to achieve them. This behaviourist approach sharply contrasts with the socio-constructivist approach advocated by Abrami and Barrett (2005) in their introduction and Wade, Abrami, and Sclater (2005), and alluded to by the other articles in this special issue.
This does not mean that the socio-constructivist paradigm is incompatible or unworkable with electronic portfolios; indeed that is not my point in raising it. Indeed, the discourse analysis of the electronic portfolios prepared by first-year pre-service teachers reported by Peters, Chevrier, Leblanc, Fortin and Malette (2005) suggests a way to approach the evaluation of electronic portfolios from a descriptive framework.
The point merely suggests that electronic portfolios are ultimately compatible with a wide variety of teaching and evaluation philosophies. The crux of that evaluation is in the rubrics and, without samples of those rubrics, facilitators and evaluators will feel challenged in applying electronic portfolios in constructivist and other contexts.
As the roles of the various participants in the portfolio process need further clarification, so do issues raised by the research conducted thus far. Specifically, three issues arise when considering the research presented in this issue. First, all of the studies reported here focus on the use of e-portfolios in academic contexts. Some, like Wade, et al. (2005), focus on the use of e-portfolios in schools, while more, like Challis (2005), Milman and Kilbane (2005), Smits, et al. (2005), and Tosh, et al. (2005) all look at the use of electronic portfolios in higher education. Indeed, three of the contributions—those by Milman and Kilbane, Peters, et al., and Smits, et al.—not only report on the use of e-portfolios in teacher education, but within the context of their own classrooms and programs.
Although the lack of reports on the use of e-portfolios in workplace contexts is a source of concern, evidence in the literature suggests that they do exist (e.g., Dagley & Berrington, 2005; Sanders, 2005).
What is more troublesome, however, is the extensive use of our own classrooms to conduct research on e-portfolios, the second issue that arises when considering the research reported in this issue. Although conducting research in one’s own classroom is widely used in educational research because of its convenience, it often poses more problems in reporting than are offset by the convenience. Milman and Kilbane address some key methodological issues in their article. But they overlook one key issue. One of the goals of researching e-portfolios is to determine the contexts in which electronic portfolios “are most effective and worthwhile” (Abrami & Barrett, 2005). But if we limit the contexts in which we conduct the research and look only at our own classrooms (and those of our colleagues in education programs), we are in no position to make such an assessment. For research on contexts to be effective, researchers need to look beyond our own classrooms in teacher education. Furthermore, other teaching disciplines, like writing centers, are experimenting with the use of electronic portfolios (Click & Magruder, 2004) and dental education, often without the involvement of researchers in the mainstream of education. Perhaps, in seeking classrooms to study, we should focus on developing partnerships with these other disciplines.
But perhaps e-portfolios are not yet ready for extensive summative research, as suggested by the third, and probably most significant, issue that arises when considering the research in this issue. As Tosh, et al.’s (2005) report of student perceptions of pilot implementations of electronic portfolios in higher education programs suggests, perhaps efforts to conduct summative evaluation of the outcomes and processes of electronic portfolios and evidence of their success, as well as the related concern about moving forward with funding and infrastructure (Abrami & Barrett, 2005) are premature. With methods for developing, teaching with, and evaluating electronic portfolios still in flux, perhaps the current research agenda should take a more modest approach: focusing on formative issues, such as validating teaching techniques, instructions to students, and evaluation rubrics. As most basic instructional design texts advise, one should only conduct summative evaluation on validated materials (Dick, Carey, & Carey 2000; Smith & Ragan, 1999). The evidence presented by Tosh, et al. suggests that, perhaps, e-portfolios are not yet validated; they are still in the formative stage. In addition to validating the educational aspects of e-portfolios, formative evaluation could address the technical and administrative issues (Abrami & Barrett, 2005) raised by the use of e-portfolios.
Admittedly, until now, this commentary has focused on the shortcomings in the literature reviews, designs, and research on electronic portfolios. That’s because the initial enthusiasm expressed for electronic portfolios resembles the initial enthusiasm for e-learning, which was projected to cause an immediate revolution in all forms of education (Milken 1998). Although some research and theory supported this enthusiasm, they ignored other research (such as Russell, ongoing) and the practicalities of designing and implementing e-learning (Van Buren & Sloman, 2003), and these ultimately meant that e-learning launched an evolution of education that continues to this day. Although I strongly believe in the potential of electronic portfolios, I would encourage those of us who advocate for their use to take a more modest approach, tempered by the lessons of e-learning as well as the evidence in our own research and theory development. Although electronic portfolios have much potential to benefit the learning and working processes, the evidence seems to suggest that they are only at a formative evaluation stage of development. So perhaps some of the theory development, research, and product development—as well as rhetoric surrounding them—should be scaled to something more appropriate to this phase of development.
In terms of theory development, while the concept of portfolios for life makes tremendous sense because portfolios have can be used in all phases of education and career development, perhaps the concept of one portfolio for life might be premature. As the research and commentary published elsewhere in this issue suggest, we have much to learn about the use of portfolios within each specific school context and even more to learn about their use within work contexts. In addition to having strong evidence of the value of electronic portfolios in these contexts, exploring the concept of a portfolio for life requires studying the use of portfolios in passages between major phases of schooling and careers. For example, how would a portfolio prepared in elementary school need to be modified for use in middle school? From middle school to high school? If we do not have validated portfolios for elementary and middle schools, how can we explore the transition of portfolios between the two?
Similarly, some technical issues arise when considering the concept of a portfolio for life. Although many people first raise the issue of standards, a more immediate issue is the likelihood that file formats will change several times over the course of a real person’s lifetime. The dominant file formats used now—such as .doc, .ppt, .jpg, and so on—are barely fifteen years old. Although their dominance limits software publishers’ flexibility in changing file formats, changes in technology might demand that such changes happen. Given the rate at which technology changes, a strong likelihood exists that, even if standards for a single portfolio for life could be developed, the practical realities of technology will require that creators of portfolios convert their electronic artefacts at least once during their lifetime. That, in turn, may require that the creator discard the previous portfolio and that software developers abandon any standards developed.
The discussion of the current limits of the portfolio-for-life concept also suggest research activities that should occur at this time. First this emerging discipline needs more formative research, like that conducted by Tosh, et al., to validate that the electronic portfolios can actually achieve the goals for which they were intended. As mentioned earlier, this research needs to be conducted not only in our own classrooms in education programs but, more significantly, in other programs and in the workplace. Only when electronic portfolios are validated should summative evaluation commence. And only after these portfolios are evaluated in the contexts for which they were designed should researchers consider linking portfolios used in one context with those used in another.
That, in turn, suggests the focus of product development for electronic portfolios. Although the concept of electronic portfolios for life has long-term appeal, in the short-term, developing software for specific contexts might make more sense. More significant than developing software for specific contexts is the development of related materials for use with this software. Creators of electronic portfolios need guidance on how to create these materials, and support and guidance as they develop their nascent portfolios. Facilitators need resources and training in how to teach people to create their portfolios and how to integrate the portfolios into their teaching and work contexts. Evaluators of portfolios need criteria for assessing portfolios, not only to assess existing portfolios, but models of assessments so rubrics for other types of portfolios can be developed.
The last challenge is neither a technical nor a research challenge, but one that was introduced at the beginning of this commentary, suggested in the section on evaluation, and is well-known to most people who introduce software into an organization: managing the change caused by the introduction of this new approach. Evidence suggests that electronic portfolios have the potential to promote self-regulation in learners, provide more authentic assessment of learning activities, demonstrate competence for jobs, and keep an ongoing record of accomplishments and learning. But if those of us who are proponents of electronic portfolios mismanage their introductions in our contexts as badly as the state department of education described in the beginning of this article introduced portfolios in its context, we won’t have the opportunity to demonstrate that.
Fortunately, even without the research community, a case for electronic portfolios is emerging if, for no other reason, many people are already creating their own electronic portfolios, many without the software tools that we are creating. For example, visit the web sites of various independent professionals, who proudly display their portfolios as evidence of their worthiness for work, like marketing writer Jack Massa does at www.guidancecom.com and communications and training firm, Fredrickson Communications (www.fredcomm.com). These people did not create their portfolios as a requirement for class or as part of a research project; they did so out of necessity.
Ultimately, our ability to integrate electronic portfolios into schools and the workplace rests not with our exuberance for the software, but with our ability to ensure and communicate the viability, practicality, and utility of these tools. One of the best ways to do is by taking the suggestion of the students in Tosh, et al.’s study: create electronic portfolios of our own work. Perhaps that’s the best way to begin introducing electronic portfolios.
Abrami, P., & Barrett, H. (2005). Directions for research and development on electronic portfolios.Canadian Journal of Learning and Technology, this issue.
AIGA. (2001). AIGA Atlanta portfolio review (meeting notice). Retrieved September 27, 2005 from http://www.aiga.org/content.cfm?alias=event calendar&id=2390
Carliner, S. (2005). Performance support approach to professional development for teachers who use e-portfolios. Pan American E-Portfolio Conference. Learning Innovations Forum: Vancouver, BC. April 18, 2005.
Challis, D. (2005). Towards the mature ePortfolio: Some implications for higher education. Canadian Journal of Learning and Technology, this issue.
Click, B.A., & Magruder, S.C. (2004). Implementing electronic portfolios for performance assessment: A pilot program involving a college writing center. Assessment update. 16(4). 1–15.
Dagley, V., & Berrington, B. (2005). Learning from an evaluation of an electronic protfolio to support general practitioners' personal development planning, appraisal and revalidation . Education for primary care. 16(5). 567–574.
Dick, W., Carey, L., & Carey, J. (2000). The systematic design of instruction (5th ed.). Reading, MA: Addison-Wesley.
Draper, N. (1999). Analysis: Why is the Profile in danger of flunking? Minneapolis Star-Tribune. February 28, 1999.
Kostelnick, C., & Roberts, D. (1998). Designing visual language: Strategies for professional communicators. New York: Allyn & Bacon.
Milken, M. (1998). Keynote presentation. Online Learning 1998. VNU Business Media. Anaheim, CA: September 1998.
Milman, N.B., & Kilbane, C.R. (2005). Digital teaching portfolios: Catalysts for fostering authentic professional development. Canadian Journal of Learning and Technology, this issue.
Mobley, K., Knight, C., & Meserth, T. (2003). Designing embedded help to encourage inadvertent learning. Technical Communication. 50(1). 33–45.
Peters, M., Chevrier, J., Leblanc, R., Fortin, G., & Malette, J. (2005). Compétence réflexive, carte conceptuelle et webfolio à la formation des maîtres. Canadian Journal of Learning and Technology , this issue.
Russell, T. (Ongoing publication). No significant difference website. www.nosignificantdifference.org.
Sanders, J. (2005). Commentary: electronic portfolios for general practitioners: the beginning of an exciting future. Education for Primary Care. 16(5). 535–539.
Smith, P.L., & Ragan, T.J. (1999). Instructional design (2nd ed.). Upper Saddle River, NJ: Prentice-Hall.
Smits, H., Wang, H.C., Towers, J., Crichton, S., Field, J., & Tarr, P. (2005). Deepening understanding of inquiry teaching and learning with e-portfolios in a teacher preparation program. Canadian Journal of Learning and Technology, this issue.
Tosh, D., Light, T.P., Fleming, K., & Haywood, J. (2005). Engagement with electronic portfolios: Challenges from the student perspective. Canadian Journal of Learning and Technology, this issue.
Van Buren, M.E., & Sloman, M. (2003). e-Learning’s learning curve: Will they come, will they learn? American Society for Training and Development International Conference and Exposition. San Diego, California, May 18, 2003.
Wade, A., Abrami, P.C., & Sclater, J. (2005). An electronic portfolio to support learning. Canadian Journal of Learning and Technology, this issue.
© Canadian Journal of Learning and Technology