a Translated from Dutch
Maarten van Wesel
Anouk Prop
Authors
Maarten van Wesel is an e-Learning/e-Research consultant and researcher in Education & Research Services, University Library, Maastricht University, the Netherlands. Correspondence regarding this article can be sent to: M.vanWesel@ub.unimaas.nl
Anouk Prop is now a psychologist in Maastricht, the Netherlands.
Abstract: Electronic portfolios offer many advantages to their paper-based counterparts, including, but not limited to working on ICT skills, adding multimedia and easier sharing of the portfolio. Previous research showed that the quality of a portfolio does not depend on the medium used. In this article the perceived support for self-reflection of an electronic portfolio and a paper-based portfolio in the same ecological setting are compared. We made use of the fact that during this study about half of the first year medical students was using an electronic portfolio (n = 157) and the other half a paper-based portfolio (n = 190). Nine questions were added to the standard end of the block evaluation, which is handed to 25 percent of year one educational groups. Findings suggest that perceptions about the support for self-reflection, and the usefulness of compiling a portfolio, do not differ between students using an electronic portfolio and students using a paper-based portfolio.
Résumé : Les portfolios électroniques offrent de nombreux avantages comparativement à leurs homologues de papier, entre autres la possibilité de perfectionner les compétences liées aux TIC, d’ajouter des éléments multimédias et de partager plus facilement le portfolio. Des études précédentes ont montré que la qualité d’un portfolio ne dépend pas du support utilisé. Dans le présent article, nous comparons l’aide à l’autoréflexion perçue pour un portfolio électronique et un portfolio sur support papier dans le même environnement. Dans le cadre de cette étude, nous avons profité du fait qu’environ la moitié des étudiants de première année en médecine utilisait un portfolio électronique (n = 157) et l’autre moitié, un portfolio sur support papier (n = 190). Neuf questions ont été ajoutées à l’évaluation normale remise à 25 pour cent des groupes de première année à la fin du bloc de formation. Les résultats suggèrent que les perceptions des étudiants à l’égard de l’aide à l’autoréflexion et de l’utilité de compiler un portfolio ne diffèrent pas entre les utilisateurs de portfolios électroniques et les utilisateurs de portfolios sur support papier.
Portfolio-based learning is being implemented more and more in a range of educational and professional learning contexts worldwide in order to monitor students’ professional development. A portfolio is usually described as a collection of students’ work and achievements during their academic career (Challis, 1999; Chen, Yu & Chang, 2007). Specifically in medical education and related fields, the portfolio is increasingly being used as an instrument to stimulate the professional growth of students (Prop, Shacklady, Dornan & Driessen, 2007).
In today’s highly digitalized world it is not unexpected to observe an increase in the use of ICT in higher education. For years the portfolio has also been moving in this direction, there is a noticeable shift from the use of paper-based portfolios towards the use of electronic portfolios. When portfolios are introduced for the first time in an educational setting, the electronic, online version is most often the preferred choice (Lambert & Corrin, 2006; Wade, Abrami & Sclater, 2005 ); as well established paper-based portfolios are being replaced by digital ones (Barlett & Sherry, 2004; Woodward & Bablohy, 2004). While the specific advantages depend on the ePortfolio system used, reasons often stated for the use of electronic portfolios1 include:
Although ePortfolios are being used more frequently in higher education, little attention has been paid to comparing them to their paper-based counter parts. Current ePortfolio research focuses mostly on the specific features an electronic portfolio has to offer. While the above mentioned points truly can be advantageous, educational innovation always asks for caution. When moving from a paper-based portfolio to the digital variant, we must assure ourselves that our original portfolio goals are not lost. In other words, electronic portfolios and paper-based portfolios ought to be compared on their shared potential merits, such as support for self-reflection, preferably in a similar ecological setting.
Driessen et al. (2007) concluded after an ecological study, that creating an electronic portfolio, or web-based portfolio, enhanced student motivation and that an ePortfolio is more user-friendly for mentors. They also found that the electronic portfolio delivers the same content quality as the paper-based variant. They also found that students spent significantly more time preparing an e-portfolio than a paper-based one. However, their questions about perceptions of students focus on the overall experience of the students, and their measurement of the quality of evidence and content was based on a content analysis. The present study focuses on the perceptions students have about the support their portfolio gives them for self-reflection.
The student perspective is relevant, since the student perception of a learning environment in a large extent affects and is affected by the way students manage to work with the environment, and thus influences their learning (Diercks-O'Brien, 2000; Gijbels, van de Watering, & Dochy, 2005; Gijbels, van de Watering, Dochy, & van den Bossche, 2006; Segers & Dochy, 2001).
Self-reflection is an essential precondition for the professional development of medical students, since it helps students “become aware of their mental structures, subject them to a critical analyses, and if necessary, restructure them” (Korthagen, 2001, p. 51). Thus in order to monitor their own learning process and to obtain an insight into their own strengths and weaknesses, future doctors must be able to reflect and analyze on a meta level.
Previous research, in general practitioners training, has revealed that working with portfolios does not always stimulate reflection (Pearson & Heywood, 2004). The process of self-reflection on the professional development of students requires support (Challis, 1999; Chen et al., 2007). This is confirmed by the study of Pearson and Heywood (2004), which shows that successful reflection will not occur without encouragement of a coach or mentor, implying a vital role for the mentor or coach in portfolio-based learning in, medical education.
At the former Faculty of Medicine, now part of the Faculty of Health, Medicine, and Life Sciences, of Maastricht University, one of the main reasons for portfolio use is the development of self-reflective skills among students (Driessen, van Tartwijk, Vermunt, & van der Vleuten, 2003). The philosophy behind this is that a medical doctor, as a life-long learner, ought to be able to reflect on his/her actions and learning. The Maastricht portfolio process is described in detail elsewhere (Driessen, van Tartwijk, Overeem, Vermunt, & van der Vleuten, 2005; Driessen et al., 2003).
From the 2001 introduction of the portfolio in the Maastricht medical curriculum onward, small-scale experiments using electronic portfolios have been carried out. First, public folders in MS Outlook were used. As of 2003, a test version of the Blackboard Content System (BBCS) was available, enabling students to have long-term ownership of their portfolio and easily regulated access for mentors. In 2004 Maastricht University was able to use a newer version (version 2.0) of the Blackboard Content System maintained by the e-Merge consortium2. 2005 saw a small-scale pilot introduce the electronic portfolio to the first year of the medical curriculum using a locally supported version of the Blackboard Content System. Five mentors of year one students were asked to participate in an experiment. One student group was assigned per mentor to use a paper-based portfolio (n = 47), while another group was assigned to use an electronic portfolio (n = 45). To provide an incentive to participate, 20 Euros was promised to students who managed to hand in a complete electronic portfolio within the deadline. Thirty-three students were able to do so. Some practicalities and results of this pilot are reported in the aforementioned paper by Driessen et al. (2007).
The present study describes a larger scale pilot in 2006, with almost half (n = 157) of the total (N = 347) first year student population creating an electronic portfolio while the remaining students (n = 190) created a paper-based portfolio. The students creating a paper-based portfolio as well as the students creating an electric portfolio were provided with an identical, general portfolio manual. This manual contained the conceptual steps students need to undertake when creating the portfolio. The students using the electronic portfolio were also supplied with a user manual about the Blackboard Content System. For the latter group, two sessions were organized focusing on functional aspects of working with the BBCS portfolio system. No exact numbers were kept, however, very few students attended (approximately two in the first session, and four in the second session).
Portfolio mentors were asked if they were willing to participate in the project. Twelve out of 27 mentors responded positively. Due to the voluntary nature of participation, there is a small chance for a selection bias, since these mentors could be the more enthusiastic ones. These mentors mentored a total of 16 mentor groups, with about 10 students per mentor group, representing about almost half of the first year student population.
The research that has been conducted on portfolio-based learning and its effects in medical education shows that the role of the mentor is an important one (Driessen et al., 2005; Pearson & Heywood, 2004). Another study shows that the context is of less importance: students and mentors have similar perceptions, whether they are using a paper-based portfolio or an electronic portfolio (Prop et al., 2007).
There were some problems surrounding the use of the electronic portfolio. Although this could have ruined an experimental setup, in practical applications, technical problems can form part of the student’s everyday reality. During periods that the workload on the servers used was high, students, because of a bug, were sometimes directed to an unavailable server. This resulted in them not being able to work on their portfolios until the server load was reduced. Some students did not have access to the Blackboard Content System at the moment they were supposed to start working on their electronic portfolio. Other students at some instances lost access to their portfolio. After resolving these problems, no portfolio appeared to be deleted and all portfolios could be completed.
Not all students supplied their mentor with access to their electronic portfolio in the correct way. Because of this not all internal links were available to the mentor. Another common problem involved coping marked-up text (e.g., MS Word text) to the Visual Text Box Editor: the text did not look the way it did in the text editor. This led to some frustration with students. Students were advised to only copy text without mark-up.
As the research aims to compare the paper-based portfolio and the electronic portfolio as a tool for developing self-reflection skills, the questionnaire focuses on this aspect. The questionnaire therefore does not ask portfolio-medium specific questions. Since these questions were to be added to the regular block evaluation questionnaire (containing 30 standard questions), to circumvent questionnaire fatigue we were asked to limited the portfolio questionnaire to nine, quasi content validated, questions (Q31 – Q39 see Table 1). Due to the limited number of questions, a response set problem can occur.
For privacy reasons, block evaluations are anonymous, thus it is not possible to pinpoint which student filled out which questionnaire. This makes it impossible to compare, for example, female to male responders (to ensure student cohorts are equally represented). However, because of the use of random sampling, it is likely that the sample truly represents the population (Neuman, 1991). The questionnaire containing portfolio questions was handed out to 25 percent of year one educational groups, which per group can contain students from different mentor groups, at the end of the academic year; 71 students filled out the questionnaire, 36 of which used an electronic portfolio, 35 a paper-based one.
Table 1. Extra questions about portfolios
a Translated from Dutch
Q31 to Q36 are statements prompting the students to express their respective opinions on a five point Likert Scale. Q37 asks for an integer number representing the total numbers of hours spent on the portfolio, excluding mentor consultations (which approximately took two hours during the whole portfolio process). Q38 prompts the students to report if they used a paper-based or electronic portfolio. And the final question, Q39, asks for a school-like grading (ranging from 1 to 10, with 1 representing ‘Very bad’ and 10 representing ‘Excellent’). The Cronbach’s alpha of the six Likert scale items (Q31 – Q36) was .877, indicating good reliability. As recommended by Jamieson (2004), the ordinal data from Q31 through Q36, answered on a five point Likert Scale, were analyzed using a non-parametric test, in this case the Mann-Whitney U-Test. Items Q37 and Q39 were analyzed using an Independent Sample T-test.
From Figure 1 – showing the histogram for Q39 – it seems clear that some students are very negative about their portfolios, especially the electronic ones (scoring a one on five occasions). The results, however, do not differ significantly (independent sample t-test: t(60) = .35, p = .73), scoring a 5.46 (SD = 1.77) for the paper variant, and a slightly lower 5.28 (SD = 2.49) for its electronic counterpart. Both types do not receive a sufficient mark (5.50) on the Dutch grading scale, suggesting students did not find it useful to put together a portfolio.
Figure 1. Grades per portfolio type
As shown in Table 2, for both the paper-based and the electronic portfolio, the majority of students had a negative perception about getting a better impression of their strong and weak points (48.5 % and 44.4 % respectively). However, for the paper-based portfolio the mode lays with Disagree, while for the electronic portfolio the mode lays with Agree.
Table 2. Results for Questions 31 to 36
Paper-based Electronic
a Strongly disagree b Disagree c Neutral d Agree e Strongly Agree
Also the majority of the students had a negative perception about the sense of professional development and insight in how to approach the study with both the paper-based and electronic portfolio (42.9 % vs. 38.9 % and 40 % vs. 47.2 % respectively). The mode for the questions about the sense of professional development with both types of portfolio is found at neutral. For the item about insight in how to approach the study the mode for the paper-based portfolio is found at neutral, whilst for the electronic portfolio this is Agree. A majority of the students perceived the subjects described in their portfolio and discussed with their mentor as relevant to themselves (57.1 % for both items by students using the paper-based portfolio and 58.3 % for both items by students using an electronic version). The mode for both items and both portfolio types lies with Agree.
On the questions if education offered enough opportunity to work on the learning goals the majority of students using the paper-based as well as students using the electronic portfolio were both positive (71.5 % and 52.8 % respectively), with Agree as the mode for both portfolio types.
As can be seen in Table 3, none of the differences are significant at the 10% level.
The students working with an electronic portfolio reported spending more time on their portfolio (see Table 4). An independent sample T-test determines this difference to be significant on a 5 % level (t(63) = -2.17, p = .03). An analysis of Effect Size shows a medium effect size (Cohen’s d = .53).
Table 3. p-value Q31-Q36
Table 4. Mean number of hours spent
The perceptions about the support for self-reflection offered by their portfolio do not differ between students using an electronic portfolio and students using a paper-based portfolio. Also they perceived no difference in the usefulness of compiling a portfolio. The fact that no statistically significant difference was found is of practical significance: it indicates that students are indifferent about the portfolio-medium used, with respect to the support for self-reflection.
There is a small cluster of students that is very negative about the electronic portfolio, both in the usefulness as on the Likert Scale items. We can only speculate these students could be those who encountered the most severe technical problems. Because of the anonymous character of the questionnaire this can not be verified. Also we were unable to extend the questionnaire into this area for two reasons. Firstly we were not allowed to add more questions to the standard questionnaire. Secondly, this would create a divergence between questionnaires filled out by students working with a paper-based portfolio and those working with an ePortfolio.
The exact impact of the technical problems on students’ perceptions of the ePortfolio remains unknown. As we are moving more and more to technical systems for student learning and evaluation support, an analysis of the impact of technical problems both on students’ perceptions and on students’ performances would be necessary. The fact that there were technical problems, and the perceptions of students working with the ePortfolio do not differ significantly from those working with a paper-based portfolio, only makes this research stronger.
We assume that students overestimate time spent on a task, as this was found for individuals in specific studies in different fields (IJsselsteijn, Bierhoff, & Slangen-de Kort, 2001; Oshagbemi, 1995). There is, however, no indication that one of the groups has an extra incentive that would lead to a more extreme deviation from the true amount of time spent than the other group. Possible explanations why more time is spent on the electronic portfolio may include: the reported tendency to write a more compact portfolio (Driessen et al., 2007; van Tartwijk et al., 2003) (which takes more time), because they enjoy working on an electronic portfolio (Driessen et al., 2007; van Tartwijk et al., 2003; Woodward & Bablohy, 2004), because there was a learning curve involved in getting to know the ePortfolio software (to be researched), or because of technical problems (to be researched).
The previous discussed research by Driessen et al. (2007) on the quality of reflection and quality of evidence does not differ between the two types of portfolios. The results presented in this paper suggest that the perceptions of the students about the support for self-reflection between the two portfolio types also do not differ. Both these outcomes add strength to the case of those proposing to replace paper-based portfolios with an electronic version, as there does not seem to be a negative effect connected to this move.
While students’ perceptions about different aspects of self-reflection using one of the portfolio types surely may offer interesting insights, more research directed at the effects on learning outcomes of the two portfolio types ought to be conducted. Some questions remain unanswered:
Thanks to Renee Stalmeijer and Diana Riksen for allowing us to add our questions to the evaluation and for returning the data to us. And to Antoinette Vesseur, Erik Driessen, Gaby Lutgens and Jeroen ten Haaf for proofreading this manuscript and supplying us with helpful suggestions. Finally, many thanks to the anonymous reviewers of CJLT for their helpful and directive suggestions.
Avraamidou, L., & Zembal-Saul, C. (2006). Exploring the influence of Web-based portfolio development on learning to teach elementary science. Association for the Advancement of Computing In Education Journal, 14(2), 178 - 205.
Barlett, A., & Sherry, A. (2004). Non-technology-savvy preservice teachers' perceptions of electronic teaching portfolios. Contemporary Issues in Technology and Teacher Education, 4(2), 225-247.
Cambridge, D. (2008). Audience, integrity, and the living document: eFolio Minnesota and lifelong and lifewide learning with ePortfolios. Computers & Education, 51(3), 1227-1246.
Challis, M. (1999). AMEE Medical Education Guide No.11 (revised): Portfolio-based learning and assessment in medical education. Medical Teacher, 21(4), 370 - 386.
Chen, H.-m., Yu, C., & Chang, C.-s. (2007). E-Homebook System: A web-based interactive education interface. Computers & Education, 49(2), 160-175.
Diercks-O'Brien, G. (2000). Approaches to the evaluation of networked learning. The International Journal of Academic Development, 5(2), 156-165.
Driessen, E., Muijtjens, A. M. M., van Tartwijk, J., & van der Vleuten, C. (2007). Web-based or paper-based portfolios: What makes the difference? Medical Education, 41, 1067-1073.
Driessen, E., van Tartwijk, J., Overeem, K., Vermunt, J., & van der Vleuten, C. (2005). Conditions for successful use of portfolios for reflection. Medical Education, 39, 1230-1235.
Driessen, E., van Tartwijk, J., Vermunt, J., & van der Vleuten, C. (2003). Use of portfolios in early undergraduate medical education. Medical Teacher, 25(1), 18-23.
Gijbels, D., van de Watering, G., & Dochy, F. (2005). Integrating assessment tasks in a problem-based learning environment. Assessment & Evaluation in Higher Education, 30(1), 73-86.
Gijbels, D., van de Watering, G., Dochy, F., & van den Bossche, P. (2006). New learning environments and constructivism: The students’ perspective. Instructional Science, 34(3), 213-226.
IJsselsteijn, W., Bierhoff, I., & Slangen-de Kort, Y. (2001). Duration estimation and presence. Paper presented at Precense 2001, Philadelphia, USA
Jamieson, S. (2004). Likert scales: how to (ab)use them. Medical Education, 38(12), 1217-1218.
Korthagen, F. A. J. (2001). A reflection on reflection. In F. A. J. Korthagen (Ed.), Linking practice and theory: the pedagogy of realistic teacher education (pp. 51-68). Mahwah, N.J.: L. Erlbaum Associates.
Lambert, S., & Corrin, L. (2006). Moving towards a university-wide implementation of an ePortfolio tool. Paper presented at 23th annual ASCILITE conference: who's learning? Whose technology, Sydney
Mason, R., Pegler, C., & Weller, M. (2004). E-portfolios: an assessment tool for online courses. British Journal of Educational Technology, 35(6), 717-727.
Neuman, W. L. (1991). Social research methods: qualitative and quantitative approaches. Boston: Allyn and Bacon.
Oshagbemi, T. (1995). Management development and managers' use of their time. Journal of Management Development, 14(8), 19-34.
Pearson, D. J., & Heywood, P. (2004). Portfolio use in general practice vocational training: a survey of GP registrars. Medical Education, 38(1), 87-95.
Prop, A., Shacklady, J., Dornan, T., & Driessen, E. (2007). Mentoring in portfolio based learning: What is an effective mentor? In W. Aalderink & M. Veugelers (Eds.), Stimulating Lifelong Learning: The ePortfolio in Dutch Higher Education (pp. 43 - 53). Utrecht: Stichting SURF.
Segers, M. S. R., & Dochy, F. (2001). New assessment forms in problem-based learning: the value-added of the students' perspective. Studies in Higher Education, 26(3), 327 - 343.
van Tartwijk, J., Driessen, E., Hoeberings, B., Kösters, J., Ritzen, M., Stokking, K., et al. (2003). Werken met een elektronisch portfolio. Groningen: Wolters-Noordhoff.
Wade, A., Abrami, P. C., & Sclater, J. (2005). An electronic portfolio to support learning. Canadian Journal of Learning and Technology, 31(3), 33-50.
Woodward, H., & Bablohy, P. (2004). Digital portfolios: fact or fashion. Assessment & Evaluation in Higher Education, 29(2), 227 - 238.