A learning outcome approach to teaching and a learning-oriented assessment are increasingly framing higher education. In an attempt to facilitate professors’ teaching activities, the TALOE (Time to Assess Learning Outcomes in E-learning) project has developed a web tool that aims to help trainers decide on the assessment methods to use in their online courses. Based on a conceptual model (ALOA) built upon Bloom’s revised taxonomy of learning objectives of the cognitive domain, the web tool automatically aligns learning outcomes stated by a user with recommended assessment methods. The web tool has been developed with the European Commission support within the Lifelong Learning Programme and it is publicly available. This paper presents the background and description of the tool and the results of the testing, where various stakeholders have provided feedback about the easiness and friendliness of the tool, as well as about the utility and quality of the recommendations it provides. Prospective of the web tool, limitations and strengths are highlighted in the paper. Show
IntroductionThe Bologna process and the implementation of the European Higher Education Area (EHEA, 2014) brought a need for a substantive change in the pedagogical model in higher education, requiring improvement of the traditional ways of describing qualifications and qualification structures, modules and programmes. The focus moved from the mere accumulation of knowledge to skills acquisition by students. The result is a shift from a content-based approach to a learning-centred approach, a change from teaching objectives to students’ learning outcomes. Learning outcomes are becoming fundamental for structuring the standards and guidelines of quality assessment in higher education and continuing education institutions in Europe and worldwide. In this context, the assessment of learning outcomes becomes a crucial process in the education system. With information and communication technologies increasingly used in education, new opportunities for improving teaching, learning and assessment are arising. Time to Assess Learning Outcomes in E-learning – TALOE (http://taloe.up.pt), is a project financed by the European Commission from January 2014 to December 2015 (Ref. 543097-LLP-1-2013-1-PT-KA3-KA3MP), which intends to promote the internal consistency of online courses. The main goal of the international project TALOE is to develop a web-based tool to help teachers and trainers decide on the e-assessment methods to use in their online courses. To be able to develop the practical tool, the TALOE consortiumFootnote 1 has set and worked on the following specific tasks:
This paper focuses on tasks 2 and 3, i.e. development and testing of the TALOE web-tool. The first task, selection and analysis of innovative e-assessment practices, was completed during 2014 and has been addressed in previous publications (Lõssenko, 2014; Gil-Jaurena et al. 2015). It consisted on the collection of 18 cases of assessment practice in online courses from different institutions around Europe, mostly from the project partners. They were selected according to a set of criteria, built upon the Standards and Guidelines for Quality Assurance in the European Higher Education Area (ENQA, 2009) and some principles agreed within the project partnership that the selected e-assessment practices had to meet:
Those principles refer to “authentic” assessment with a learning-oriented approach (Rodríguez-Gómez & Ibarra-Saiz, 2015). The goal of this selection of case studies was to obtain examples that can be used as a showcase of current practice and also as testing material during the second year of the project (as will be explained in a later section). It as well enabled the insight how much teachers acquainted with learning outcomes and about the relation between learning outcomes and assessment methods. This selection also provided information about assessment methods that are usually used in online courses. The case studies are written on predefined templates and detail information about the learning outcomes, course content, teaching practice and assessment methods. In addition, a separate section was developed to linking the assessment to the selection criteria. Theoretical framework: the ALOA modelNot all assessment methods are valid for each type of learning outcomes. The TALOE project uses an existing framework called the ALOA model (Aligning Learning Outcomes and Assessment) (Falcão 2013). This model highlights the connection between the intended learning outcomes and the assessment strategy used during a course. So, it provides tools for linking learning outcomes and assessment tasks. The ALOA model was developed by Falcão (2013) from the concept of alignment (Anderson & Krathwohl, 2001; Biggs & Tang 2007; Bloom, 1956; Boud & Falchikov, 2006), which means that the learning outcomes of a course should be used to define the teaching and learning activities and the assessment tasks. The main principle that supports the ALOA model is that, in order to ensure the validity of assessment in relation to what is intended from a course, it is necessary that the outcomes measured by the assessment tasks are the same as the ones expressed in the learning outcomes. In this sense, the ALOA model matches the basis of learning-oriented assessment (Carless, 2007), “an attempt to emphasise the learning features of assessment and promote their development” (p. 58). The ALOA model uses the revised version of Bloom’s Taxonomy of the cognitive domain (Anderson & Krathwohl, 2001) to establish the link between the learning outcomes and general assessment methods. Thus, the model has three components:
Fig. 1 The cognitive process dimension, revised Bloom’s taxonomy. Adapted from Anderson and Krathwohl (2001), 67–68. Source: Iowa State University (2015). http://www.celt.iastate.edu/teaching-resources/effective-practice/revised-blooms-taxonomy/ Full size image The TALOE project intends to apply the ALOA model to the specific context of e-learning. Development and description of the TALOE web toolThe TALOE web tool (available at http://taloetool.up.pt) is being developed to help teachers and educators (users) decide on the assessment strategies that will be used in their online courses. It provides users with a possibility to analyse the provided courses’ or modules’ learning outcomes, and offer the most appropriate e-assessment methods consistent with the intended learning outcomes. The tool is envisioned to be used by teachers/faculty/trainers, either to check if the existing assessment methods of their course or module are consistent with the stated learning outcomes, or to be advised on the most appropriate assessment methods for the existing or new course or module. The TALOE project partnership produced a first version of the web-tool at the beginning of 2015. At that point it was decided by project partners to simplify the tool procedures during the first phase of testing due to the complexity of the ALOA model (Falcão 2013). The present version of the TALOE web tool does not discriminate between knowledge types, as suggested by the theoretical model. After initial testing was completed (phase 1, explained in the coming section), it has been confirmed that the setup matrix is working properly. The best (most appropriate) e-assessment methods are selected on the base of the absolute matches between input (learning outcome) and the e-assessment methods. The welcome page of the TALOE web tool (Fig. 2) explains how to use it. Fig. 2 First page of the TALOE web tool Full size image In the first step (Fig. 3), the user describes the learning outcome that students should achieve in the course or module. The learning outcome should be described in a clear way and kept simple. For example: Apply algorithms based on prime numbers on practical problems. Fig. 3 Describing the learning outcome in the TALOE web tool Full size image Basic information about writing learning outcomes and the ALOA model are available within the TALOE web tool. After describing a learning outcome, in the Step 2 the user chooses up to three verbs that best describe it (Fig. 3). This step also enables the user to check and review the stated learning outcome. After describing the learning outcome with verbs, the user sets the process in motion and receives an assessment advice for the defined learning outcome (Fig. 4). The received outcome provides a list of potential e-assessment methods to use, with the corresponding description. Fig. 4 The TALOE web tool page with results Full size image A specific section in the TALOE website provides information and examples about assessment methods. The decision engine consists of the estimation of the score that measures the best match between the cognitive processes submitted by the user and the specific ones of a given assessment method. The assessment methods are selected on the basis of score, calculated as the ratio between the number of matches between input, and the model assessment over the number of the selection of the assessment method. Based on the received inputs, the tool calculates the best assessment methods for the defined learning outcomes. For each suggested assessment method, a description is provided to give users a better idea on how to use and integrate it in the course. Therefore, the TALOE web tool can be used in two ways: to check if the existing assessment methods in a existing course are in line with the stated learning outcomes; and to help users decide on the most appropriate assessment method for a new course or module. The TALOE web tool can be used to learn more about learning outcomes, assessment and e-assessment, as it provides guidance to teachers to formulate the learning outcomes in accordance to Bloom’s revised taxonomy (Anderson & Krathwohl, 2001). Testing of the web toolOne of the commitments and tasks of the project was to test the implementation of the web tool. In this section, we present both the testing method and the results. MethodThe dimensions considered in the evaluation of the tool were:
These criteria were agreed among project partners after some working sessions (face-to-face in the project meeting that took place at the University of Padova (Italy) in February 2015, and virtually in the audio meetings we have held monthly). The instrument used was an evaluation form, which included nine close-ended questions (Likert scale) and three open-ended questions. It was first piloted within the project consortium (phase 1); annex 1 includes the evaluation form used in that first phase. Based on the results of phase 1, the web tool was improved and converted into a web form (phase 2). We used Google forms to collect the information. The structure of the evaluation form was the following:
The evaluation of the TALOE web tool was done in two phases.
The participants in the online testing could remain anonymous, as demographical questions were optional, except the question about the home institution. Nevertheless, the majority of participants left their contact and wanted to receive further information about the TALOE project and web tool. This was important since it reveals the relevance of the TALOE project results to the stakeholders and helps the Project Consortium build a database of stakeholders for dissemination and exploitation activities. Sample descriptionThe profile of the 42 respondents (i.e., stakeholders invited by the project partners) to the evaluation form in phase 2 is presented in Figs. 5, 6, 7, and 8. Fig. 5 TALOE web tool testers’ scientific field Full size image Fig. 6 TALOE web tool testers’ course level Full size image Fig. 7 TALOE web tool testers’ country Full size image Fig. 8 TALOE web tool testers’ academic profile Full size image Different scientific fields have been represented among the web tool testers: 38 % come from the field of Education, followed by 19 % from Computer Science (Fig. 5). Twelve percent come from Humanities and another 12 % from Social Sciences. Seven percent work in Health Studies and another 7 % in Mathematics. More than 92 % have tested the TALOE web tool in relation to a higher education level course (Fig. 6). Forty-five percent have tested it in a Bachelor or Undergraduate level course; 31 % in a Master or Postgraduate level course; 14 % in a continuing education level course. Testers of the web tool work in 11 different countries. Most respondents are from Hungary (24 %), Spain (21 %) and Portugal (14 %). All the project partners’ countries are represented in the sample, plus some stakeholders from other regions not directly involved in the project, such as Germany, Ireland, Finland and Serbia (Fig. 7). Almost 80 % are professors and/or researchers in a higher education institution (Fig. 8). Thus, they have direct and practical experience in defining learning outcomes and deciding about assessment methods in higher education. Besides this demographic information that was included in the feedback survey, given the home institution we can state that most respondents (specially those in higher education) have experience in e-learning: some of them work in distance education universities and others use a blended learning mode. In the next section we present the findings related to each dimension considered in the evaluation of the tool, as explained in “method” section. Research findingsThe testing done in the first phase (with the TALOE case studies) showed that the majority of the teachers have defined the appropriate e-assessment methods for the learning outcomes. In some cases, the TALOE web tool suggested additional possibilities in terms of e-assessment methods. For several cases the TALOE web tool indicated that the existing assessment methods should be revised. The analysis of received feedback in the second phase (through the web form) from 42 participants is presented in Figs. 9, 10, 11, 12, 13, 14, 15, 16, and 17 and explained in this section. The legend in those figures ranges from 1 (totally disagree) to 5 (totally agree). In the interpretation of results, for calculating percentages we have added replies 1 and 2 as disagreement and replies 4 and 5 as agreement. We have considered reply 3 as neutral. Fig. 9 Easiness of the TALOE web tool Full size image Fig. 10 Friendliness of the TALOE web tool Full size image Fig. 11 Usefulness of the TALOE web tool Full size image Fig. 12 Skilfulness of the TALOE web tool Full size image Fig. 13 Quality of the TALOE web tool results Full size image Fig. 14 Agreement of the TALOE web tool results with previous assessment plans Full size image Fig. 15 Validity of the TALOE web tool Full size image Fig. 16 Help provided by the TALOE web tool Full size image Fig. 17 Recommendation of the TALOE web tool Full size image The majority of participants found TALOE web tool easy to use (Fig. 9). This is an important feature as it influences teacher’s attitude towards the use of the web tool. Answers to the question about use of the web tool without written instructions are distributed along all ranges (Fig. 10). The majority of participants (64 %) did not need any written instruction to use the tool. This indicates that the tool is well designed, but also that participants may be experienced in defining learning outcomes and familiar with the revised Bloom taxonomy. Nineteen percent of participants needed additional instructions to be able to use the tool. Although some participants needed instructions to use the tool (Fig. 10), the majority of respondents found it is very useful (Fig. 11), which is very relevant for the project. Some participants (17 %) did not have an opinion about the usefulness of the tool. This perception can depend on their knowledge and experience in defining learning outcomes and the extent to which they have been achieved. Those more experienced may find it not as useful as they are quite experienced in writing learning outcomes and linking them with teaching and assessment. Eighty-six percent of respondents found the web tool easy to use or become quickly skilful with it after first use or by reading the posted instructions (Fig. 12). This result is relevant for the project, as it compensates to same extent the difficulties that first users may experience (Fig. 10). Results provided by the TALOE web tool, i.e, recommendations about assessments methods for each learning outcome, are of good quality for most testers (Fig. 13). Some participants (26 %) could not conclude about the quality of the received results. The majority of participants (81 %) found that received results are in agreement with their own assessment plans (Fig. 14). Only 5 % of respondents found that the assessment methods recommended by he TALOE web tool don’t match their methods. This is an interesting result that would require further research: at first sight it suggests that professors are correctly aligning their learning outcomes and assessment methods, and that the TALOE web tool identifies this alignment. In relation to the previous question (Fig. 14), most respondents state that the tool helps them to better align learning outcomes and assessment methods (Fig. 15). It could mean that the tool reinforces the assessment plan they were already using (given that there is a high match between the recommendation provided by the tool and the existing plans, Fig. 14). Almost 30 % of respondents don’t have a clear opinion about this question. The majority of participants (76 %) find the TALOE web tool can help them to understand how to better define the assessment methods in their courses (Fig. 16). Finally, the majority of participants would recommend the TALOE web tool to a friend; still, there are some of them (14 %) who are not sure about it (Fig. 17). The information collected from the stakeholders in the open-ended questions of the TALOE feedback form provides some clues about missing elements or interface recommendations that could help improve the tool. From the 42 respondents, 23 have submitted their comments about changes needed in the web tool, missing aspects and removable aspects. Regarding changes in the TALOE web tool procedure or interface, some relevant feedback is present in the following quotations:
About missing aspects in the web tool, we could highlight a demand for examples of use of the different assessment methods, expressed by the testers in the following terms:
There are not remarkable removable elements addressed by testers. In summary, the feedback shows a positive evaluation regarding the usability of the tool and the quality of the output/results, i.e. the e-assessment method suggestions. Invited users found this tool easy to use and useful, but would like to see it improved for more complex testing. These results also confirmed that the web tool provides support and guidance to teachers to formulate the learning outcomes in accordance to Bloom’s revised taxonomy, as intended and planned by the task. It also increases the accuracy of e-assessment methods received by the tool and the alignment between learning outcomes, assessment techniques and teaching methods. Discussion and conclusionsThe European project TALOE has successfully transformed the ALOA model into a web tool that is public and freely available to any interested user. The web tool helps users to align their intended learning outcomes in a course (defined using the Bloom’s revised taxonomy of learning objectives in the cognitive domain) with appropriate assessment methods. After a period of testing of the tool, the evaluation results presented in this paper report that the tool is easy to use, written instructions are helpful for users, the tool is useful for its purposes and provides valuable recommendations. The requested improvements that the TALOE project will try to address are related to two main aspects:
Besides this prospective that the TALOE project wishes to achieve, we would like to briefly highlight some concluding remarks about strengths and limitations of the TALOE web tool. A main limitation that users ought to be aware of is that the tool is focused on the revised version of Bloom’s Taxonomy of the cognitive domain (Anderson & Krathwohl, 2001) (Fig. 1), thus it does not contemplate other learning domains, namely the affective and psychomotor domains (Bloom et al. 1964; Kennedy et al. 2006), and the social domain of learning (Vygotsky, 1978; Wenger, 2000; Garrison & Anderson, 2003). It is important to notice this limitation, so users can be aware that not all the learning outcomes they state for their courses would be represented in the TALOE web tool (but only those in the cognitive domain). Among the strengths, we would like to highlight the value that testers of the tool have stated and the fact that they would recommend its use to their colleagues. In the basis of the European projects it is the dissemination and exploitation of the developed products. In this sense, we feel encouraged to share the TALOE web tool with any user interested in checking if the existing assessment methods in an existing course are in line with the stated learning outcomes, and in making decisions on the most appropriate assessment methods for a new course or module. An extended use of the TALOE web tool could lead to another research project, focused on analyzing the impact of the tool on the improvement of e-assessment and, furthermore, on exploring the impact of the use of the TALOE web tool on the improvement of learning, which is, at the end, the purpose of the educational experiences we design and develop. Notes
Anderson LW, Krathwohl DR (eds) (2001) A taxonomy for learning, teaching, and assessing: A revision of Bloom’s Taxonomy of Educational Objectives, Completeth edn. Longman, New York Google Scholar Biggs JB, Tang C (2007) Teaching for quality learning at university: what the student does (3rd ed.). Berkshire, England: Open University Press Google Scholar Bloom BS (1956) Taxonomy of educational objectives; the classification of educational goals, 1st edn. Longmans, Green, New York Google Scholar Bloom BS, Masia BB, Krathwohl DR (1964) Taxonomy of Educational Objectives (two volumes: The Affective Domain & The Cognitive Domain). David McKay, New York Google Scholar Boud D, Falchikov N (2006) Aligning assessment with long-term learning. Assessment and Evaluation in Higher Education 31(4):399–413, http://dx.doi.org/10.1080/02602930600679050 Article Google Scholar Brown G, Bull J, Pendlebury M (1997) Assessing student learning in higher education. Routledge, London Google Scholar Carless D (2007) Learning-oriented assessment: Conceptual bases and practical implications. Innovations In Education And Teaching International 44(1):57–66 Article Google Scholar EHEA. (2014). Bologna Process - European Higher Education Area. Retrieved September 15, 2015 from http://www.ehea.info/ ENQA (2009) Standards and Guidelines for Quality Assurance in the European Higher Education Area, 3rd edn. European Association for Quality Assurance in Higher Education, Helsinki, Retrieved September 15, 2015 from http://www.enqa.eu/wp-content/uploads/2013/06/ESG_3edition-2.pdf Google Scholar Falcão R. (2013). Aloa: a model for aligning learning outcomes and assessment. 45th EUCEN Conference: Transferring knowledge in a globalised world: a ULL responsibility. University of Geneva, Switzerland. 29-31 May 2013. Retrieved September 15, 2015 from https://www.unige.ch/formcont/files/5714/3921/7163/Falcao-CS-FINAL.pdf Garrison DT, Anderson T (2003) E-learning in the 21st Century: A Framework for Research and Practice. Routledge Falmer, London & New York Book Google Scholar Gil-Jaurena I, Aguado T, Malik B, Cucalón P (2015) E-assessment in a Master online course. A case study. INTED2015 Proceedings, Madrid, pp 3661–3667 Google Scholar Kennedy, D., Hyland, A. & Ryan, N. (2006). Writing and Using Learning Outcomes: a Practical Guide. Retrieved September 15, 2015 from http://www.tcd.ie/teaching-learning/academic-development/assets/pdf/Kennedy_Writing_and_Using_Learning_Outcomes.pdf Lõssenko J. (coord.) (2014). WP 3.01-TALOE E-assessment Practices. Retrieved September 15, 2015 from http://taloe.up.pt/wp-content/uploads/2015/03/AnnexXI_WP3_01_E_Assessment_Practices.pdf Ministry of Science, Technology and Innovation (2005) A Framework for Qualifications of the European Higher Education Area. Bologna Working Group on Qualifications Frameworks. Copenhagen, MSTI, Retrieved September 15, 2015 from http://www.ehea.info/Uploads/Articles/050218_QF_EHEA.pdf Google Scholar Rodríguez-Gómez G & Ibarra-Saiz MS. (2015). Assessment as Learning and Empowerment: Towards Sustainable Learning in Higher Education. M. Peris-Ortiz & J.M. Merigó Lindahl (eds.), Sustainable Learning in Higher Education (pp. 1-20). Springer. http://dx.doi.org/10.1007/978-3-319-10804-9_1 Soeiro A, Martins I & Correia T. (2014). WP 3.02 ‐ ALOA model with integration of e‐assessment practices. Retrieved September 15, 2015 from http://taloe.up.pt/wp-content/uploads/2015/03/AnnexXII_WP3_02_ALOA.pdf Iowa State University (2015). Revised Bloom’s taxonomy. Center for Excelence in Teaching and Learning. Retrieved September 15, 2015 from http://www.celt.iastate.edu/teaching-resources/effective-practice/revised-blooms-taxonomy/ Vygotsky, L. (1978). Mind in society: development of higher psychological processes. Harvard University Press. Wenger E (2000) Communities of Practice and Social Learning Systems. Organization 7(2):225–246, http://dx.doi.org/10.1177/135050840072002 Article Google Scholar Download references Author informationAuthors and Affiliations
Authors
Corresponding authorCorrespondence to Inés Gil-Jaurena. Additional informationAuthors’ informationInés Gil-Jaurena holds a PhD in Education and is an Associate Professor at the Faculty of Education, Department of Theory of Education and Social Pedagogy, National Distance Education University (Universidad Nacional de Educación a Distancia-UNED), Spain, where she has been teaching and researching since 2005. She has been involved in various European projects related to open and distance education. Sandra Kucina Softic hold a Master’s Degree in Digital Education by the University of Edinburgh and is Head of the E-learning Centre and Assistant Director for Education and User Support at the University Computing Centre - SRCE University of Zagreb. Previously, she was Coordinator for Development and Implementation of Information and Communication Technologies at the University of Zagreb for ten years. She is a member of EDEN Executive Committee. Rights and permissionsOpen Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. Reprints and Permissions About this articleCite this articleGil-Jaurena, I., Kucina Softic, S. Aligning learning outcomes and assessment methods: a web tool for e-learning courses. Int J Educ Technol High Educ 13, 17 (2016). https://doi.org/10.1186/s41239-016-0016-z What are aligned learning outcomes?A learning outcome is a clear and specific statement that identifies what students must demonstrate at the level and standard required to successfully pass their study at program and course levels. Read more here.
Is there alignment in the learning outcomes?Building alignment between assessment and learning outcomes also allows you to develop and communicate the pathway for students' learning progression. It enables you to explain what knowledge and skills were expected on entry to the course and the knowledge and skills that will be developed throughout the course.
What is alignment of learning?Alignment of learning, teaching, and assessment is a process where each of the critical elements (learning objectives, assessment of learning, and anticipated knowledge, skills, and/or dispositions) of instruction interact and support learning outcomes.
What is the importance of learning alignment?Alignment between activities and assessments helps minimize wasted time as students are able to focus on skills geared towards the learning objectives. Instructors can ensure that the course objectives are clearly defined at the beginning of the program by providing a table for the module.
|