Evaluation of learning attributes of courseware for university science courses

The quality and attributes of the learning achieved by the use of technological tools such as computer aided learning programs (CAL) or web-based packages are difficult aspects to evaluate effectively. Learning technology should fulfill the needs of the student users as well as achieve the pedagogical goals of the teachers. Excellence in this context ideally should comprise not only a high quality product in technical terms, but also one that fulfills the learning aspirations for which the product was designed. In the past, evaluation of technologically delivered learning has focused more on the quantity of knowledge transferred than the quality. Recently a number of evaluation tools or packages have been devised in the search for something that will provide information for teachers using technological teaching aids, regarding not only how much their students have learned, but also what depth and type of knowledge they have gained.

An ideal evaluation process in this context might be expected to combine the efficiency of a quantitative method with the richness of qualitative information, but should also be easy to use and, importantly, easy for the teacher to interpret in terms of improving their students' learning experience. Such an evaluation package should also be responsive to the needs of the users, rather than simply imposing an objective ideal of quality derived from externally imposed standards.

This report details the development of an evaluation method based on the premise that the most effective learning experience will be achieved when a technological tool is designed to align most closely with the pedagogical requirements of the clients, both teachers and students. The study sought to establish whether such a correlation could be identified and expressed in a format the academic client could easily comprehend. The evaluation method was designed specifically for university level teaching, and trialed in undergraduate science units. The evaluation process was trialed with three successive CAL programs in order to develop the method, with modifications being made to the evaluation process after each trial. The modifications to the evaluation process were based on critical reflection of the effectiveness of the previous evaluation cycle. The evaluation reports produced for this study were used diagnostically to pinpoint factors with the potential to reduce the effectiveness of the learning experience. Recommendations for improved implementation resulted :from negotiation with the academic clients regarding the areas of conflict highlighted by the evaluation.

In order to express the learning quality and characteristics of CAL programs, the Pedagogical Dimensions profile developed by Reeves (1994) was adopted as a framework. The profile consists of a semantic differential scale for each of fourteen pedagogical dimensions. Amendments to the Reeves scheme were considered desirable to allow improved function in this context and such amendments have been proposed. The evaluation process was designed to facilitate comparison of the pedagogical characteristics of courseware with the pedagogical needs of the stakeholders, in this case the lecturers and students using the program. The intention was to determine the degree of correlation between stakeholders' requirements and courseware characteristics in order to establish the programs' fitness-for-purpose, in terms of pedagogical characteristics. Based on surveys and interviews, pedagogical profiles of both lecturers' and students' perceptions were prepared and compared with a profile of the learning software prepared by a panel of reviewers.

The resulting comparison of profiles was intended to evaluate the appropriateness of the programs' learning attributes, rather than to measure quality. As such, it should be viewed as a means of employing a program to the best advantage, by maximising the effectiveness of the learning experience. Each dimension was examined for congruence or dissonance between the results of the three profiles (student, academic and courseware). Courseware was considered suitable in the relevant context if the profiles exhibited congruent results. Dissonant dimensions were drawn to the attention of the academic, in order to negotiate a means of effecting improvement.

Having critically assessed the evaluation process it was noted that the pedagogical dimension profile functioned very effectively as the foundation for an evaluation process designed to determine learning effectiveness of CAL programs. However, certain aspects of the pedagogical dimensions were found to be less than ideal and critical examination of these aspects led to proposals for changes to the profile in order to improve its capacity to serve as an evaluation tool.

In order to clarify the profile results to permit easy interpretation, the dimensions of the profile were rearranged into several broad groups. This modification of the array was justified by increased convenience for the user, and improvements to the theoretical grounding. The clusters to be employed (Hannafin, 1997) were as follows: psychological, pedagogical, technological, cultural and pragmatic.

It was considered that alteration of several existing dimensions and inclusion of some additional dimensions would facilitate application of the profile. A new dimension of Curriculum integration was devised to recognise the capacity to link computer learning with existing teaching programs. The need for reflective learning was recognised by including a new dimension called cognition type, while the students' capability to manipulate and interact with the program was reflected in a new dimension called interactiveness. Revised names were proposed for several dimensions including Engagement instead of Motivation, Informative feedback rather than Value of Errors, and Program modifiability for Flexibility. A modification was proposed for the Underlying psychology dimension in an attempt to better define a continuum of perspectives. The modifications were applied to a case study and elicited a more informative and diagnostically useful report than the original method.

An effective evaluation method has been designed and tested, which facilitated the diagnostic assessment of courseware in the context of a unit of study. Alterations to the application of the courseware in practice were found to improve student acceptance of the programs. A modified profile was designed and applied, and found to be even more accessible and useful for academic clients.