The University of Ulster has been a user of Questionmark software for many years, since around 1996. It cut its teeth on the “For Windows” version of the software, using it in a standalone capacity within small pockets of enthusiastic academics. It was not until 2000 when the institution awarded a grant for a project to implement CAA university-wide that the decision was made to upgrade the existing systems to the latest version of Questionmark Perception and to deliver to its students the many features and the accessibility of online assessments that Perception could offer.

According to Keith Adams, Senior Lecturer and Course Director in the School of Health Science, “Once we had received the grant we established a committee for developing a CAA strategy across the university and part of the initial decision process had to be the software that would form the basis of that strategy. We looked around to see what was available at the time and we looked at other institutions’ CAA projects. Very quickly the choice became clear cut. Questionmark Perception offered the levels of support we required, a whole host of features not present in any other software, as well as accessibility, availability and reporting facilities that were second to none. It quite simply came out on top.”

The broad objective of this project was to improve the effectiveness of lectures and to provide students with an additional resource that would help them to gain maximum benefit from their studies both within the university and from home. Before the introduction of CAA, students would be lectured for 12 weeks and then tested on the material taught. This would tend to be a very time-consuming, paper-based, hand-marked process, making the feedback to students less than timely and therefore less than useful. Also, it did not give a clear understanding to the tutors of the level to which students were engaging with the course material and therefore the expected retention of students to the course itself.

Keith Adams comments, “Feedback to students is essential but it needs to be timely. There are those who consider that feedback should be delivered to students within 48 hours or you may as well not bother. This is something that is difficult to achieve when you have very large groups. Perception overcomes the problem by providing instant feedback and explanations to the students as they work through their questions. In effect it acts as a virtual tutor. It also gives us reporting tools that provide comprehensive statistical reports on the performance of students and the efficacy of questions. As a result, we can hone our classes to accurately reflect student needs and concentrate our efforts on areas where the students are struggling, giving a more tailored approach to their studies. Also, we can spread the student effort more evenly through each module. If they know that they are to have a test every two or three weeks, they are more likely to attend classes and to read through the hand-outs and to talk through their issues with their peers and with the lecturer as the course develops rather than cramming for the final exam.”

A central theme of the CAA project is the engagement of the students with the course material and the lectures. There is a widely held view that students do not gain a great deal from the traditional lecture scenario and that their interest is greatly heightened from a more interactive approach. To that end, the chemistry teaching program has been split into a series of handouts and module chapters. The times allocated for lectures are mainly used for groupwork, where students work on extended problems collaborating in small groups and where they can gain specific help from the lecturer on problems they encounter within each session. The aim is to provide a much deeper understanding of the subject and to encourage the students to work through problems rather than to learn verbatim. Essentially, the emphasis is taken away from lectures towards smaller group teaching, therefore using lecture time much more productively.

Keith Adams says, “The Government is very keen to see more students progressing through the higher education sector but to achieve this we need to be reducing the drop out rate, especially within the first year. CAA gives us greater powers to track each student’s progress through a given course and this information can be fed back to the students quickly and effectively. Then, decisions can be made and help given to the students as and when it is needed. We are already noticing an improvement in student retention since the system was implemented.”

In 2002/3 and 2003/4 more grant money was awarded by the Learning and Teaching Support Network (LTSN) (now part of the Higher Education Academy (HEA)) to the CAA project at the University of Ulster to improve teaching within individual subject areas. The aim of this grant was to provide the means for the university to continue to write questions using Questionmark Perception and to provide copies of these questions to be used by colleges across the UK. In this way, banks of questions for CAA are created for all subject areas and these are shared between all institutions, reducing the level of duplication of effort that might otherwise occur. Perception allows compressed files of question banks (Qpacks) to be produced so that they can be distributed easily by e-mail or on CD. Editing of all questions is made possible to allow an element of customisation by individuals and to enable the emphasis of questions to be changed or the feedback to be tailored.

In all, nearly 700 questions have now been written at the University of Ulster. All but a few have detailed feedback and a significant number contain graphics. Questions are presented in virtually all the available formats: MCQ, Multiple Response, Drag and Drop (Hotspot), Pull Down List, Select a Blank, Text Match, and True/False. The question banks created by the university are on introductory aspects of chemistry/biological chemistry.

At the University of Ulster CAA Chemistry questions are being used for two groups of 1st year health science students at the Jordanstown campus and a 160 strong group of science students on a first year Chemistry module at Coleraine. A two stage testing approach is being used: in the 1st stage students undertake, over a 2 – 3 week period, CAA practice questions, all of which have feedback. At the end of this period, in the second stage, students undertake an invigilated summative test.

It was felt that this approach would certainly spread student effort more evenly through the semester. Currently student activity on practice tests is being monitored and student evaluation has been very positive. In fact, it has been noted already that there is a positive relationship between the introduction of CAA formative tests and examination marks.

Use of Perception at the university is established and growing with time. There are a number of subject areas, for example radiography, physiology, anatomy, microbiology and engineering which require students to identify key features of graphical images by placing markers on them. Questionmark Perception offers great advantages in the writing of such questions. Thus other faculties want to develop their own questions and this will be encouraged and assisted by those already using CAA to their advantage. The process of developing new questions and augmenting question banks will continue and these will be shared with the higher education sector as before. There is one major new project to be undertaken in the imminent future which will see the integration of Questionmark Perception within the WebCT learning environment at the university.

Keith Adams concludes, “We have already experienced many successes with CAA and we intend to maintain the momentum we have built up. We are funded to talk about our projects with other institutions and we will be publishing the results of our current projects in a bid to help others to achieve similar success. We are confident that the integration of Perception with WebCT will be a smooth process and we look forward to working closely with Questionmark in the future, bringing the benefits of CAA to more students at Ulster and throughout the UK.”