I launched a survey in September 2014 to get a feeling of how much lecturers could handle and to get some feedback. All in all it gives a very positive picture. The survey was sent to 53 people, i.e. to 43 people involved in D-HEST exams and to 10 main examiners of ETH also involved in moodle exams.
21 people (39%) responded to the survey. Most of them are main examiners and 11 are from D-HEST.
Entering questions was the most tedious task I did for the lecturers. Already before the summer session 2014 I had encouraged people to do it themselves, hire somebody to do it or use the available templates (for import). This graph clearly shows that examiners can get it done.
The following question tried to find out the ‘happiness-level’. I would say, it is quite high!
The “experience in general” is rated ‘very good’ or ‘good’ by 72%. This is very cool!
Computer rooms are also okay though one can see that there have been some bad experiences. 14% rate them as ‘bad’ or ‘very bad’.
The moodle-happiness-level looks like this:
I did not expect such a good evalutation. Moodle is not intuitive. It is a great tool when you know some tips and tricks and your way around it.
Entering questions is one tedious task. Only 5% rate it ‘very good’.
The results about the correction show what I expected. 67% are (very) happy (‘good’ and ‘very good’). This is one of the advantages of exams at the computer that are mostly mentioned. Unfortunately, if you have to do some re-marking or checking on already automatically corrected questions, things get tricky. Some people, unfortunately, did not ask for my (or anybody else’s) help and ran into problems.
I needed to know if the level of support I was offering was really needed. Seeing that 50% of the respondents did not think so (‘disagree’) or did not care (‘neutral’) showed me that it was possible to do without or with less.
All lecturers and supervisors have to go to a workshop on technical support if the exams are done at the computer. I think that LET should be responsible for this technical support. Almost 30% do not agree with me. In the end, it is a question of how much support is needed and how many people can be around. So far this session (winter session 2015) not much technical support has been needed as everything is just going very smoothly.
Every student’s seat is prepared before the exam: an instruction sheet is laid on the keyboard and a sticker (name and Legi-Nr) is put on the monitor. I wanted to know if people thought that this was necessary and was astonished to learn that people appreciate this.
Below are the last three questions which I will not summarize because every comment has its own importance but I have made some comments (check out the comments made with hypothes.is along the right border of this window; only visible in online version).
| Q9 | What is your take on the computer examination infrastructure? (Enough rooms, nice rooms, bigger rooms, rooms available during semester, computers, keyboards, need of lockers/locker room, noise level, room temperature,…) |
| 1 | The Rooms E19, 27 are not Suitable for Exams cause of missing Air Condition (Students in these rooms have not the same chances). I Think it would be good to have in all rooms these “Trennwände” so that each Student has his personal Space. |
| 2 | no experience with that |
| 3 | infrastructure is sufficient |
| 4 | The infrastructure is very good as is. |
| 5 | It would be REALLY NICE to have the large computer room also available during the end of semester examination time. For big classes (we have up to 200 students) it is quite complicated to organize the exams otherwise (but it does work). |
| 6 | Perhaps a room should be available during the semester, in order to perform a practice exam, for the students to get used to |
| 7 | more of the larger rooms (as G1) are needed (to safe personnel running the exams), rooms should be CLEAN before arrival of personnel (no trash around), allocated and large enough space for students’ clothing+bags(not available in largest room), cooling of rooms facing streets (E19, 27 I believe) is needed (very unequal conditions when these rooms are used in conjunction with G1 E26.1.3), less noisy keyboards would be fantastic for free-text -> sounds like pooring rain during the exam |
| 8 | the rooms for the semester end exams are too small and we have too many rooms (up to 4) that need to be used 2x (with Schleuse). This is very time consuming and also annoying, especially if suddenly the system does not work as expected between the two exams. This is what makes the exam a bad experience although I very much appreciate to have the exam online. I would not want to go back to hand written exams. |
| 9 | We need more rooms. It would be ideal to have 500 slots for each exam (no groups swaps required). |
| 10 | Bigger rooms to do larger examinations, lockers or a pre-room to deposit bages |
| 11 | Cannot comment on that |
| 12 | The Situation to split large classes 600 students in to two groups, which are distributed on several rooms is a very bad situation. |
| Students are stressed in the waiting room | |
| The risk of failures, cheating and unfair conditions is always a highly critical issue | |
| And the organizational effort is huge | |
| Its a clear trent, that the amount of computer based exams is increasing | |
| Also the ETH is pushing this development so the sufficient infrastructure for enough students is a must | |
| 13 | No idea |
| Q10 | What else do you wish for and what needs to be done for computer exams @ ETH? |
| 1 | LET should test the Infrastructure before the first Exams (Software should run on all PCs). And they should Upgrade the Number of CPUs already, not during the first exam. |
| 2 | Examiners should be asked to offer a mix of question types and not only single choice questions |
| 3 | There should be enough people during the exams. |
| 4 | I always struggle somewhat with the organization in moodle. It would be nice, for example, if I could see in my question bank, which questions I have used for which exam.I know I can see them if I select the respective exam, but it would be nice to have an overview directly in the question bank. I guess I could create an extra category, but as far as I understand it each question can only be assigned to one category and I want to use my questions several times. So basically I tend to put all the questions just in one category… |
| 5 | drawing |
| 6 | – the problem with the units has to be solved. Perfect solution: a question type where a numerical answer can be used, followed by a drop down menu, where the students can chose the unit. the Cloze type comes pretty close, however, the correct unit should not be awarded the same amount of points as the result and the correct unit itself without the correct result should not be rewarded with points at all. |
| – the Cloze question type would be useful for some purposes, however only then, if the weighting of the different answers can be defined within the syntax code. | |
| – the k-prim question type has to be corrected. the issues with the answers disappearing is not usable. Also, the backup for the k-prim question types does not work, what means the backup is not working properly | |
| – I would wish for, that once the exam is performed, the whole exam data is safed. the only thing that can be adjusted is the correct answer (remarking) or in case of the k-prim questions if a statement is correct or incorrect. All the rest should be saved in a read-only fashion | |
| – Correction of the exam should in no way be able to affect the answers of the students | |
| – the backup should really contain All the information that the students produced at the exam | |
| 7 | Question 5. If I (or my delegate) can do that, this does not mean that I really want to do that … so this question should have a multiple-answer possibility. |
| Since teaching is a to all HEST-students, since computer exams are really great, since these should be run very carefully and properly, I feel that computer exams should have central support rather than every professor having to have an employee taking care of the technical stuff around the exam. | |
| Thus I favor to have a group of people who is very very familiar with moodle and can take care of all the administration around the exams. | |
| It could be that some of these are sitting within scientific groups but the groups being responsible for large exams might get some extra funding towards the salary of such a person who will clearly not only do this. | |
| If it’s running as is, I favor clear rules of what shall and will be done by LET, by the departmental teaching supporter and by individual professors (or their staff). | |
| 8 | The online platform should be more stable. We had a number of issues in our exam, related to keyboard language settings, server crashes and clickable equations, that should be absolutely avoided in the future. |
| 9 | We need templates for the important questions (especially K-prim) since it is not possible to teach every professor in moodle. |
| 10 | To include as many written exams as possible |
| 11 | Creating good computer based exams is quite challenging. |
| Support and coaching how to set up a best in class computer based exam would be an excellent service | |
| 12 | Be able to draw ie the students can diagram/show structures as answers. This would be awesome. |
| Q12 | If you have any other comments, ideas, feedback, etc. use this box. Please click ‘Done’ (further down) when you’re finished. Thank you very much! |
| 1 | It would be grate to have a Tool in Moodle to easy read and work on Questions already inserted into the System. So that it’s possible to Work on these questions directly in the System. (@Sarah: Hast du allenfalls schon eine Anforderungsliste geschrieben?) Then it Would be possible to build up a Question Database for the exams where the good ones can be taken out insertet to the Quizzes (Upload from the template needs a download in an other format to change some important features, like shuffling the answers. And doing so errors may occur) |
| Import Export of Kprim questions is a need! Possible the Import should be done over an easy format so that it’s possible to generate an Template for the Lecturers. | |
| Building up a TestCase for new Questiontypes, so that the Problems we had with Kprim won’t appear again with other new Questiontypes. | |
| 2 | For me computer exams provide no advantage over paper exams. A service for scanning answer sheets would be a lot more helpful. |
| 3 | Sarah is doing an incredible Job: assisting, helping out in any possible way, answering questions of the professors/lecturers and ensuring a smooth Computer exam. |
| 4 | The only difficulty I had recently was when I discovered that one of the default “correct” answers was wrong. After the exam had been taken, this could not be changed (for the automatic correction). |
| Otherwise, it is a brilliant tool. | |
| 5 | Question 5: |
| We can do all the things, as stated in the question. However, with the exception of two points (Change questions’ assessments (if necessary) after the exam (re-marking); Enter grade calculations for Swiss grades) we do not want to do it. These two points we unconditionally want to do ourselves. | |
| 6 | Question 6 – not all of it applies … |
| 7 | LET is a great help / but for large classes we need more support and infrastructure |
| 8 | Sarah is wonderful, thank you. I was pleased with the grading of the exam–I could do it very quickly since it was legible. I would like to consider this for my toxicology class in the spring. I have around 80 students taking the exam. But I need to include drawings at least in the questions, preferably in the answers |