Spanish for lawyers was one of the structured activities in the PIPVIC (Piloting IP VIdeoconferencing) project (Sasse et al. 1998), in which multicast videoconferencing is being used in a number of typical activities in institutions of higher education in the UK: research, teaching and administration. The aim is to understand the issues which are involved in using multicast videoconferencing for the tasks like small group tutorials.

23/2/98 – 16/3/98, 1 one hour and 1 two hour session per week.

Two students at Exeter University. One tutor at UCL. A number of observers and technical support staff at both sites.

The students were using two SGIs. The tutor was teaching from a Sun workstation. A Sun workstations was used for playing out videotapes used during the tutorials. A VCR was plugged into the Sun workstation and the audio and video were played out through the audio and video tools. The VCR was controlled by one of the technical support staff at UCL, and the tutor had to ask for the videotape to be played, stopped, rewound etc.

The following SHRIMP tools were used: RAT (audio tool), VIC (video tool), NTE and WB (shared workspaces). As it was, both students and tutor used separate tools. [IS THIS TRUE].

The tutorials were very varied. The tutor would give the technical support staff pages of text (questions, newspaper articles etc.) which would be scanned and imported into WB. She would also make use of video clips presenting Spanish law cases.

Every week the tutor would send articles by post to the students which they had to prepare for the tutorials.

The tutor, students and technical support staff at both sites were requested to fill out a short web-based questionnaire after each session. They were prompted to rate the quality of the audio and video on a scale of 0-100. They were asked to rate the quality in terms of how adequate they thought the quality for their tutorial. There was also textboxes for making comments on the audio and video and to make comments on the session in general.

When the course was completed, the students, tutor, observers and technical support staff met to discuss the outcome of the course. The students were in a separate discussion group from the tutor and observers.

Finally, the students and tutor were asked to fill out questionnaires covering their overall impression of the course.

In the following sections, each of the three stages of the evaluation will be presented together with the conclusions which have been drawn. The final section will present the overall conclusions on the basis of all the stages of the evaluation.

Review of Audio Performance

The audio quality overall was very good. The average rating of the audio quality was 75 per cent. Both students and tutor felt very happy about their ability to hear one another particularly with respect to their understanding of the Spanish words and phrases. The student’s did however feel that the audio quality of the videotape was not really good enough for them to fully understand the Spanish, given their level of language proficiency. This is most likely due to the quality of the tape recording itself since the programs are quite often recorded from one tape to another in order to produce the final version. Another difficulty, which unfortunately was not discovered until after the session, was that silence suppression had been kept on in RAT through which the videotape’s audio was being transmitted. This has a tendency to clip parts of the speech and since most of the background sounds such as music are cut out, it can sound very interrupted and is difficult to comprehend overall.

The problem with the faulty set of headphones at Exeter which became a problem at the very end of the French for lawyers course, had unfortunately not been resolved. There were a number of sessions where the students and the tutor complained of a continual background "hum" which was at times quite disturbing. This is almost certainly due in part to the headsets since on one occasion, when one of the students joined in with the session which had been going for a few minutes already, the noise began immediately although the student whose headsets were at fault, couldn’t hear any of the interference.

In discussions after the series of tutorials, the students mentioned that they had found the volume amongst participants to be unbalanced. It was unfortunate that these comments were not voiced earlier on in the series since simply adjusting the individual volume controls of the microphones of the participants can rectify the imbalance.

It is noticeable that there are often misunderstandings amongst participants with the meaning of the term "video". In a session where a videotape has been played, questions such as "Please rate the video quality" will often receive an answer purely relating to the tape that has been shown. For future trials, it may well be worth developing a different terminology to eliminate these discrepancies.

The quality of the students’ and tutor’s video was generally regarded by the participants to be adequate for the purposes of the tutorial. The average rating was 78 per cent. The students received a mix of video quality from UCL, as the frame rate ended up depending on who provided the technical support at UCL at each session. The tutor’s image would either be transmitted with 8 f/s or 2 f/s. This however, produced some interesting reactions from the students. They did notice that the quality changed, especially when the framerate dropped and that as the lip synchronisation deteriorated it was not as easy to understand the tutor. The video image, although not the most essential part of the videoconferencing toolset, definitely helped to make the students feel more at ease and comfortable in conversing with the tutor, this becomes even more important in a situation such as this where the participants are conversing in an unfamiliar language

The videotape was more of a problem. Over the course of the tutorials, two tapes were shown. One of them was a documentary about euthanasia, the other a scene involving a lawyer and his client. The documentary was overall easier to understand because the soundtrack was mostly narrated at a slower pace than normal speech. Unfortunately the programme was based around a village in Spain which speaks more of a dialect as opposed to pure castellano and so subtitles were included to help with the comprehension of the language. The video images were not big enough to adequately read the text of the subtitles and so the students suffered as a result. The second video was much harder to comprehend than the first. The speech was much faster and the quality of the actual video recording was not very good, and since the main part of the video was the dialogue with very little help from the visual images, the students found it mostly incomprehensible. The system of operating the video player was still the same as for the French for lawyers course where the video player was connected to a different machine and operated by a member of the support staff. This is not the most practical system from the point of view of the tutor, in that they themselves are unable to operate the tape. We are currently working on a solution to record the tape in advance and store the data in a file that can be played back by the tutor. This will benefit both the tutors and the students whereby difficult passages or important pieces of dialogue can be paused and replayed immediately following the initial hearing.

Halfway through the tutorials, the students switched to using an integrated interface rather than individual tools. In the integrated interface, it is only possible to enlarge one of the images at any one time. According to the technical support person in Exeter, this vastly improved the quality of both audio, video and shared workspaces for one of the students who until then had used up most of the computer’s CPU power, opening, closing and moving windows around on the screen.

The text editor was not used very much during the sessions, and most of the work was conducted with the aid of the whiteboard. Over the course of the sessions the format of the slides was adapted according to the suggestions of the tutor and the students. On a number of occasions, the tutor wanted to display some newspaper articles for the students to discuss in the tutorial. The only version available was a photocopy of the original print, however once this had been scanned and saved as a postscript file, the quality of the text on the whiteboard was barely legible. For the next session, more time was spent in scanning the documents and the image was run through an OCR programme that was able to recognise over 80% of the text. About 10% of the characters were lost due to the inability of the software to recognise Spanish characters such as ‘ñ, é, ó’, a function which is available in more recent releases of OCR programmes. The rest of the loss was due to the bad quality of the photocopy. The result was much improved although there were still problems with the WB interpretation of the postscript files at the Exeter end. This was possibly due to the font setting of the original document, in retrospect the files should have been posted on the WB as text-only.

The students commented that under certain circumstances WB was not as effective as NTE. In particular this was when the students were requested to fill in text that they heard on the videotape into gaps on a postscript slide. The students were unaware that the text size is adjustable and they found that it would not necessarily fit into the gaps already made. The only way to position the cursor is with the mouse which means that good mouse/keyboard coordination is required.

Overall the students and tutor enjoyed the experience, commenting that it was an effective way to bridge the problem of distance learning. The tutor commented at one stage that it was hard from the teaching point of view to really know whether the students were fully understanding her since she could not see their expressions clearly enough, although as the sessions progressed she became noticeably more relaxed and accustomed to the technology.

One of the initial concerns from the tutor was the relative level of the student’s language comprehension. As the tutorials progressed however, this proved to be less of a problem and both of the students seemed to benefit very much from the sessions.

The tutor commented that the sessions were noticeably more productive when the students had had a chance to read over the material which she wanted to discuss in advance of the tutorial. They were able to concentrate just on the screen and the keyboard as opposed to trying to refer to extra sheets of paper or relying purely on the slides in WB for reading the articles.

Discussion with Students

This section lists the main points arising from the students’ workshop. The Workshop was held at UCL on March 28th.

At first the technology can be quite intimidating, both students agreed it would be productive to organise at least one full length training session in advance to help them get accustomed to the tools. By the end of the series the students found they could really relax and concentrate on the discussion instead of dealing with the controls on the desktop at the same time.

It is important for the tutors to be aware of the fact that the students are learning in very unfamiliar surroundings and they must not get frustrated if they take a little longer to do something.

The slow framerate of the tutor’s image during most of the tutorials was quite distracting for the students. They felt that lip synchronisation was quite an important way of reassuring the participants during the tutorial and helping them to feel like they are participating in more of a real discussion.

They found the videotapes, which were played, very difficult to understand for two reasons: -

• The level of the Spanish was quite high, there was a lot of legal terminology which they did not understand.
• A combination of the poor sound quality from the tape and the slow framerate made it difficult for them to fully understand what was being said.

One of the students commented that it was often quite distracting not being able to hear your own voice through the headphones, saying it is like talking with earplugs. The students had also noticed that one of the pairs of headsets from Exeter had a faulty connection that caused a low hum in everyone else’s audio.

Another issue which was discussed was that of balance of the volume of the audio between participants. The students were not fully aware that the volume of the individual microphones needs to be adjusted individually which requires a period of testing before the tutorial begins.

It was also agreed that comfortable headphones are essential when participating in a long teaching session, otherwise if the headphones rest heavily on the ears it can cause a great deal of discomfort, which in turn will prevent the participants from concentrating adequately.

People wanting to listen into the sessions from the same room as the participants must be aware that any background noise such as talking can be extremely distracting for the person trying to concentrate on the remote discussion.

The integrated interface at first seemed quite complicated to one of the participants who initially preferred to run the tools individually (this student had taken part in the French for lawyers tutorials and was very familiar with using individual tools). However, as their confidence in the technology grew it became a more practical and preferred method of controlling the tools. Having everything in a confined area was neater and involved far less moving of screens around, although they did feel it was a shame not to be able to see both the shared workspaces at the same time.

For practicality, the NTE was regarded as the much easier method of sharing text, especially when taking notes at the same time as listening to a videotape, WB requires a greater degree of keyboard/mouse co-ordination.

The students asked whether it would be possible to reconfigure the shared workspaces to accept Spanish characters. They considered this to be an important addition to improving the tools for language teaching.

The shared workspace was considered to be a major benefit in making the tutorial more interactive, as it is a function which would not normally be available in a face to face class and allows for more individual attention from the tutor.

The students felt that it would have been helpful to have kept a copy of the edited slides on WB after the lesson. This would also have been more conducive in encouraging them to take notes during the session on the shared workspace as opposed to on paper.

The legibility of the slides on WB was a key point for the students. It was important for them to be able to read all the text clearly, especially since there were times when they had not been able to read the material in advance of the session and were thus much more dependent on the information on the screen.

It would help if fewer people were involved in the smooth running of the session such as the technical support staff for each participant. One of the students suggested trying to teach the tutors how to give technical advice to the students in the event that they are having a problem with the tools.

The two weekly sessions were of different lengths. The Monday evening session was two hours and the Thursday evening session was one hour long. The students felt that the two-hour session was too long and the extra hour was really not very productive.

The most effective size for the tutorials was considered to be between two and three students, although three was preferable. The reason for this being that there are still few enough students for the tutor to give them individual attention, but there is not so much demand on the individual students themselves to respond to questions and maintain the general flow of the discussion. With a third person present, there is more opportunity for a student to listen and think about the substance of the discussion and ultimately to contribute their ideas.

The three most important things, which according to the students, should be changed: -

• The framerate on both the videotape image and the image of the tutor should be higher, although the videotape is more essential.
• Having an area somewhere on the screen for the individual participants to type notes would be very useful, they should only be viewable locally to the machine that they are using.

• Comfortable headphones.

The following is a summary of the discussion with the tutor and observers. This workshop was also held at UCL on March 28th.

The course only had two students, again due to the persistent problem with audio cards for PCs. The two students turned out to be very different in their Spanish proficiency. This was one of the main points of concern for the tutor. It was agreed that prior to remote tutorials, the tutor should have a 10-15 minute remote meeting with the students to assess their level and discuss issues like how much homework to give etc.

The fact that the level of the two students was so different, also raised the issue of having individual channels for private conversations between a student and tutor, similar to the tutor sitting down next to a student in class to talk if the students appears to have problems. Initially, until we can make the technology support this, we can schedule one student to arrive 10-15 minutes earlier than the other in turn, in order to have some "private" time with the tutor. Alternatively, the tutor and students could talk to one another on the phone.

The tutor had to make the course easier towards the end as one of the students simply could not keep up with the planned course. The tutor was very unhappy about this and said that compared to a parallel course she was running, the students did not learn nearly as much. She said it was more relaxed, but a waste of time. The students, however, hugely enjoyed the end of the course and felt they had learned a lot. This, to a certain extent, is the same experience as the tutor had in the French for lawyers course.

Issues relating to the physical setup were raised:

Camera position. For people whose experience with video images of "talking heads" are mainly from TV and news programmes, having images of people who are not looking into the camera can be disconcerting. There is no eye contact. Presently, there is not much we can do about that, as we cannot ask people to look into the camera while they are working. The situation can be improved by putting the camera on top of the workstation if possible, so it looks like the person is looking in roughly the right direction.

The field of view on our current cameras is rather restricted. This means that if a students bends down to take notes, he or she disappears from view. We should experiment with more wide angled cameras.

Lighting. This is an issue which keeps coming up. It is important to ensure that the lighting in the room does not make the participant appear too light or too dark.

The size of the video images was deemed to be fine. The tutor would enlarge one or both of the students at different times.

The number of students was briefly discussed. The tutor would have liked to have had more than two students. She said that if there are more students, they can share the load, so the rest do not have to be alert all the time. This is a view reflected by the students.

The tutorials consisted of 1 one-hour session and 1 two-hour sessions a week. The tutor much preferred the one-hour sessions because of the intensity of the tutorials.

The shared workspaces were discussed to some extent. The main problem was that the tutor was not always sure what the students were looking at. Initially, she was not aware that if she clicked on a WB page, the students’ WB would display that page also. Another time, one of the students decided to use NTE rather than WB, which caused confusion. The tutor was not a very confident computer user and one or two bad experiences with the shared workspaces made her mistrust them completely.

One of the problems the tutor experienced was when she wanted the students to work individually. Neither of the shared workspaces supports this very well. One solution might be to have a console with shared workspaces, parallel to the video tool, and the tutor being able to enlarge each of them in turn.

Neither of the shared workspaces could produce Spanish letters, which were a minor irritation. NTE can probably be modified to allow this, whereas this is not possible with WB. The observer suggested using a graphics pen and simply (hand)-writing on WB. In the long run, a shared workspace must be found which can support a variety of alphabets.

In general it was felt that the tutors must have more thorough training in the use of the tools before use, in order to make them aware of the possibilities and restraints that they impose.

Finally, the possibility of building up repositories of language teaching material which would be available for the tutors was discussed, including the possibility of access to online dictionaries.

Summary of Answers from Student Questionnaires

[QUESTIONNAIRES NOT YET RETURNED]

[QUESTIONNAIRES NOT YET RETURNED]

The quality of the audio and video was a few exceptions deemed to be adequate for the tutorials. The average rating was 75 per cent, and this includes a rating of 95 from one of the students accompanied by the comment: "Audio quality was very good today and there was no sign of the gaps that interrupted the last session… for what we did today the sound could not have been better". This tells us that we cannot expect a rating of 100 per cent even if the quality is found to be adequate.

However, the tutorials did suffer from broken up audio at times. The low ratings at times seem to be due to an echo which interfered rather than packet loss. The echo was probably caused by a faulty headset which one of the students was using.

The quality of the video was also generally adequate. Though, the students did not find the video image of each other and the tutor crucial for the tutorials, they found that if it was there, it should be fast enough to allow some sort of lip synchronisation to take place. The quality (framerate, resolution and size) of the videofilm should be sufficient to read any subtitles and other text which might appear. This will be different from film to film.

Bugs

Students and tutor agreed that one-hour sessions are preferable to two-hour sessions, and that there should be more than two students – preferably three or four.

The material to be displayed on the shared workspace needs to be prepared well in advance of the tutorial and tested with the site which will be receiving the data.

The main issues covered by the discussion with the tutors and observers related to training/preparation and shared workspaces.

• Initial meetings must be had between tutor and students to ensure that the level of the students is "right" for the course.
• The tutors must receive better training in order to be more aware of possibilities and restrictions imposed by the system.
• Students should also receive adequate training to make them accustomed to using the equipment, in order that problems for example with the balance of the audio can be rectified from the beginning.
• Shared workspaces should support students workings individually.
• The system in general should support separate channels in order that the students can work in groups, and to allow the tutor to talk to students individually.

Good equipment such as that used for recording or playing back tapes is essential, as is good, comfortable headphones.

It is important to clarify the terminology in advance of the evaluations to the participants.

In general, students and tutor were enthusiastic about the possibilities that the system has to offer. As an added bonus, the students found

M. A. Sasse, L. Clark & C. Perkins (1998): Piloting IP Multicast Conferencing of SuperJANET: The PIPVIC Project. In Proceedings of Networkshop 26, Univ. of Aberdeen, 31-March -2 April 1998.