TRI-ORM - Project evaluation

The independent evaluator undertook a heuristic evaluation of the website, focusing on navigation and ease-of-use in November 2007, and changes were made to the site to respond to the issues highlighted.

TRI-ORM heuristic evaluation report (rtf, 70KB)

 

CLOSE

The evaluation of the TRI-ORM workshops contained four phases:

1. Pre-workshop questionnaire (online)
   This obtained the background information from participants for example discipline area / role / experience and their motivation for attending the workshop.
2. Workshop questionnaire (paper based)
   This focused on participants' immediate reaction to the workshop. Qualitative feedback was obtained on what they liked/disliked and how they plan to use their new skills.
3. Workshop observation
   The independent evaluator attended and observed the workshops to evaluate the effectiveness of the training and obtain feedback from the participants.
4. Post workshop feedback (email)
   Participant were contacted six months after the workshop and asked how they have used the skills they learnt at the workshop. They were also asked to reflect on what was the most important part of the workshop was for long-term use and how they used the methods in research/teaching.

 

CLOSE

A detailed evaluation of the online module was conducted during the first year. Formative data was gathered as the module was conducted. Firstly, the evaluator participated in the module as a learner and kept a weekly diary. Secondly, the development team created a message board to enable participants to post any problems they were experiencing. A summative evaluation was conducted at the end of the first iteration of the module using an Online Questionnaire and a synchronous online focus, which enabled participants’ to feed back their experience of participating in the module. An evaluation report presenting the findings was produced for the module development team, they were able to use this feedback to make minor changes to the running for the module form the second year.

CLOSE

TRI-ORM provided six introductory face-to-face one day workshops, providing a basic grounding in online research methods. Workshops were held at ESRC Regional Training Centres (Qualiti at Cardiff and at the Cathy March Centre for Census and Survey Research (CCSR) in Manchester) and at the host University for the project. All workshops were oversubscribed.

Table 1 provides details of the workshops that were conducted, including the date, location, number of participants registered for the workshop and the number of participants who took part in the workshop. The original intention of the project team was to limit the number of participants at each workshop to 20. However, as the project progressed it was clear that there was such a high demand for places at the workshops, the project team decided to allow more than 20 to register for each event. The workshops presented in grey in Table 1 represent additional workshop that were requested and run during the project.

Table 1: TRI-ORM one day workshops

Table 1 illustrates that in total 179 (135 at planned workshops / 44 at additional workshops) people attended the series of one day workshops. This exceeds the estimate of 120 people that was outlined in the project bid. The project team has clearly met the objective of training a core group of people in basic online research methods.

1 Workshop Evaluation

The primary aim of the evaluation was to provide formative feedback to the project team on the workshops (areas for improvement) and to identify participants’ views about the success or otherwise of the workshops. A series of evaluation methods were utilised:

• Participants were asked to complete an online pre-workshop questionnaire: the aim was to gather data on the experience and level of knowledge of workshop participants prior to attending the workshop. The data was provided to the workshop team who were then aware of the discipline area and the level of knowledge of online research methods of participants. This enabled the workshop presenters to customise the workshop to fit the knowledge and experience of participants.
• Observation: The three workshops during the first year of the project were observed by the evaluator to provide feedback to the team on how the workshop might be improved; recommendations were fed into the next workshop. By the end of the first year it was agreed that the workshop format had been perfected and observation of the later workshops was unnecessary.
• Feedback questionnaire (paper-based): participants were asked to complete a paper based feedback questionnaire during the workshop. Participants were requested to fill in a section of the questionnaire at set points in the day to reflect on the workshop session they had just experienced. Participants were also asked to make suggestions how to improve the event, provide their overall reaction and describe how they planned to use the knowledge they had gained at the workshop in the future.
• Six month follow up online questionnaire: participants who consented to be contacted were sent an online questionnaire, six months after the workshop. The aim was to gather data on whether they had actually used online research methods, in order to ascertain the longevity of knowledge transfer provided by the workshop.

The following sections describe the key results from each of the evaluation methods.

1.1 Pre-workshop questionnaire

People who registered for the workshops were asked to complete an online pre-workshop questionnaire. The aim of the questionnaire was to capture profile data about the type of person who was attending the workshop, their currently level of knowledge of online research methods and the key areas of the workshop that were of particular interest to them.

The profile data was sent by the evaluator to the project team, who were able to see a profile of the participants attending the workshop e.g. their discipline area, their particular area of interest in online research methods, what they wanted to gain from attending the workshop. This information enabled the workshop presenters to shift the workshop aims and focus appropriately to suit the people attending the workshop.

Not all participants who attend the workshops opted to complete the pre-workshop questionnaire. However, the profile data gathered indicates the groups that the participants were drawn from. Table 2 provides details of the workshops and a breakdown of the roles of the people who opted to complete the pre-workshop questionnaire.

Table 2: Participants’ roles

Table 2 highlights that a wide range of users have been trained, the tendency is towards the Research student and Early Career researcher, which perhaps is to be expected as these are the groups who are most familiar with new technology and most interested and motivated to explore using new technology based research methodologies.

The pre-workshop questionnaire captured information on the participants’ institutions and discipline areas. This illustrates that participants came from a wide range of social sciences, medical sciences, government research organisations, university administrative sectors, a whole spectrum of disciplines and institutions. Table 3 provides a list of the institutions/organisations that participants were drawn from; the number that attended a workshop from that institution. A summary of the broad range of discipline areas is also presented alphabetically.

Table 3: Institutions and Discipline areas

The geographic spread of institutions highlights that participants were willing to travel some distance to attend the workshops (even though the workshops were spread across the country) illustrating the high demand for these workshops. The wide range of disciplines highlights the interdisciplinary interest in online research methods and the demand for training in online methods across all academic subject areas.

The face-to-face workshops were designed to provide an introduction and a basic grounding in online methods for people new to this area of research. The pre-workshop questionnaire asked participants to reflect on their current knowledge of online research methods. Table 4 highlights that the majority of participants saw their pre-workshop knowledge of online research methods as basic.

Table 4: Level of knowledge

The fact that the majority of participants at the workshops identified their knowledge of online research methods as basic illustrates that there was a clear need for the introductory level of training. It also indicates that the marketing and publicity for the workshops was clearly reaching its target group.

The pre-workshop questionnaire asked participants Which, if any, of the following online methods have you experience of as a researcher?’’. Table 5 illustrates that some participants had experience of online questionnaires but the majority had no experience of online research methods at all. These workshops were thus clearly catering for base level training needs.

Table 5: Experience of Online Research Methods

In the pre-workshop questionnaire participants were asked what they were most interested in gaining from the workshop. Clear categories emerged from the responses, for example; broader knowledge of online research methods, confidence, good practice, practical application of online research methods. Selections of quotes illustrating participants’ responses are included below:

It is clear that the pre-registration questionnaire was important for establishing the needs of participants’ prior to the workshop and this data helped in the allocation of groups within workshops. This information enabled the project team to ‘mould’ the training to participants’ needs.

1.2 Workshop Observation

The three workshops conducted during the first year of funding were observed by the evaluator (by the end of the first year it was agreed that the workshop format had been perfected and observation of the later workshops was unnecessary). The main objective was to provide immediate feedback to the project team on the delivery of the workshops. The evaluator wrote a report following each workshop providing formative feedback to the project team which included a series of recommendations for the next workshop. The team acted on the recommendations, leading to a series of minor changes to the following workshop format. For example; using data from the pre-workshop questionnaire to draw on participants experiences, tailoring the workshop to specific needs of the group, ensuring the host institution provided administrative support for workshop registration, slight changes to the structure of the sessions reducing the time spent on group feedback in the morning session, including key references in the delegate packs, extending access to Bristol Online Survey (BOS) for a period following the workshop to allow participants to complete their online surveys etc.

The six workshops followed the same format, which was modified slightly as a result of feedback during the first year and updated by the project team during the second year e.g. new ethical case studies were introduced, new references were added, timing of sessions was perfected, pre-workshop liaison with the host institution was improved etc.

A key aim for the project team was to make the workshop as practical as possible, enabling participants to gain hands on experience of a range of online research methods. These hands on sessions were highlighted by many participants in their feedback as one of the most beneficial aspects of the day. However, the project team had to put in a lot of liaison work with the various host institutions and faced a number of practical challenges to get these hands on sessions to work. The project team had to liaise with the host institutions IT support staff to arrange guest computer accounts for all participants; they had to ensure that the host institutions network firewall would enable participants to access the Connect software that was being used for the online interview practical session. Even when all these checks had been made on one occasion the project team faced technical issues on the day of the workshop accessing the interview software. Fortunately the issue was resolved by the local IT support staff in time for the session to run successfully.

The format for the workshops

The morning session focused on online questionnaires, covering theory and providing hands on activities. The opportunity to use Bristol Online survey (BOS) to design their own basic online questionnaire was highlighted by many participants as a positive element of the workshop, illustrated by the quotes from participants below:

The afternoon session focused on online interviews covering the theory and presenting an illustrative case study. A scenario was presented and participants were given the opportunity for hands on experience of an online synchronous interview. Working in groups, one member of the group was given the role of online interviewer and others adopted the roles of interviewees using the scenario that had been presented. Hands on experience really gave participants a taste of online interviews. Participants’ enjoyed the role play and also realised;

Workshop participants’ were asked to complete a paper based feedback questionnaire at the end of each session, reflecting on the session and providing their overall feedback on the workshop. Their responses’ are discussed in the following section.

1.3 Paper-based feedback questionnaire

The participants at the workshops were asked to comment on the best and worst aspects of each of the workshop sessions. This formative feedback enabled the project team to understand the aspects of the workshop that worked well and not so well. The questionnaires were transcribed and analysed by the project evaluator and feedback was given to the project team, enabling any necessary changes to the workshop format to be made.

The overall feedback from participants on the workshop was very positive. Some of their comments are presented below:

Participants were also asked to rate the workshop using a scale of 1 to 5, where 1 indicated not satisfactory and 5 very satisfactory (Table 6)

Table 6 - How satisfactory was the workshop

It is clear the workshops were rated highly. A majority of participants rated the workshop as 4 or 5, indicating they were very satisfactory.

Participants were asked to suggest other areas they would have like to see covered in the workshop or any improvements to the workshop for the future. Key themes that emerged from the comments were: to develop a two day workshop (repeated by many respondents), including more practical experience, a session on online experiments, a session on online surveys engines, a session on using the Web as a way of collecting/ presenting analysis, more examples of successful online research and information on the legal facts. The wide range of suggestions for areas to include in future workshops highlights the breadth of material that could be covered in the area of online research methods and the demand for training in these areas.

The workshops focused on ‘training the trainers’ with the aim that the enhanced knowledge of established researchers would be passed on (‘trickle down’) to research staff and postgraduates through follow-on teaching and supervision. The feedback provided by participants at the workshops illustrates that many intended to use their new knowledge in their own PhD research, their teaching or pass it on to their PhD students, as illustrated by the following quotes:

1.4 Six month follow up questionnaire

In the paper based feedback form participants were asked if they would be happy to be contacted after six months. Those who consented to be contacted were sent a link to an online questionnaire which enabled the evaluator to gather data on how they had used the information from the workshops. The qualitative feedback provided by respondents highlights that the majority had used the knowledge they had gained at the workshop to implement an online questionnaires either using BOS, SurveyMonkey, or their own existing software. Illustrated by the responses included below:

The six month follow up data illustrates that participants’ were using the knowledge they had gained for their own research or in teaching, six months after the workshop. This illustrates the deep impact and long term sustainability of the training provided by the TRI-ORM workshops.

2 Additional benefits and demand

The workshops themselves provided an excellent opportunity for networking, which is a key aspect of training sustainability. Participants were encouraged to network; the workshop offered the opportunity to make contact with other researchers using online research methods.

All the workshops were oversubscribed and approximately ten additional requests have been received by the project team following the final workshop: Illustrated by the correspondence included below:

The fact the workshops were oversubscribed and requests for workshops were being received by the project team following the end of the workshop programme illustrates that there is a clear demand for this basic level of training in online research methods.

3 Workshop Conclusion

The project team have successfully trained a core group of people in basic online research methods. In total 179 people attended the series of one day workshops, this exceeds the estimate of 120 people that was outlined in the project bid.

A range of users have been trained, the tendency is towards the Research student and Early Career researcher, which is perhaps to be expected as these groups are possibly more motivated to explore using new technology based research methodologies. The wide range of discipline areas that participants’ came from clearly highlights the interdisciplinary interest in online research methods and the demand for training in online research methods across all academic subject areas.

The aim of the workshops was to provide a basic grounding in online research methods. The pre-workshop questionnaires illustrated that participants were very interested in online research methods but their pre-workshop knowledge was basic and their experience limited. This illustrates that the marketing and publicity for the workshops was clearly reaching its target group. It also highlights the great interest in online research methods and the need for a basic level of training. The interest in this area is demonstrated by the fact that each of the workshops was oversubscribed and additional workshops have been requested.

The workshops have been delivered appropriately and efficiently. The observations of the evaluator and feedback from participants’ highlight that the workshops were very well run, with a high level of hands on relevant experience for participants. However, it should be noted that running such interactive workshop required a lot of time and effort from the project team.

The immediate feedback from participants at the workshops highlighted their intention to use their new knowledge of online research methods in their research and teaching. Evidence that this ‘intended use’ has been transferred into ‘actual use’ of online research methods was gathered in the six month follow up feedback questionnaire. The six month follow up data illustrates that participants’ were using the knowledge they had gained for their own research or in teaching, six months after the workshop. This practical use of online research methods clearly demonstrates the deep impact of the workshop and long term sustainability of the training provided by the TRI-ORM project team. The practical transfer knowledge gained at the workshops into research practices provides evidence of the lasting value the project will have for the social science community.

 

CLOSE

The second key element of TRI-ORM was the fifteen credit M level training module ‘Advanced Online Research Methods’. The aim of the module was to provide a professional development opportunity for researchers to develop their practice and undertake high quality online research. The design of the module was built on the pedagogic framework outlined in E-tivities and E-moderating (Salmon, 2000; 2002). Salmon’s five stage model, listed below:

1. Access and Motivation
2. Online Socialisation
3. Information exchange
4. Knowledge construction
5. Development

The online module ran January-April 2008 and January-April 2009. The module aimed to enable participants to:

• Evaluate when online research methods are appropriate for social science research
• Discuss the key ethical and methodological issues surrounding the use of online research methods
• Critically evaluate the use of online research methods to address their own and their peers’ research questions
• Design and pilot an online questionnaire or online interview to address their own research questions using appropriate software/hardware
• Evaluate their online research design in the light of feedback from the pilot
• Identify the key issues in successful implementation of online research methods from the perspective of researcher and respondent
• Reflect on their experience as an online researcher and identify further opportunities for development

The Advanced Online Research Methods module was advertised through the series of introductory face to face workshops run by the TRI-ORM project team. The module was also promoted through the University of Leicester and RDI websites and through direct email to potentially interested groups. People interested in participating on the module completed an online application form and were asked to provide a description of a research project they would be interested in undertaking using online research methods.

Table 7 presents details of the number of people accepted on the module, the number that withdrew and those that completed.

Table 7 Figures for module participants

The module was oversubscribed; the retention rate for both years was very good. During the first year six withdrew during the module, their reasons related to changes in their personal circumstances, listed below:

• Husbands ill health
• No longer going to use any online research methods for their study
• Finding it difficult to make the Tuesday commitment
• Substantial and unexpected changes in work responsibilities

Table 8 provides a breakdown of the roles of the participants on the module in 2008 and 2009.

Table 8: Participant roles

A high proportion of the people registered for the module especially during the second year were post graduate research students, this perhaps to be expected as this group is possibly more motivated to explore using new technology based research methodologies

The module was structured around weekly synchronous meetings. Participants completed tasks to prepare for the weekly synchronous session and follow up tasks after the session. During the second year of the module participants were also required to keep a weekly blog of their experience.

1 Module Evaluation

The evaluation was conducted over the two years that the module ran. Evaluation in the first year sought to capture formative data to provide feedback to the development team and enable changes to be made for the second iteration of the module. Feedback on the module was gained through the following methods.

Evaluator/Learner Diary (2008): The evaluator participated in the module as a learner, completed the reading and e-tivities, joined in the synchronous discussion and posted to the discussion board. This personal experience was fed back to the development team on a weekly basis.

Message board: Any problems participants experienced could be posted to the ‘I’m stuck’ message board. The information on the message board has been reviewed to highlight issues that arose.

Online Questionnaire (2008): The aim of the online questionnaire was to capture feedback from participants on; the registration process, the weekly sessions and their thoughts on whether they felt they had achieved the learning aims set out in the module description.

Online Focus Group (2008): To gain more in-depth information from participants the evaluator ran a synchronous online focus groups at the end of the first year of the module. The evaluator used the responses from the online questionnaire to inform the focus group questions and follow up key issues that emerged from the questionnaire.

Feedback from Module Co-ordinators (2008): A Formative Evaluation Report was written at the end of the first year of the module, which presented a series of recommendations. The module coordinators responded to this report and outlined the changes that would be implemented for the 2009 running of the module.

Participant Blog’s (2009): The module development introduced a reflective blog for all participants on the module during 2009. The blogs provide in depth qualitative data tracking participant’s experience of studying the module. Participants were asked to post a reflective blog at the end of the module providing feedback on their experience.

1.1 Formative Evaluation 2008

Evaluators Diary 2008

The evaluator participated on the module as a learner, and kept a Diary detailing the experience from a personal perspective. The following section describes the issues emerging from the diary entries. 

One of the key issues that emerged from the Diary was the weekly feeling of uncertainty prior to entering the synchronous meeting, would the technology work, would she be able to make a valuable contribution to the discussion. This feeling of uncertainty prior to the online meeting each week was balanced by the actual experience of participating in the synchronous session, the tutors kept the discussion focused and the other learners were encouraging and supportive.

I always enter the discussion with a feeling of trepidation and come away with a feeling that I’ve just been involved in something really interesting and that I’ve learnt a lot. (Diary, Week 6).

As part of the module the tutors gave the participants experience of working in groups of different size for the synchronous meeting to illustrate the difference of working with smaller and larger groups online. Eight participants took part in the discussion in week two and the diary illustrates that it was difficult to keep up with the flow, reading the comments of others and then having the confidence to participate quickly prior to the discussion moving on. The issue of varying group size was picked up in the online focus group. One member of the group felt that a discussion with two people was best for making a connection but a group of three or four worked for keeping the discussion flowing.

The diary entries highlight the skill of the tutors moderating the groups and ensuring that discussion moved forward and that all members of the groups had the opportunity to participant. Key questions were posted in week three prior to the synchronous meeting and diary entries illustrate that these questions helped to focus the tasks.

‘I’m Stuck’ Message board 2008

Any problem that participants experienced could be posted to the ‘I’m Stuck’ message board. The issues highlighted on the message board were dealt with straight away (where possible) by the development team.

Accessing Connect: Several participants experienced problems entering the Connect chat room for the first online meeting. This issue of was followed up in the online questionnaire, where six respondents highlighted that they experienced problems with access to connect.

Books: One participant requested that the development team included the isbns of the books that were suggested to make ordering them easier.

Links to message boards: One participant requested that the links to the message boards were put on the left hand side menu; this was done immediately to aid navigation.

Links to readings: A faulty link to a background paper was highlighted by participants and immediately corrected by the development team.

Photos: Several participants experienced problems adding their photo to the message board, instructions were immediately sent.

Evaluation questionnaire

A link to an online questionnaire was circulated to participants at the end of the final synchronous session. The aim of the questionnaire was to capture feedback on the registration process; the weekly sessions and whether participants felt they had been able to achieve the learning aims set out in the module description. It also offered participants the opportunity to feedback any problems they experienced during the course of the module. The response rate was 50% (10 out of 20).

Registration and administration arrangements

Participants were asked if they were happy with the registration process for the module, 100% of respondents were happy with the process.

In filling out the application to take part I had to think through how it might be appropriate to me personally. Thought this was a good way of beginning things (Anonymous 12, 2008).

Participants were asked if they were happy with the administrative arrangements for the module, 75%, were happy. The 25% who were unhappy with the arrangements listed the late arrival of login details. A key recommendation from the formative evaluation report at the end of the first year was that Login details be sent to participants well in advance. This was taken on by the development team and arrangements were made for the early dispatch of login details for the 2009 cohort.

Participants were asked if they were happy with the week to week running of the module 84% were very positive.

16% who were less positive about the weekly running of the module highlighted that instructions should be provided in advance. Reading lists should also be available in advance and the following week’s activities should be posted as soon as the previous week’s session had been completed. These recommendations were fed back to the development team and implemented for the 2009 module.

Participants were asked whether they had experienced any problems and if they were happy with the way the team dealt with them; 75% were happy. Advance notice of technical requirements was highlighted. This change was implemented during the registration process for the 2009 cohort.

Content

Participants were asked whether the weekly preparation/reading was too much, 84% said it was the right amount, 16% said it was a bit much.

Respondents were asked how many of the follow up task they undertook. None of the respondent failed to undertake any of the follow up tasks  8% undertook all the tasks, 68% most, 16% some and 8% very little.

Respondents were asked about the time commitment for the module. 50% said the module took more time than they expected, 50% said the module took the amount of time they expected. The issue of time was explored further in the focus group where one member of the group said time had been a real issue, trying to fit in the module with work. Another member of the group said that the module had been critical for them as using online methods would form part of their research, so although they had a full time job and were studying they had managed to find time for all the reading. Other members of the group thought the workload for the module was manageable. The group suggested providing guidelines at the start of the course about how long participants are expected to spend. This information was available on the website for the module and was clearly highlighted in the registration process for the 2009 cohort.

Participants were asked how useful they found the weekly activities. The responses were very positive during the key learning weeks however, some dissatisfaction emerged towards the end focusing on the pilot. Participants in the focus group were asked to suggest how the development team might improve this stage. One member of the group said they had thought they would have to write a research proposal but had not realised they would have to do a pilot project. Another member of the group said they felt the module petered out towards the end and that the long Easter break did not help as momentum was lost.

The participants were asked to list the three best and the three worst things about studying the module. The issues that emerge as ‘best’ included, the scope of the reading, working with people from other academic backgrounds, best practice, and peer support. The issues that are highlighted as ‘worst’ include technical problems and time to do the work.

Participants were asked what one piece of advice they would give to someone taking this module in the future.

1.2 Feedback from Module Co-ordinators 2008

A Formative Evaluation Report was submitted to the development team (November 2008) based on the Evaluator Diary, Online Questionnaire and Online Focus Group. The Report contained a series of recommendations. The recommendations and a series of questions related to the time they spent preparing and delivering the meeting were also emailed to the module coordinators. This enabled them to reflect and respond to the recommendations. Table 9 provides a list of recommendations and the actions that were taken for the second iteration of the module in 2009.

Table 9: Key recommendations formative evaluation 2008

1.3 Advanced online research methods module 2009

Participant Blog Entries 2009

The module coordinators introduced a reflective blog for all participants during 2009. The aim was to enable participants to build up a rapport with each other, draw on shared experience, reflect and discuss their experience of the module. The blog also contributed towards the second element of the assessment process, a reflective portfolio. The blogs provide in depth qualitative data tracking participants’ experience of studying the module. Participants were asked to write a blog reflecting on their experience of the module as their final post, providing qualitative data for the evaluation.

The participants on the module were expected to write an entry to their blog each week. The number of postings over the period of the ten week module ranged from 4 to 17.

• 8 participants exceeded the minimum number of blog contributions
• 7 made at least or slightly less than the minimum number of Blog contributions
• 3 made less then the minimum number of Blog contributions
• 3 made a very limited contribution

The blogs provide a week by week insight into the participants’ experience. The participants used the blog to outline their thoughts on the weekly reading and introduced new articles of interest to the group. They reflected on the synchronous meetings and highlighted the key issues that emerged from the discussions. This enabled individuals from different meeting groups to share the experience of other groups. There was very positive feedback on the benefits of the blogs.

The blogs enabled participants to introduce themselves and the aim was to build rapport in the group, the group was split of how successful building up rapport had been. One participant felt they did not gel with group until near end of module this was reflected by others; however it was also contradicted by other members of the group who felt they had built up rapport, especially during the piloting of the projects. The positive and negative reactions to building up rapport in the group are reflected in the quotes below:

The aim was to build rapport through the module blogs, through participants contributing and commenting on the contribution of others. As the module progressed the time commitment necessary to read and comment on blogs emerged as an issue. In their feedback several participants commented that they had underestimated of time needed for module and they struggled with the commitment of weekly blog. The time taken to read the blogs and then contribute possibly contributed to the difficultly of getting sustained discussion going on the blogs.

The issue of time was also raised by one participant in relation to planning and piloting their project.

Piloting projects

Participants used the blogs to compare their experience of using synchronous and asynchronous methods and to describe their pilot projects to their peers. The blogs provided a means for the group to remain in contact during the piloting of their projects and provided a forum for participants to describe their experience to their peers.

Reflections

For their final post the group were asked to reflect on their experience of participating on the module and comment on the structure of the module. The feedback was extremely positive, the aims of the module were described as good, as was the structure and organisation, illustrated by the selection of quotes below:

There were very positive comments about studying online e.g. no travel, fitting in with other work. The issue of fitting in the module around work was seen as a positive but as the module progressed comments related to pressure on participants time appeared in the module blogs.

There were very positive comments about module tutors, a few participants would have liked additional support others were happy. The resources were positively received, although several participants noted that the references needed to be kept fully up to date.

The online resources on the TRi-Orm website are superb. I really hope that website stays put, because I will be going back to that resource many times I suspect (Anonymous 19, 2009)

The online reading and viewing materials were excellent, thought some seemed to be getting dated in a fast moving field (Anonymous 12, 2009)

I’m Stuck’ Message board 2009

The technical issue raised during the first year of the course in relation to problems accessing Connect were addressed in the registration process for the second year of the module. Details were provided to ensure that the necessary ports were open. However, changes made centrally to the Connect system at the University meant that a different port was required to be opened. The module coordinators were unaware of this change which meant that some participants in the 2009 cohort also experienced problem accessing Connect for the first meeting. In response the module coordinators set up a test room in Connect and gave participants experiencing problems to direct email contact to the person responsible for he connect system at the university.

Participants experienced problems editing profiles adding photographs to their blogs. Unfortunately, this problem could not be easily resolved and the work around suggested by module coordinators was to add an image of themselves to their blog and then include it in a blog post.

Participants experienced problems login into Connect and their blogs at a mid point (week 5) in the module. It was necessary for them to reset their passwords. This problem had not occurred in the previous iteration of the module and module coordinators were not aware of the necessity to reset passwords at regular intervals. The problem of login was resolved once passwords were reset but the inconvenience and frustration was noted by participants in their feedback.

A contributory factor in the technical issue was the dispersed nature of all the participants on the module, they came from different institutions and one was login in from overseas. Although technical issues had been highlighted during the first year of the module and solutions had been put in place changes to the central Connect system meant the recommended ports were no longer relevant. Initial login problems were quickly resolved, the mid point that requirement of login passwords had to be regularly changed had not emerged during the first year of the module so was a new issue for the module coordinators. Again the problem was resolved but was frustrating for participants. The technical hurdles of running an online module should not be underestimated. 

2 Advanced Online Research Module Conclusions

The aim of the Advanced Online Research Methods module was to provide advanced training in online research methods, an aim which has been fulfilled. The ran for two years (semester two 2008 and semester two 2009) and was extremely successful. The module was oversubscribed in both years, during 2008 26 participants were accepted on to the module 21 people completed, in 2009 27 participants were accepted on to the module and 25 completed. The total of 46 completing the module exceeds the estimate of 40 people that was outlined in the project bid.

The module has been delivered appropriately and efficiently. The observations of the evaluator and feedback from participants’ highlight that the module were very well run, with a high level of hands on relevant experience for participants. The positive feedback from participants illustrated most effectively through the blog entries in 2009 reflects the skill of the tutors, designing and developing an innovative module. Also their skill in moderating the synchronous meetings and maintaining motivation within the group throughout the duration of the module.

The convenience of learning online and fitting their learning around their daily work was attractive too many participants however, during both years some participants had underestimated the amount of time the module would take (although the time commitment had been presented in the registration information). During the first year the time for planning and piloting projects was highlighted by some and during the second year the time commitment of reading, commenting and adding to their own blog was highlighted by some participants.

There were technical issue during both years the module ran, the problems with Connect resolved after the first year reappeared due to a different cause during the second year. The module coordinators were based in departments and were unaware of central changes to the connect system. The module attracted participants form a range of UK and overseas institutions, each institution has it’s own firewall and it was necessary for participants to ask their own IT services to open specific ports to enable them to access Connect. The technical issues and hurdles associated with online learning can not be underestimated.

 

CLOSE

The TRI-ORM website built on the work of the project ‘Exploring Online Methods in a Virtual Training Environment’, funded by the ESRC Research Methods Programme (RMP)(Phase 2) (Award no: RES-333-25-0001). This project produced an online training web resource addressing the use of online research methods, the website included sections on web-based questionnaires, synchronous and asynchronous interviews and online research ethics. The website underwent a thorough evaluation and peer review in 2006 and was extremely well received. . The following sections document the changes and additions that have been made to the website, linking back to the original plans for the website that were outlined in the original bid.

1 Enhancements to the website

A new home page and introductory pages to the site were added to reflect the new TRI-ORM project. The main structure and functionality of the main learning areas of the site remained the same as the functionality and ease of use of these areas had been fully evaluated during the initial design and development of the website. The key aims for the website under TRI-ORM were the development of an Online Research Methods Trainers Resource Bank and Enhancement of self-study resources. The following sections review the changes that have been made to the website to meet these two aims.

1.1 Online Research Methods Trainers Resource Bank

A Resource for Tutors area of the site has been created as a single point of reference for tutors. This section provides tutors with direct links to learning objects and content within the website. It includes materials that have been tried and tested in workshops and the Advanced Online Research Methods module which have been made available for reuse by tutors/trainers. There are activity guidelines with resources to support presentation of key topics, guidance has been incorporated on how existing self-study activities within the website can be repurposed for training. The videos of face-to-face workshop presentations for self-study use have been incorporated into the training area alongside the original PowerPoint slides. This ensures that trainers can easily access examples of how these resources have been used prior to incorporating them into their own teaching. A clear list of objectives for the trainers section of the site were listed in the original bid, these are presented in Table 1 along with how the project has met the objective.

Table 1: Features of the Trainers Resource area of the site.

A review of the initial set of enhancements for trainers/tutors set out in the original project bid illustrate that the project team have made the key changes to the website that they envisaged.

There is one key area that has not been achieved. Originally, it was intended that the training materials would developed as individual learning objects that would then be incorporated into the JISC Online Repository for Learning and Teaching Materials (JORUM). This was prevented by the lack of institutional membership of the service at the University of Leicester. 

The other key area of enhancement set out in the project bid was enhancement to the self study resources, this is reviewed in the following section.

1.2 Enhancement of self-study resources

It was originally envisaged in the project bid that all the new materials would be housed in a single new area entitled 'living archive'. However, a decision was taken by the project team to retain the original structure of the site, adding new materials related to the existing area of the site to those sections while newer methodologies would be presented through real-life examples in an Online methodological futures area of the website. The retaining the original structure of the site ensured that information was easier to locate, and the materials are presented in a more structured way.

Materials from the other strands of TRI-ORM have been incorporated into the self-study areas. This includes video lecture materials captured from the face-to-face workshops and Advanced Online Research Methods module, along with reading activities and discussions from the online module.  This material has been incorporated into the existing framework of the site's self-study modules to ensure it is embedded and contextualised by introductory materials.  This has allowed key materials to be made available to those who were unable to attend the other training strands and will ensure that this material is maintained for future reference.

The resource listing and in-text referencing within the self-study area has been extensively updated to ensure that key publications from the project period have been incorporated. This has ensured that the materials have continued to be informed by the latest developments, and has meant that the in-text references collector has been developed and expanded.

The TRI-ORM website has further developed the self-study resources. A key new area of the website is Online methodological futures. This section of the website aims to explicitly explore new and innovative online methodologies. Case studies from key researchers in the field provide contextualised introductions to a range of the latest online methods, including the use of blogging, the analysis of web traffic using website analytics tools and the automated analysis of the use of social networking sites.

2 Website Evaluation

The formative evaluation of the website ran throughout the project, the external evaluator and Principle Investigator reviewed the website and provided feedback to the Education Technologist. The summative evaluation of the website took the form of a peer review by academics experienced in the field of online research methods

2.1 Formative Evaluation

The evaluator reviewed the new areas of the site that were developed under TRI-ORM. A heuristic evaluation was undertaken by the evaluator in November 2007 to review the new look and feel of the home page and structure of the site. ideally a heuristic evaluation should be undertaken by three to five evaluators. However, this was an evaluation of an established site for which two heuristic evaluations had already been undertaken (September 2004, May 2005). Therefore, navigation, page design and content of the site had already been evaluated. This evaluation focused on the new TRI-ORM pages of the website. There were very few problems encountered in the TRI-ORM section of the website, recommendations included rechecking links, enlarging the font on the key texts pages, providing more information in the “Resources for Learners “ box on the home page. A final review of the complete website was undertaken by the evaluator in April 2009, this listed broken links and text changes, no major usability problems were found.

Feedback on the use of the website was provided through the usage reports that were written during the period of the project. They illustrate the wide use of the site as summarised in the following section.

3 Usage

A comprehensive report on usage written by Learning Technologist Rob Shaw is available from the link below. Key quotes from the usage report are included below to illustrate the extent to which the site is being used:

Usage: “Web usage statistics show that over the period 1st November 2006 to 30th April 2009, the website was accessed 62,656 times (an average of 69 visits per day).”

“A comparison of five equal 6-month periods for overall site usage reveals high levels of usage throughout the period of the TRI-ORM project, with the average number of visits daily ranging from 52 to 77.”

Quality: “In analysing these figures, however, it is vitally important to consider 'quality' of visits as well as quantity…The number of return visitors, over 10,000 users have visited the site repeatedly, and around 1000 users have clearly made it part of their researcher toolkit, returning to the resource on an average of a monthly basis or more.”

Geographic reach: Between 1st November 2006 and 5th February 2009 visits were made to the site from 179 countries/territories.  49.61% of traffic during this period came from the UK, followed by 12.92% from the US, 4.84% from India, 2.52% from Canada, 2.41% from Australia, 1.75% from Malaysia, 1.53% from Germany, 1.32% from the Philippines, 1.2% from Slovenia and 1.04% from Ireland… The overall picture is clearly one of high and increasing levels of geographical reach.”

Usage figure highlight that the TRI-ORM website has attracted users from a range of geographical locations and is well used. To provide more insight into how the site is currently being used the evaluator sought feedback from a group of academics who use and teaching online research methods, their feedback is presented in the following section.

Exploring online research methods/TRI-ORM Website usage report, May 2009 (pdf, 82KB)

4 Peer Review

To obtain further insight into how the website is and might be used in teaching a series of academic reviewers were asked to provide feedback. The reviewers were Ted Gaiser (Director Academic and Research Services Boston College); Katja Lozar Manfreda (assistant professor of statistics and social informatics, Faculty of Social Sciences at University of Ljubljana, Slovenia) and Henrietta O'Connor (Senior Lecturer at the Centre for Labour Market Studies at the University of Leicester). The academic reviewers were asked to provide a short description of their general impressions of the website, describe how they might use the website and provide details of any technical problems they experienced (no technical problems were noted by the reviewers).

The reviewers general impressions of the website was very positive their comments focused on, that the information is useful to students and trainers, references, self study resources, ability to download presentations, and the ease of use of the website. The quotes from the reviewers presented below illustrate their positive impression of the website.

Two of the reviewers highlighted that the website would be less useful for the more experienced user of online research methods. The reviewers were positive about how they use the website or could envisage using it in the future.

Additional feedback was also provided by Dr Steve Strand (Institute of Education, University of Warwick)

The academic peer review of the website was positive. However, the fact that this is a resource that will need to be updated and expanded was highlighted. In such a fast moving field the references will looked dated within a short period of time. The reviewers also picked up that the site is an introduction to online research methods and the more experienced online research methods practitioner may require more in depth information. The TRI-ORM team catered for the training needs of the more advanced online research methods through the Advanced Online Research Methods module, which was very well received. The module ran twice during the period of funding and would require further resources to run in the future.

5 Website Conclusions

The TRI-ORM website has been extended and enhanced during this period of funding, enhancements have been made to the self study sections of the website and a new Resource for tutors section has been developed. The website has enabled the three elements of TRI-ORM to be brought together. The materials from the workshops and online module are available for self study or for trainers to incorporate into their own training of online research methods. These developments meet the aims set out in the original funding bid.

The one key area that has not been achieved is that the training materials would be developed as individual learning objects and incorporated into the JISC Online Repository for Learning and Teaching Materials (JORUM). This was prevented by the lack of institutional membership of the service at the University of Leicester. The issue of the maintaining the resource in the future was highlighted by the academic reviewers. Online Research methods is a fast moving area of research and the references on the website will look dated within a short period of time. For the website to remain an attractive learning resource it will need to be updated and enhanced. The project team have established an agreement with ReStore, which would maintain existing material but would not provide scope for funding new material.

CLOSE