The evaluation process

A programme of evaluation was established to ensure ongoing feedback could be obtained and used to inform the design and development process. The evaluation process was based on the 'Evaluation Lifecycle Toolkit' created by Julia Meek of Birmingham University. This is an integrated package designed to place evaluation at the heart of the development of learning technology, building evaluation activities into every stage of the process. The programme aimed to ensure the training package would meet the needs of different user communities as effectively as possible.

The timeline of evaluation activities is shown below.

 

Further details about the evaluation activities carried out at different points during the project can be seen below.

 

CLOSE

Heuristic evaluation

A heuristic evaluation was carried out by Julia Meek (University of Birmingham) with a team of three evaluators.

This led to the following changes in the navigation and design of the site:

1. Ensuring that the side menu bar was available as a navigation option on every page and that it was moved to the normal position for such a bar on the left-hand side of the page;
2. The number of links in the project background section was reduced to improve clarity and usability;

The profiles of the evaluators are as follows:

Dr Stamatina Anastopoulou

Research Fellow, University of the Aegean, Dept. of Products and Systems
Design, Ermoupolis, Syros, 84 100 Greece.

Dr Anastopoulou is interested in studying learners' interactions with technology, especially in those interactions that involve multiple sensory and communicative modalities. During her PhD research in University of Birmingham, she explored the effect of multimodal interactions in science learning and developed a theoretical framework for the design of educational technologies. Now in the University of the Aegean, she explores the applicability of this framework in different cultural and educational settings.

Dr Liz Masterman

e-Learning Researcher

PhD in Educational Technology, University of Birmingham, 2004

Currently working as evaluation officer on contract with the Learning Technologies Group, Oxford University Computing Services.

Mike Sharples

Professor of Educational Technology at the University of Birmingham and Director of the University's Centre for Educational Technology and Distance Learning (CETADL).

His current research interests include human-centred design of learning technology, the theory and development of mobile learning, and application of models of cognition and social interaction to the design of technology for learning and knowledge working. He is the author of 7 books and over 150 other publications in the areas of interactive systems design, artificial intelligence and human-computer interaction.

The report is available from the following link (Requires Adobe Acrobat, link to Adobe Reader)

Heuristic evaluation report (pdf, 175 KB)

 

CLOSE 

 

CLOSE

 

CLOSE

Content evaluation

Upon completion of the modules in October, a detailed evaluation of the content of the web site was carried out. Chris Mann (Oxford Internet Institute) and Parvati Raghuram (Open University) were asked to carry out a detailed evaluation of the modules. A variety of academic and non-academic evaluators also provided feedback on specific areas of the site and/or the site in general. The following table shows the key evaluators along with their institutional affiliation and the scope of their evaluation.

Evaluator	Institution	Scope of evaluation
Chris Mann	Oxford Internet Institute	Introduction; Interviews; Ethics.
Parvati Raghuram, Lecturer in Geography	Open University	Introduction; Questionnaire design.
Christine Hine, Senior Lecturer specialing in Virtual Ethnography	University of Surrey	Ethics.
Chris Taylor, Project Manager for the ESRC-funded Teaching and Learning Research Programme (TLRP) Research Capacity Building Network (RCBN), Cardiff University	Cardiff University	Questionnaires.
Martyn Denscombe, Professor of Social Research	De Montfort University	Site in general.
Claire Hewson, Lecturer in Cognitive Psychology	University of Bolton	Site in general.
Christine Gratton, E-Learning Co-ordinator	University of Nottingham	Technical guide.
Jane Hanford, Market researcher	Nokia	Site in general.
Rob Negrine, Doctor	NHS	Site in general.
Colin Hyde, Researcher and Outreach Officer	East Midlands Oral History Archive	Site in general.

The evaluations were organised and structured by the project's evaluation consultant Julia Meek (University of Birmingham). The outcomes were fed back to the design team, and some of the key changes made as a result were as follows:

• The introductory section required revision to provide clearer pointers to the sections in the main body of the material;
• A clearer section outlining how the site and the modules can be used was included;
• Minor changes and additions were made to the academic content;
• A full site map and module index were added to allow navigation direct to module content;
• Glossary links were added and more links were included between module pages.

As an example of the feedback forms returned by the evaluators, feedback from Christine Hine (University of Surrey) is available to view from the following link (Requires Adobe Acrobat, link to Adobe Reader). The feedback from all evaluators can be viewed in the evaluation report in the section below.

Feedback from Christine Hine (November 2005) (pdf, 82 KB)

 

CLOSE

The training and dissemination activities carried out by the team are to be evaluated and details of this evaluation will be posted here.

 

CLOSE

The evaluation consultant, Julia Meek, produced a final evaluation report of the evaluation process in March 2006. Her summary is outlined below and the full report can be viewed by following the links below.

The 'Exploring online research methods in a virtual training environment' project, funded by ESRC Research Methods Programme (Phase 2, award no: RES-333-25-0001) has successfully met the aims outlined at the outset. This report concludes that:

1. The website provides an excellent resource for training, meeting key training guidelines specified by ESRC and the Joint Skills Statement. The website can be used for both face-to-face training and online dissemination and is a useful resource for researchers at all stages in their career;
2. The website includes modules on online questionnaires, interviews, ethics and a technical guide;
3. There is extensive reference to papers, websites, resources which the learner can collect and export to other bibliographic software e.g. Endnote;
4. This is a totally new resource; there is no other training of this kind widely available. All evaluators commented that they had not come across a similar resource;
5. The website is professional presented, well designed and easy to navigate. The following a quote from Dr Chris Mann illustrates how well the website has been received by academic reviewers:

   "This is self-study online training of the very best kind: practical and hands-on; theoretically sound; technically exacting; supportive and inspirational. The opportunities and challenges of online research are presented by an interdisciplinary training team with comprehensive expertise in the methods discussed".
   Dr Chris Mann (Oxford Internet Institute) November 2005

The full report is available to view from the following link (Requires Adobe Acrobat, link to Adobe Reader).

Evaluation report (pdf, 439 KB)

Appendix A: Evaluation timeline (pdf, 70 KB)

Appendix B: Heuristic evaluation report: November 2004 (pdf, 172 KB)

Appendix C: Feedback form from Group trial (pdf, 116 KB)

Appendix D: Heuristic evaluation report: April 2005 (pdf, 34 KB)

Appendix E: Full feedback from external evaluators (pdf, 213 KB)

 

CLOSE

The following report on usage of the site in the first five months following its release was written by Learning Technologist Rob Shaw in October 2006.

1. Introduction

This report aims to provide information on the usage of the 'Exploring Online Research Methods’ website from the period 3 April to 31 August 2006 (the first five months following the website’s completion and release). Firstly, it will provide an overview of the site's traffic as recorded in the server log files. This will include indications of total traffic as well as analysis of the nature and origin of this traffic. Secondly, the report will provide an indication of the reach and effectiveness of the site by outlining how far it has been included in key resource listings and search engines and by providing a summary of testimonials and recommendations from users. Finally, a short evaluation of the use of the site is given.

2. Site traffic

Accurate figures of website usage are notoriously difficult to gather and prone to inaccuracy. Analog 6.0 was the software chosen for the analysis of the server logfiles as it offered a high degree of accuracy in its logfile reporting and provided a cost-effective long-term solution. The method of counting the number of requests for pages was used to gather estimates of total usage as this ensured that the figures were not artificially inflated through the inclusion of requests for graphics and other resources packaged within each single page. Alongside the measurement of 'distinct hosts' or the number of individual computers which have accessed the site, these figures can be obtained with some certainty to provide an indication of minimum usage.

As the following graph (Figure 1) shows, a total of 52,755 pages were requested during the 5 month period making an average daily request for 352 pages.

The graph shows that there has been a consistent growth in page requests from about 10.000 after one month, to over 50,000 after 5 months. The rate of growth has shown a slight reduction in monthly growth rate over the second half of the period, but this is likely to have been caused by the nature of the annual academic cycle. If this proves to be the case, the figures for the next two to three months which will cover the beginning of a new academic year are likely to show an upturn.

When considering these figures, it should be emphasised that they are completely accurate only as an indication of minimum usage levels. There is likely to be a degree of underestimation in the fact that following a first visit to a site, the information may be 'cached' or saved to a user's hard drive for access at a later date. In this case, future visits would not be included in the calculation. In some cases, users of the same computer, network or Internet Service Provider may also share a cache so that only the first request from a user for a particular page would be counted.

The requests shown originated from a total of 10,167 distinct hosts. This can be considered an accurate measurement of the number of different machines that have downloaded pages from the site, but this may again be an underestimation given that multiple users may share a single machine.

Traffic to the site originated from a total of 94 different countries worldwide. The geographical breakdown is shown in the following chart (Figure 2). Although about half of the users so far have originated from the UK (25%) or commercial companies (including Internet Service providers) (22%), significant numbers of users have also originated from Europe (6%), Australia (2%), Asia (2%), US (2%) and south America (2%). Smaller numbers have originated from Canada, New Zealand, Africa and the Middle East. The site appears to be gaining widespread international usage.

The referring site information shows the top twenty websites from which a link to the site was followed. This can be seen in the following table (Figure 3).

The majority who reached the site via another website did so through search engines, and from Google in particular (1, 3, 7, 8, 19, 13, 14, 16, 17). Portal and listings websites made up a substantial number of the others (5, 6, 9, 15, 20) along with academic institution and organisation websites (11, 12, 18) and blogs (3, 19). This information reflects the range of websites which added links to the site and also the importance of ensuring that the site is accessible through simple searches such as 'online research methods' in popular search engines as well as through links in specialist websites.

So despite the caveats about using site traffic to estimate website use, it is clear that the site is well-used, use is growing, the site has an international audience and is being researched via a variety of search engines, academic institutions and through personal recommendation. In the next section an assessment of the website's usefulness will be attempted.

3. Reach and effectiveness

3.1 Links

One way of assessing usefulness is to explore the organisational and individual websites that included links to the 'Exploring Online Research Methods' site. These often included descriptions and/or recommendations for the site. Such portal sites and listings featured highly in the list of referring sites. Some of these links were solicited and some were initiated by the referring site through contact with the authors. Many were complimentary about the site. Examples include:

Institutions and organisations:

Association of Internet Researchers (AoIR): The site was highlighted on the front page as being a useful resource for members. The President of AOIR, Dr. Michael Allen, contacted the authors to confirm this with the following comments: '…this is, from first impressions, an excellent site which does a lot of things we at Curtin have attempted to do (poorly!) in a unit of study on similar issues. We will definitely be using it in our degree programs. I will also bring this to the attention of the whole AoIR mailing list (1000+ subscribers) and will add it to the website.'

Other organisations and institutions that included links with recommendations and descriptions include WebSM, QUALITI, University of York, University of Nottingham, University College London, Barnsley NHS Trust, Association of American Geographers.

Portals:

Sosig (now 'intute: Social Sciences') made the site an 'Editor's choice' for Research Methods tools. Since the 'Exploring Online Research Methods' site's inclusion, it has featured consistently in the 15 most popular research methods websites listed and in the 30 most popular of all the sites on the listing.

Higher Education Academy: Sections of the Subject Network resource sites such as Business, Management, Accountancy and Finance (BMAF) and Social Policy and Social Work (SWAP) gave the site a prominent listing and description.

Resources for Methods in Evaluation and Social Research: The site was added as a featured link.

Blogs and personal bookmark listings:

The site was included in several blogs and personal bookmark listing web pages as is reflected in the inclusion of these websites in Figure 3. An example of inclusion in a personal blog is the following which was included in a blog called Marcus's Musings – Thinking and writing about FLOSS and digital sustainability: 'Yael Levanon, who attended last year's Summer Doctoral Programme of the Oxford Internet Institute with me in Beijing, passed on an interesting virtual training environment hosted by the University of Leicester called Exploring Online Research Methods. It features several modules, e.g. online questionnaires, ethics, and offers a well sorted area of helpful resources on the web dealing with the question of how to run online surveys. Thanks Yael, thanks folks at Uni of Leicester!'

3.2 Comments and recommendations

A wide range of comments and recommendations have been received from users, teachers and experts in the field of Online Research Methods. These have emphasised the value of the website as a free openly-available resource both for individual use and for teaching purposes.

Some examples are given below:

 

 

 

4. Conclusions

Quantitative site traffic statistics, in addition to qualitative responses and recommendations, suggest the site has robust and consistent levels of usage. Many users of the site appear to be finding it an accessible, useful and original resource.

 

CLOSE