Open/close
all![[Skip instructions]](../../generic/trans.gif){kind=small}
![[i]](../../generic/instructions.gif){kind=small}
Click on the headings to open them. They will open on this page. Open the following link
for further
information about these headings if required.
Your browser does not support these headings. To ensure that the contents remain accessible, they have been automatically opened so that all the information on the page is displayed.
However, to take advantage of the headings and to ensure that the layout and design of this site are displayed correctly, you are recommended to upgrade to a current version of one of the following standards compliant browsers:
- Internet Explorer (http://www.microsoft.com/ windows/ie/downloads/ default.mspx)
- Mozilla Firefox (http://www.mozilla.org/ products/firefox/)
- Opera (http://www.opera.com/download/)
CLOSEGlossary links are included within this page. If a word appears as a link, clicking on this link will show the definition of the word in a 'pop-up window'. Select the following link for information about these glossary links if required.
- Select the links see the definitions in a pop-up window.
- NB. If you use pop-up window blocking software, you will need to deactivate it for pop-ups on this site to use the glossary links. Alternatively, all glossary definitions can be seen on the 'Glossary' page in the 'Resources' section.
- Use of the glossary links is JavaScript dependent. If JavaScript is disabled, it will be necessary to open the 'Glossary' page to view the definitions. Opening this page in a new window may allow you to refer more easily to the definitions while you navigate the site.
Early planning
The planning process involved new ways of writing and working for the project team. They soon became aware of the differences between writing for traditional paper-based publication and writing material for online delivery. They began with a low-tech approach, drawing out ideas for web-pages on poster-size paper. This allowed them to think in terms of space allocation and connectivity between different areas of the site, and helped them to avoid the relatively linear formats of paper-based writing.
Examples
An example of a planning sheet from an early stage in the design process
Description
Image of scanned planning page. Shows the following notes in rough form: 'Section 1. 1) Introduction (who? what? why?) | Introduction (CM) - Aims of website (why do online res / what do participants get out in conjunction onsite multiple perspectives/International) - Content - Quotes (disrupt linear flow) - postcard (feedback) | 2) Research process (HOC) - Photos / videos / autoethnography - Grant - RMF - Brainstorming - Evaluation - Consultants - Final - New ways of communicating | 3) The team (JW) Photos/quiz - Us - significant others - other activities (us) | Section 2. Resources Links - Ethnog website / book reviews - Mann/Stewart / Hine / Johns / Hewson / Chen - Recent phDs (via VM Discussion list)
An early home-page design
Description
Image of scanned planning page. Shows the following notes in rough form: 'Page 2 Home page | LOGO (written in a circle top-left) | Content - 1. Welcome - The research process The team 2. Modules 3. resources (written in a box on the left) | Home Modules resources site map help contact (written across the top) | Page 2 Exploring Online Research Methods in a Virtual Training Environment is an ESRC (webaddresslink) funded project running from May 2004 to April 2006 under the Research Methods Training Programme (webaddresslink). The project will develop the first online training package in online research methods (link to aims page 3). It aims to train undergraduate students, postgraduate students, established members of the social science community and also policy makers how to use online research methods (link to training page 4 ). (written in a box in the middle)
Pedagogic
approaches
The pedagogic approaches underpinning the training package were of those of constructivism and experiential learning. These approaches view learning as an active process of constructing rather than acquiring knowledge. Laurillard (2002) provides a useful framework for considering the design of teaching materials which explores both the pedagogical requirements and the capabilities of the technological media to meet these requirements. This framework was used as a basis for developing the training package. The process involved:
- Identifying users' needs with respect to:
- Training in different areas of ORMs (e.g. online questionnaires, online interviews, technical issues);
- The needs of a wide target audience including researchers and postgraduates in the HE sector, and researchers working for other organisations, such as those involved with public policy and market research.
- Defining the learning objectives for the different modules in order to help users identify the ways each module could support them and what they would achieve by using it.
- Deciding on the nature and design of specific learning activities
to ensure that there were opportunities available for users to:
- experience ideas and concepts presented in the training package;
- clarify understanding;
- reflect on experience;
- relate theory to practice and vice-versa;
- challenge conceptions/misconceptions;
- gain support and feedback on their learning.
- Ensuring the training materials were sufficiently flexible to meet the needs of a wide range of target users. This involved considering the ways in which different users would utilise the training package with respect to pace, location and prior experience and the resource was designed accordingly. For example, it is possible to either 'dip in and out' of the package or to work through the modules systematically. Each topic is initially presented in summary bullet point format to enable users to get an overview of the content. Users can then choose to 'drill down' to more detailed content according to their need and prior experience. All modules include learning activities but these do not have to be completed in order to move forward. For academic users, one useful feature is the 'personal references list' facility which allows users to collect a 'shopping basket' of references as they navigate the package.
Main content areas
Once the initial plan was prepared, the team met for sessions to brainstorm content ideas for the main sections of the site. Each member of the team took lead responsibility for different sections. They prepared a content outline and draft which was uploaded to the project management website for comment and redrafting. This was then used to inform the process of drafting and redrafting the materials.
Example
An early draft of the questionnaires module, showing the planned table of contents
Description
Image of draft of the questionnaires module, showing the planned table of contents. Includes the following information: Page 1 Module 2 Online questionnaires Table of Contents 1. Aims and learning outcomes (CM) 2. Introduction: appropriate use of online questionnaires (CM) 3. Advantages and disadvantages of online questionnaires (CM) 4. Different types and delivery modes of online questionnaires including email surveys, web-based surveys and (Jane to add).. (JW) 5. Sampling issues including recruitment, identity verification, response rates and incentives (CM and JW) 6. Key design issues with respect to online questionnaires, including (Jane to add). and piloting (JW). 7. Techniques of analysis for online questionnaires (JW). 8. Link to technical guide (RS) 9. Glossary (all) 10. Additional resources (all) 11. FAQs (all) 12. Print version (RS) 13. Bibliography (sources consulted in module- all) (in red)Please note, throughout this text, I am using onsite as a generic term for ftf questionnaires/telephone etc surveys. And I am using questionnaires not surveys. Any notes are written in red. It might be useful to include these on the website, if the issues they refer to have not been resolved.
Writing
and development
The original design of the website was reworked and refined with consultation with the project team, until a version was produced that was generally considered to be ready for more formal evaluation and usability testing. The idea was that feedback from this process would continue to inform the design process and lead to ongoing redrafting and improvement.
The module materials were produced on paper by the writers and included notes for development. They were then passed to the project's Learning Technologist, Rob Shaw, for conversion into workable online formats, again through a process of ongoing feedback and refinement until they were ready for more formal evaluation.
Examples
Extract from a draft of the 'Online questionnaires' module written by Clare Madge with notes to Rob Shaw in bold.
Section 3. ads and disads online questionnaires
Flexible design (add after 'affiliated institution')
Online questionnaires also enable multi-lingual formats
and pre-populating data about respondents. They can also
include prompts if the respondent skips a question and can
include audiovisual stimuli.
Non-response bias (add after 'rates are low')
Bosnjak et al (2001) have identified seven patterns in a
typology of non-response. Click here for more information.
Insert diagram and on each of 7 types of respondent click to give definition of type of non-response. Any interactivity? Use as a quiz question? Drag and drop? I've put hard copy in your pigeon hole.
Extract from a draft of the 'Ethics' module written by Clare Madge with notes to Rob Shaw in bold.
Indeed Hewson et al. (2003, 117) note that a researcher using attachments can become a 'global pariah' and it is best to refrain from using attachments all together and stick to text-based messaging. (Rob- check q'aires module that we do advise this)
Useful links (Rob should this come here or in resources section? I think perhaps here? Each needs a sentence to explain what the organization is about.)
The 'Online questionnaires' module was the first to be written and developed. By the end of this process, a module framework had been developed which could be followed in the writing and development of subsequent modules. The writers found that there were particular challenges involved in producing materials for the web as opposed to paper-based publishing. Project PI Clare Madge< recorded her initial impressions of the writing process and the difficulties faced in the production of the 'Online questionnaires' module as follows:
Issues
in writing questionnaires module
Several issues arose for me in writing this module. One aspect that was quite difficult was thinking through writing at different levels for the different presumed audiences of the website. This involved me using bullet points with hide and click for more information, so you could get a quick summary or a more in-depth account. Also embedded student activities were a way of allowing users to skip the active learning and use the site as a page turner if required.
Another issue is that of citation, plagiarism and ethics. I was not sure all the time if I should have been citing as much as I was (I basically followed a journal article citation system). And then I was not sure where the bibliography should be located. Also, in making links, I was not sure if we required permission to use these links.
Also thinking about how to write - whether to use a more chatty style in the first person or the more formal third person - was a further issue. In the end I chose the third person as it follows conventions of other sites, I suppose it may also be seen to give more credence to the text, and also strategically it might be easier to draw on this type of writing for a book if required at some later date. But these are all contentious issues and choosing the writing style/format is not a neutral choice but charged with 'political' decisions.
CM October 2004
Evaluation
and revision
An evaluation process was established to ensure ongoing feedback could be obtained and used to inform the design and development process. At each stage of the evaluation process, feedback led to changes in the design and contents of the site. Details about the evaluation process can be seen in the 'Evaluation' section, but examples of these changes can be seen below:
Evaluation
of site design
From October to May 2004, heuristic evaluations, usability studies and case studies were carried out, which led to changes in the overall design of the site. The key changes were:
- Ensuring that the main menu was made available on all pages of the site and positioned on the left rather than the right;
- Making a clearer differentiation between site instructions, activity instructions and general site text;
- Introducing arrow graphics and status bar instructions to clarify what is happening when links open onto the same page;
- Refining the instructions for the personal references list in the modules and introducing a method of storing selected references to allow them to be maintained across different pages;
- Reorganising the main navigation areas to ensure that they were grouped intuitively.
The following screen shots show some of these changes:
Screen shots
Image of the top of a module page before the evaluation
process
Image of the top of a module page after early evaluation
Image of the bottom of a module page before the evaluation
process
Image of the bottom of a module page after early evaluation
The two pairs of images show the same pages from the questionnaires module before and after evaluation. Some of the changes outlined above are apparent, notably the movement of the main menu from right to left and the introduction of styles and graphics to make different types of text and links clearer, and the reorganisation of navigation links so that links of different types are positioned in different locations (indexes and search top-right, main sections top, contacts and information bottom)
Content
evaluation
Upon completion of the modules in October 2005, a detailed evaluation of the content of the web site was carried out. A variety of academic and non-academic evaluators provided feedback on specific areas of the site and/or the site in general, following the format outlined below:
Evaluation
format
Extract from correspondence from the project's Evaluation Consultant, Julia Meek to evaluators.
We would like you to:
- Focus on the academic content; ensure that the material
provided covers all the areas you would expect to
see. Please approach the task as you would a peer
review for an academic paper and provide feedback
on the
academic content. - Provide details of any technical problems you experience e.g. broken links etc.
- Provide a short description of your general impressions of the site.
- Outline whether you have come across a resource like this before.
- Describe how you might use the site.
The feedback was returned to the Evaluation Consultant for collation and redistribution to the module writers, who commented on each of the points raised and amended the module as necessary. The commented feedback and changes were then passed to the Learning Technologist for implementation and returned to the Evaluation Consultant for analysis and the production of an evaluation report.
This led to key changes in content and organisation, some of which were as follows:
- The introductory section required revision to provide clearer pointers to the sections in the main body of the material;
- A clearer section outlining how the site and the modules can be used was included;
- Minor changes and additions were made to the academic content;
- A full site map and module index were added to allow navigation direct to module content;
- Glossary links were added and more links were included between module pages.
Further details about the content evaluation, including a copy of the report can be found in the 'Evaluation' section.
An example of the returned and commented feedback is given below:
Example
Extract from feedback from Parvati Raghuram with comments by module writer Clare Madge (shown in bold).
Perhaps you could discuss a bit more the pros and cons of incentives? Because that can skew the response rate towards people who like shopping in the shops where vouchers are coming from etc. almost always incentive based interviews are pre-arranged and have agreement before proceeding to send questionnaire. Also difficulty is that incentives don't travel i.e. can't give incentives to Debenhams and also internationalise so you lose some accessibility issues with incentives.
Please add to incentives, after this sentence
Bosnjak and Tuten (2003) also show that pre-paid incentives have no advantages in terms of willingness to participate and actual completion rates, when compared to postpaid incentives.
Add:
Finally, it must also be remembered that introducing
voucher incentives has the potential to skew the response
rate towards people who like shopping in the shops where
the vouchers may be used. This also raises issues with
respect to internationalisation, as clearly many vouchers
are place specific and cannot be used beyond national
boundaries.
How to select a sample: specify a sample population
- some groups may also be over-represented like young
people
Do not need to address as have already said elderly will be underrepresented.
Final
proofreading and checking
Final
proofreading and checking
Extract from checklist of final tasks for proofreading and checking.
- Final proofread.
Please do the proofreading on the print version and/or the individual pages rather than just on the print version. If you find an error, could you copy and paste it to a word file with the surrounding context and the error highlighted in bold, and then add the correct version. E.g.:
a. Using online questionnaires enables the researcher to correct large volumes of data quickly and at low cost. Harris (1997), for example, reports that
Using online questionnaires enables the researcher to collect large volumes of data quickly and at low cost. Harris (1997), for example, reports that
If you notice any inconsistencies in the way that terms or acronyms are used, please make a note also.
- Glossary items
- Links
On each page, could you check that all the links are working, as follows:
a. The glossary and personal reference list links open the correct item and do not cause an error message to be displayed in the bottom left of the browser.
For each page with references, could you also check that the reference list works properly (i.e. that the add all/sort/delete buttons etc work and do not cause any error messages)
The open/close all links under the menus and the open/close headings for each section work correctly and do not cause an error message. Also, please click on each open and close heading to check that the correct heading is opened and that the page jumps to the right place.
The navigation links all work (the menu links, the BACK/UP/NEXT links at the bottom, the breadcrumb links at the top, and any links in the text)
External links work and open the correct page (these should always open in a new window - and the new window icon should be displayed. Please let me know if any do not)
Internal links work and do not open a new window unless it says so in the instructions or intro.
- The site map
Please could you check that the links to the sections you are checking in the site map work and open the correct heading, jumping to the open heading.
- Images, learning activities
Please check that these display correctly and are not missing. Also if anything in learning activities causes errors or does not work properly.
- Writer acknowledgements
Please check that there is a correct reference to the page author on all the module pages.
- References and resources
Please check for consistency as per the format previously decided on.
- Titles
- Module index
Please check a random selection of links in the module index to pages from the module you are checking and let me know if there are any problems.
please make a note of any items that should be added to the glossary – again if you could copy the word with its context and paste it into the document this would be helpful.
Please let me know if any page has a missing title (in the top bar or the browser) or a title that contains an error or is inappropriate.
Face-to-face meetings
Team meeting
The management of the project has been influenced by the fact that the team is made up of researchers and educational developers from a range of different disciplines and located in a range of different places. Various influences such as maternity leave, childcare choices, writing and teaching commitments, and geographic distance have led to a situation in which face-to-face meetings have only been possible with a frequency of between once a week and once a month.
However, the regular meetings between key members of the team have provided an important focus for reviews of progress, discussion of ideas, provision of feedback and handing over of notes and materials. In between these meetings, email and blackboard have been used as an everyday means of communicating and steering the project.
Email
and Blackboard
In the early phases of content design and development, the main tool for contributing ideas and material, eliciting opinions, and sharing ideas was a space on the University of Leicester's Virtual Learning Environment, Blackboard. This has areas for the uploading and sharing of content, resources and links. It also provides an asynchronous discussions area in which feedback on materials can be entered and collated. An example of a feedback discussion is shown below:
Example
Discussion between Learning Technologist, Rob Shaw and Project investigator Tristram Hooley, October 2004.
Thanks for these comments, Tris - very useful. I've added in comments beneath yours in bold, so it's clear what they refer to.
Overall I think the site is developing really well. I think that it looks great and the content is starting to be really interesting. Most of the ideas I have here are just opinions so feel free to ignore them.
So here are some random thoughts about the general usability of the 12/10/04 version of the website:
1) Most sites use their logo as a link back to the home page. At the moment ours doesn't. I know that you've got the home link beneath it but I'd like the whole gif to be hyperlinked back to the home page on every page in the site.
Fair point - originally I took the link off as it was showing up a background colour being the logo what the user hovered over it which looked untidy - I'll set up a new Style Sheet class to stop this happening and link them all.
2) I think that the meet the team stuff should be linked to from the "Contact Us" area as well as from the "Introduction" area.
I'm yet to complete the contact us area. I agree it should be linked but I also want to add in other ways of contacting the project (using the project email address and possibly completing a web form).
3) External links. You seem to be firing us into a new browser window when we access external links. Is this a deliberate choice. I know opinion is divided on this, but I guess I'd generally avoid this approach as I think that it is confusing for users who spend time trying to work out why the back button won't work.
I made the decision to open new windows for external links and to keep all internal links in the same browser window. It is a difficult decision with advantages and disadvantages on both sides, but I think it's important to try to maintain a clear distinction between the site and external pages and to avoid 'losing people'. I planned to include info about this in the 'help' and 'about the website' sections. - it's very much open to debate, though.
4) As you know I'm not usually a huge fan of Flash, but I think that the use you've put it to on the Evaluation page is really good.
Great.
5) I like the way you are using lots of images throughout the site. Have you seen the AVS image database. This contains lots of stock images that you can use for free.
Thanks - very useful info indeed. I've found the site but it doesn't seem to be available at the moment so I'll check back later - maybe I need to be on campus - I can't get to it through the CWIS Remote Access site.
6) The menu on the right hand side is sometimes called "Quick Links" and sometimes "Main Menu" but I'm not sure I'm completely clear why.
They are all main menu now (except for the introduction page which is still called 'quick links' in error. I think you are right that main menu better reflect what these navigation links do.
7) I really don't like the Flash that you've used to deliver the "meet the team" section. I think that it is rendering in a fuzzy difficult to read way. I'd much prefer to see these as HTML pages. Also can you make it so that you can select people's photos to go to their page. At the moment you have to select the "details" button which I think a lot of people will miss.
I'll look again at this. I think I might be able to correct the 'fuzziness' and I can certainly link the photos to the details pages. I'll get back on it. In the meantime, there is an HTML page with this info which I'm using to link to when team members are mentioned elsewhere in the site.
That is probably enough for now. Perhaps we should try and do some testing with people outside the project once you've developed the next version.
Yes, thanks again Tris - v useful
This use of blackboard proved to be especially useful in establishing working patterns and holding draft materials for review. In latter stages when the draft material was uploaded directly onto the test server, these methods of communication tended to be replaced with email contact.
Email thus proved to be the main method of communication between the members of the team. This was particularly useful as a means of providing ongoing feedback between face-to-face meetings and providing a vehicle for reporting on progress and discussing content choices. Major updates to the materials on the test server would be highlighted via email and this would then be used to provide initial feedback and discuss revisions.
Examples
Email from Learning Technologist, Rob Shaw to the project team, January 2005.
Hi all,
Happy New Year - hope you all had a good holiday.
I've added the latest version of the site to the test site.
The design section took rather a lot of time to do, especially the example questionnaire which I've added validation routines for to ensure it is filled out correctly before submission - I'm planning to also use this for examples for the technical section (If you get the chance, could you try this out and let me know if you notice any problems with the ongoing feedback/instructions).
I'd particularly appreciate your comments on the whole of the design section and on the new intro and home page designs, if you have the chance before the meeting.
I've made some good progress, but I'm still working on the technical section which is not yet in a presentable form. The items for the FAQs and glossary also need to be decided on.
See you all tomorrow,
Rob
Email from Clare Madge, Principal Investigator, to Rob Shaw, Learning Technologist, July 2005.
Rob,
It looks great. Really like the way you have reorganised it.
Some minor points:
- On key journals the link to ethics and information technology doesn't link to that journal. Also the JCMC link isn't working.
- Do we need a subsection in the bibliography on the technology module (and intro module?)
- Maybe Julia could ask the consultants if there were any other terms they did not understand that needed to be in the glossary?
- Links. Need a short explanation of ESRC RM Programme (for consistency). Should we cut the journals as a more comprehensive list can be found under literature?
- Need to update events.
- Under intro and then context, the Phase Two specification paper, the commissioning report and list of phase 1 and 2 projects cannot be found.
- Under project organisation, the last few entries look quite thin. What about the list of milestones to date we gave Angela in the meeting we had with her?
I really like the way you have reorganised these two main sections. I think they have a lot more consistency with the modules now.
Hope these comments make sense! I am using my home computer so do bear that in mind!!!
Anyway, well done. It's looking good!
Clare
2004 Review meeting
On 21st October 2004, a review meeting was held between the project team and Professor Angela Dale, the director of the ESRC Research Methods Programme. The aim was to provide an overview of the project up to this point and highlight key project milestones.
Project
milestones 2004
The following document was presented at the meeting by the Principal Investigator, Clare Madge.
13 Project meetings held so far on the following dates (see attached minutes): 28.11.03, 20/1/04, 27/2/04, 12/3/04, 2/4/04, 26/5/04, 24/6/04, 7/7/04, 29/7/04, 18/8/04, 16/9/04, 30/9/04, 14/10/04
January
Attended ESRC Research Methods Training Workshop January
2004
Baseline website produced and project logo designed
February
Grant publicized to ESRC Virtual Methods Discussion List,
Leicester University Bulletin
Session organized for Research Methods Festival- speakers
invited
March
Published Research Methods Festival to colleagues
Background preparation for Educational Technologist appointment-
job spec, job advert, person spec all written, recruitment
and selection courses attended, interview panel organised
April
More background preparation for Educational Technologist
appointment - short listing questions produced, shortlisting
meeting held, job advert circulated, interviews organised,
interview questions formulated, letters sent to successful
applicants
May
Project officially started
Interviews held for Educational Technologist appointment
Provisional work plan constructed
June
Powerpoint presentation written for
Research Methods Festival session finalized including publicity,
recording, photos etc Computing matters sorted
July
Research Methods Festival session held on problems and potentials
of online research
Rob Shaw's induction days organised
Brainstorming session to decide on content and organization
of site
Attended ESRC online materials meeting
August
Set up Blackboard site for management and dissemination
and sharing of information
Evaluation consultants meeting organised
Materials written for introduction and resources modules
September
Rob Shaw starts
Tris Hooley joins team
Rob's induction
Consultant meeting Julia Meek
Brainstorming meeting for questionnaire module
Preliminary site delivered by RS
Evaluation of preliminary site by project team
October
Meeting with Angela Dale
Writing of module materials for questionnaire module and
delivery to Rob
Initial usability evaluation by Julia Meek
2005
Review meeting
On 16th November 2005, a review meeting was held between the project team and Professor Angela Dale, the director of the ESRC Research Methods Programme. The aim was to provide an update on the development of the project.
Project
milestones 2005
The following document was presented at the meeting by the Principal Investigator, Clare Madge.
Nov 2004-Nov 2005
1. Site development
Intensive phase of site development.
All 5 modules written and placed online. This involved
brainstorming content, writing the module, transferring
the written module to the online portal. Modules included:
Introduction
Online questionnaires
Online interviews
Online research ethics
Technical guide
These modules are now completed and are undergoing detailed peer review.
2. Evaluation
Evaluation has been an ongoing process through this project and is one of the project's strengths.
The site has undergone heuristic evaluation, case study evaluation and usability studies and has been altered accord to feedback from this evaluation process. It is currently undergoing content evaluation from key researchers in the field. We envisage that changes will be made as a result of this evaluation, and aim to have a final site ready for release by Feb 2006.
3. Training and dissemination
We are now moving into the final phase of the project which involves training and dissemination of the site. This will occur from Feb 2006- July 2006 (end of project).
Several training session have been planned, including through M1/M69 programme, as part of NCRM programme, and through postgraduate training exercises.
Final
reflections
Immediately prior to the site being made publicly available at the beginning of March 2006, the project team were asked to reflect on the research process and project outcomes. An amalgamated final report produced by the team is available below.
Report
1. What have you learnt from this project (wrt online methods, website production, training in a VILe, team working, that the esrc might learn from)
Clare: I have cemented my understanding of ORM and certainly increased my knowledge and expertise of internet ethics. Regarding website production and training in a VLE, I have realized that writing for the web is different and requires the development of new skills. I have enjoyed the increased interactivity it provides but also underestimated the time and energy it would involve. I have learnt a lot about the importance of evaluation and having an evaluation programme set in place from the outset of a project. Regarding team working, I have enjoyed it and very much appreciate everyone's hard work on the project. Next time round I would want to specify in more detail particular roles in the project and expected time inputs. I would also want to devolve budgeting within departments to account for this (although this will now probably occur automatically owing to FEC). I think working as a team has been facilitated by a variety of face-to-face methods and communications and this has generally worked well, suiting particular technologies to particular types of group interactions. This has ranged from face-to-face individual and group meetings, phone calls, emails, virtual discussion boards. I think we have generally worked well as a team, particularly so as we have negotiated the birth of several babies, maternity and paternity leave, childcare arrangements etc around this project.
Henrietta: Like everyone else on the team I have definitely learnt a great deal more about ORMs and issues around e-learning. The team working aspect has been great - one of the best parts of the project. The grant covered the costs of appointing an Educational Technologist, Rob Shaw, and he has been a huge asset to us. Having someone working full-time on the project has been fantastic.
I have also learnt a great deal about the importance of regular meetings and scheduling discussions. There has been very little time slippage on this project and that is an admirable achievement on the part of the team. This has been due to the discipline and structure 'imposed' (very gently and kindly!) by the project leader, Clare, who has ensured that everything has happened on time.
Jane: Like Clare, I feel that through working on this project I have consolidated my knowledge of ORMs. I was on maternity leave for 10 months of the project during which time the website and training resource developed considerably and as a result I missed out on some of the key decisions regarding some design and content issues. On my return I found some aspects of the site to have evolved in different ways to those that I'd envisaged. Having a good record of the processes underlying the site's development including the rigorous evaluation procedure has been really useful in helping me to understand and catch up with the site's development and I'd recommend documenting the process to any teams embarking on a similar project. For me a key issue that emerges from this (and many other projects developing online resources) is the need for there to be a continuation strategy in place to ensure that the resources maintain currency and the site doesn't become moribund. At present all the responsibility for doing this, particularly finding continuation funding is placed on the project team. I think the ESRC and other funding bodies could look more strategically at how they support the longevity of web-based projects such as this one, perhaps by having a central facility that can take on some of the responsibility for such sites in the future.
Tristram: I guess that I've learnt the importance of producing on-line resources to the highest quality possible and aiming for longevity. I think that we've taken the long term preservation of the ESRC's investment very seriously right from the start. I've worked on a lot of on-line resource development projects over the last few years and relatively few of them are still in regular use. Partially I think that the lack of staying power of many on-line learning resources has come down to a lack of ambition of behalf of those who are creating them. With this project however we set out to create a key text in the field, something that would be regularly used and cited and would have a life expectancy that was at least as long as a text books. I think that we can have some confidence that people will still be using the site in five years time. I also think that we can hope that if the ESRC, or anyone else, ever funds anything else dealing with ORM the site that we've built will be the foundation for it.
Rob: I have learned a huge amount from being involved in this project. Being responsible for the whole of the technical side of the process of developing the site and ensuring the materials meet standards of design and accessibility has given me an insight into some of the technical aspects of web development that I had not had to deal with previously as a teacher, materials writer and developer. Being involved in usability testing and evaluation, as well as dealing with team feedback and discussion on the design has also taught me a lot about the importance of taking on board what the team and potential users have said, maintaining an objective view of what's working and what isn't, responding to new ideas and experimenting with a range of solutions to find the best one.
Producing the site and its learning activities has allowed me to develop my technical skills in technologies such as XHTML, CSS, JavaScript, Flash and ASP.NET a great deal. This has happened in a highly effective way through finding solutions to genuine problems as and when needed as I've gone along, and I've appreciated the time to focus on this. Working with the team on the production of the modules has also taught me a lot about ORMs, and helped me to further work on my multimedia development skills and to learn more about the design process and liaising with authors. Researching and writing the technical guide sections has also been very developmental and rewarding in terms of improving my own knowledge, of developing my writing skills – especially writing for the web and for a non-specialist audience, and of producing something which was not really available before and which will (hopefully) make the technical aspects of ORMs more accessible to people who might otherwise be put off.
It has been a genuine pleasure to work with the team, and I've been impressed by the effort and energy that they have put into the writing and reviewing the modules, despite there being so many other things to keep them busy. As I've spent much of the time working on the project from home, I've also had to learn to learn to adjust to the different kinds of pressures this can bring. Overall, I've been vastly impressed by how successful team working and project management can be from a distance, and I'm sure a lot of that comes down to the fact that the project team (and the wider support network in Leicester) are made up of such a good bunch of people.
2. What advise would you give someone embarking on a largest ESRC collaborative (Training?) project like this?
Clare: Specify roles and time inputs expected from the outset; Decide on budget organization at grant allocation stage; Keep detailed sets of minutes and decide who will keep these and circulate; Use a variety of communication methods for group working; Evaluate from the outset.
Henrietta: Mainly to keep timing issues at the forefront of everyone's mind. This was probably the single largest issue and as ever the time went far too quickly and it was difficult to 'keep up'. Having regular meetings and email communication and not letting deadlines pass would be my main advice.
Jane: A project like this is just the same as any other research project – you need to set aside dedicated time for research and project development (including team meetings). An added dimension is ensuring that there is good dissemination of the resources produced. This can't be left to the end – networks need to be established at an early stage. As an overall comment I'd say simply 'Don't underestimate the amount of time involved!'
Tristram: Make sure that you have built in a long period for development and evaluation of the site. Because we had a relatively long development period we were able to really respond to the feedback that we received and to improve the site. By evaluating throughout we have seen the original conception of the site develop and we can be really confident that the final product will be academically excellent and also able to meet people's training needs. I think that a real strength of the team that has been working on this has been our diversity. Hen and Clare obviously provided the academic expertise in ORM. However having Jane and me on board to shout about the pedagogy/training side of the site has helped it to develop in the way that it has. I've also really enjoyed working with Rob and I hope that my (rather limited) technical background has given him enough of a sounding board to bounce some of his technical and design ideas off of.
Rob: From my perspective, evaluate early and often, take on board as many opinions as possible and keep an open mind - not always easy if you get too close to the materials you're producing. Work to get the details right, but stay focused on the big picture and the overall purpose of what you're doing, and work to a dissemination plan as soon as possible. Try to ensure you work with a good team and that the PI is as supportive, dedicated and proactive as Clare.
3. What has been the best thing about the project? (best practice)
Clare: I suppose overall I am very proud of the whole site! I think we have achieved our aims and have produce a good quality site for training in online research methods. I think there is a lot of good intellectual quality in the site and certainly this is supported by the external reviews. I am particularly proud of the ethics module as this involved a lot of reading and thinking and I like the transparency of the research process in the background section as this is so often left out of websites and their production taken as given. I have also enjoyed working closely with Rob and will miss him when he goes!
Henrietta: The best thing has been the team. The team have worked really well together and knowing that we had a full-time member of staff was reassuring throughout – things were always happening in the background when other team members were tied up with teaching commitments and so on.
Jane: Working with a great project team to produce a high quality and useful resource.
Tristram: I think that our approach to on-going evaluation of the site has been a really good aspect of this project. So much of the time evaluation is talked about a lot and then left out as soon as time gets tight. By bringing Julia in as an evaluation consultant we really prioritised this side of things.
Rob: Seeing the package come together in the latter sages and seeing the positive reactions of people towards it. Being a part of making it a useful resource which will (hopefully) serve a real need.
4. What has been the worse thing (what would you do differently)
Clare: The time input required. During this project I have most been on a 60% contract (therefore not paid very much) and I have had to negotiate doing this project at the same time as a full teaching load and big administrative task. At the end of the day, I am pleased with the site we have produced but it has taken an enormous effort.
Henrietta: The worst thing has been the juggling of time. To continue with this metaphor, it was very difficult to keep all the balls in the air when working on the project and trying to deal with all the other usual demands on time.
Jane: Well it's essential so I wouldn't do it differently, but I found all the final checks of links, spelling, etc, very tedious. Basically I'm not a good proof reader and so I guess getting someone else to do this would be my top tip here.
Tristram: I think that the obvious thing to say is that we were all too tight on time. I've got rather too used to 'snoozing' Outlook pop-ups that said 'ORM'. However, the site is a real achievement which a huge amount of work has gone into. So while putting it together has probably stressed us all out, I think that we are all pretty happy with the result.
Rob: Wait until the module contents are 100% right before producing print versions!
5. Which bit of the site/project are you most proud of and why?
Clare: see above
Henrietta: Like everyone else this has to be the website itself. It will be great to have the site launched and to be able to use it as a day to day research and teaching aid.
Jane: All of it!
Tristram: I think that I'm particularly fond of the learning activities that Rob has built throughout the site. As we start to disseminate the site hopefully we'll get the chance to talk more about how you can use it as part of a course that you might teach through either by e-learning or in the classroom. I think that the case studies and the learning activities are a goldmine for a research methods teacher and I really hope that these get well used.
Rob: All of it – but seeing the Technical guide as a finished resource was a particularly good moment.
6. Which part of the site/project are you most dissatisfied with and why?
Clare: I would have liked to have expanded the site to cover online experiments and virtual ethnographies but there was not the time/finance to do so nor the expertise. I would like to have had guaranteed continuance funding so that we can update the site, make it more interactive, and so that we had not lost Rob's skills and expertise in the project.
Henrietta: One of the most difficult issues is that of technology. It sometimes seemed impossible to keep up with advances in technology. For example, in the online interviewing module the emphasis is on text-based interviews. With the advent of easily accessible software such as 'Skype' it is now possible to hold no cost, realtime conversations via computers. This has huge implications for online interviewing but is something we have not yet had time to explore or include on the site.
Jane: Early in the project we had hoped to develop an interactive tool to enable users to develop their own online questionnaires within the site. We developed a good proposal that was well received by the reviewers but wasn't funded. Rob has done an excellent job in providing a technical guide for those wanting to develop online questionnaires, but it would have been great to have incorporated this service as a one-stop-shop within this site.
Tristram: I think that it is all of those things that we struggled to do justice to within the site. The ones that I was most involved with were those in the technical module where we were always trying to work out what we should cover and what we should leave out. I think that we also struggled with the problems of talking about technology at all when everything is changing so fast. However, we have hopefully come up with a technical module that will be really useful. Indeed without this section the usefulness of the whole resource is dependent on IT departments co-operating with the needs of researchers. If we've got it right the technical guide will put some power into the hands of researchers and gives them technical options and a vocabulary to discuss the technical issues on which this kind of research depends.
Rob: I suppose the fact that it always had to be a completely self-supporting resource has limited the opportunities for interactivity and the building up of a community of users of the site, but I think this will still happen to some extent, and certainly if further funding is secured to build on it for the future.
7. What problems do you envisage in the future for the site and training in online methods?
Clare: Updating it. Responding to enquires. Finding time to do training.
Henrietta: This is really 'see above'. As technology continues to advance there are more and more possibilities for ORMs and we are probably just scratching the surface at present.
Jane: Finding continuation funding to ensure it remains current and useful to the social science community.
Tristram: I think that keeping resources up to date is a challenge – but keeping them in the public eye is an even bigger one. We all have to start putting high quality websites on course reading lists and referring students and each other to them rather than just hoping that Google will deliver us to everything that is relevant. We also have to work out a way of rating this kind of site as research output that ranks alongside articles and monographs to ensure that more resources of this quality are created.
Rob: I think it will be a very useful and well-used resource as it stands as there's not really anything like it at the moment and there seems to be a real demand for the package. However, it would be good to know it's going to remain a living and current resource in the long term - hopefully the funding and the time and energy can be found to make sure this happens.
8. Anything else?
Clare: No, I think I have probably said quite enough already!
Henrietta: It's been a great experience and I've learnt a lot. I'm looking forward to getting the site 'live' and knowing that people are benefiting from our hard work.
Jane: Well we're talking about another bid to further develop resources in ORMs at the moment so it can't have been too bad...
Tristram: I've really enjoyed the experience of working on this project. Thanks to Clare, Hen and Jane for inviting me along for the ride. I hope that people find it useful and let us know if they use it in teaching or if it is the resource that enables them to get going with ORM.
Rob: Nothing, apart from a big thank-you to the team for an enjoyable and productive 18-months, and for the valuable experience I'll be taking on to my new role.