Frequently-asked questions

Yes, many people are now using online questionnaires. There are a number of general social science and psychology-related sites which advertise online questionnaires and experiments:

• Lab-United - International Online-Research
  http://www.w-lab.de/lab-united/actual.php;
• The Social Psychology Web-lab.
  http://socpsy.psy.ed.ac.uk. (Contains links to
  listings of online studies)
• Online Psychology Research UK
  http://www.onlinepsychresearch.co.uk/. (Specifically advertises for participants from the UK)

Some key online questionnaire and research methods sources can be found at:

• Web Survey Methodology, a portal funded by the EU Fifth Framework program
  http://www.websm.org/;
• The Web Centre for Social Research Methods at Cornell University
  http://www.socialresearchmethods.net/;
• School of Library, Archival and Information Studies at the University of British Colombia
  http://www.slais.ubc.ca/resources/ research_methods/general3.htm;

There are, of course, many more.

CLOSE

Given the increase in the number of internet users, many writers are increasingly optimistic about the ability of internet studies to reach a representative sample of the population for quantitative research. However, where such a sample is required, the researcher is still likely to face major problems. Attempts to create an accurate sampling frame are hampered by the fact that email listings for broad cross-sections of the population are difficult to come by. Use of the internet remains unevenly distributed both socially and geographically. People from poorer countries and from particular groups within richer countries (such as those from lower social classes or the elderly) are less likely to have access. (Denscombe 2003)

Generally, online questionnaires are best used in situations where a particular group is targeted through non-probability sampling or self-selection (see 'Sampling' section of the 'Online questionnaires' module).

CLOSE

Research suggests that online and postal questionnaires return roughly similar results in terms of response rates. Though research is limited, the studies that have been undertaken also suggest that the quality of response is roughly equivalent. (Denscombe 2003)

CLOSE

The following have been reported to increase response rates:

• Send introductory letter outlining project and estimated time needed to complete the questionnaire;
• Include an institutionally sanctioned website to validate researchers' identity;
• Provide clear instructions on how to complete the questionnaire;
• Request personal information at the start of the questionnaire rather than the end;
• Use simple questionnaire format and avoid unnecessary graphics;
• Avoid grid questions, open-ended questions and requests for email addresses;
• Design survey so it takes approximately 10 minutes to complete;
• Do not include more than 15 questions;
• Send one or two follow up reminders;
• Include 'social presence' or missing data messages to reduce item non-response;
• Emphasise confidentiality.

CLOSE

Research suggests that if people are going to complete a web-survey they will do so in the first few hours or days of receiving it. However, it has also been found that increasing response rates can be achieved by follow-up reminders, a single reminder can double the number of respondents while it is suggested that four repeated contacts yields the highest response rate. The type of internet connection and hardware and software used in accessing the internet will also impact on how quickly the questionnaire is completed. But generally speaking your response rates will tail off after the first few days and you are unlikely to get many responses from your initial request after four weeks but of course, this varies with the target population and type of research project.

CLOSE

Design issues are an extremely important consideration for online questionnaires because of the highly visual nature of the web environment, and the variations in technical skills of survey respondents. The massive range of purposes of questionnaires and diversity of the populations to be studied mean that there is no single design approach that is appropriate for all online questionnaires. Some general good practice guidelines are noted below.

• Ensure consistency of appearance with different browsers and computers;
• Include a welcome screen;
• Provide clear and precise instruction on how to complete and submit the questionnaire;
• Keep the questionnaire as short as possible - 10 minutes maximum;
• Include a progress indicator to reduce drop-outs;
• Use colour sparingly - yellow and blue most effective;
• Use 12 or 14 point Times New Roman or Arial font;
• Use a variety of question types appropriate to the research question and type of respondent;
• Only use multi-media stimuli when essential to the research project;
• Pilot and do usability testing.

See the 'Design of online questionnaires 1: Appearance' and the 'Design of online questionnaires 2: Content' sections of the 'Online questionnaires' module.

CLOSE

Forms fields can be added to web pages by adding appropriate HTML '<input>' tags or inserting them using WYSIWYG (What You See Is What You Get) editors such as Macromedia Dreamweaver or Microsoft FrontPage. A full guide to creating forms in web pages is available in the 'Technical guide' module, along with links to further information in the resources section.

It is also easy to create questionnaires within Word documents that can be sent as attachments. Participants will, however, need to have a suitable version of the software installed on their machine and it should be remembered that the use of attachments may seriously reduce response rates. Attachments may expose respondents to more computer viruses and the online researcher must ensure that they never forward any viruses with their emails or attachments. Indeed Hewson et al. (2003, 117) note that a researcher using attachments can become a 'global pariah' and it is best to refrain from using attachments altogether and rely on text-based messaging.

However, if a researcher chooses to proceed with this method, it is possible to create word documents ready-made for questionnaires. By inserting form elements from the 'forms' toolbar into a template and then protecting it, the word document is locked so that users are only able to add information to the appropriate form fields and cannot change any other sections.

A step-by-step guide to doing this is available on Microsoft's TechNet site:

http://office.microsoft.com/en-us/word/HA100307461033.aspx

and further information can be found via the University of Essex's Computing Service pages:

http://www2.essex.ac.uk/cs/documentation/use/word6/howto/forms.html

Other resources are also easy to obtain through a simple Google search.

CLOSE

Questionnaires are generally based on a quantitative methodology but that does not preclude the use of more open discursive type questions. However, you will need to think carefully about how you will analyse such responses so you do not fall into the trap of simplistic description. The following site from the School of Library, Archival and Information Studies at the University of British Colombia is a useful starting point for thinking about qualitative analysis.
http://www.slais.ubc.ca/resources/research_methods/qualitat.htm

CLOSE

In theory, no. Indeed, it is likely that in the future we will see an increase in the use of 'mixed method triangulation' with onsite and online methods both used to interrogate and verify the intersections between real and virtual infrastructures, enabling research to take place across a variety of online/offline domains. However, practical issues may emerge relating to compatibility of sampling in the two domains, transference of online and onsite questionnaire design and data format comparability. This remains a very interesting area of methodological enquiry which requires further investigation.

CLOSE

As with any research, there are a wide range of ethical issues that should be considered when carrying out research via online questionnaires. Examples include issues of privacy, consent and confidentiality. It is also essential to maximise data security in collection and storage. See the 'Ethics' module of this training package for further information.

CLOSE

It is possible to carry out text-based interviews which are either facilitated 'asynchronously' through email or bulletin boards or 'synchronously' using a chat room or chat facility. You do not need to provide any video link up with the interviewees. However, the increased availability and lowering cost of webcams may provide a new dimension to online interviewing.

See the 'Types of online interview' section of the 'Online interviews' module for more information.

CLOSE

It does not matter whether you choose to carry out interviews as synchronous or asynchronous encounters. There are advantages and disadvantages to both approaches. There are advantages and disadvantages to both approaches, for example, some researchers have found that interviewees do not feel comfortable in a chat room environment and have decided to use email interviews instead (Illingworth, 2001). The decision should be based on your research aims and your relationship with the participants.

CLOSE

A mixed approach to researching online communities has been adopted by a number of researchers for different reasons. For example, Orgad (2005) recruited respondents via health related websites and developed rapport in email based exchanges. She then went on to interview a number of her respondents face-to-face and provides some comparison of the different approaches. Similarly, both Sanders (2005) and Rutter and Smith (2005) successfully combined online and offline approaches to data collection.

CLOSE

One of the advantages of using an online as opposed to an 'onsite' approach is that some individuals who may find it difficult to express themselves verbally prefer to 'talk' online through a text-based discussion. For example, many teenagers are now used to expressing themselves online rather than face-to-face. However, there are issues associated with speed of typing with and some respondents may have difficulty 'keeping up' in a synchronous interview.

See 'Advantages and disadvantages' section of the 'Online interviews' module for more information.

CLOSE

Asynchronous interviews can be carried out via email or through discussion board software. This will largely depend on the number of people interacting simultaneously.

For synchronous interviews, the following types of software provide the means to interact through text:

• Free online chat providers
• Free software and messaging services
• Online meeting software
• Online customer support software

Audio and video can also be used where interviewer and participants have adequate access to technologies such as high-speed internet connections and webcams.

Please see the 'Technical guide' section of the 'Online interviews' module for more information on this.

CLOSE

There is much debate over this issue. Some argue that because online/offline worlds are mutually constituted, and we carry our real-world assumptions, norms and behaviours into cyberspace, then we can clearly draw on onsite ethical guidelines while others suggest that there is something special about the online research environment that necessitates the development of a set of ethical guidelines specifically pertaining to the virtual venue. According to the Association of Internet Researchers (AoIR) ethics working committee (quoted by Ess 2002a, 180), online research can entail greater risk to individual privacy and confidentiality, greater challenges to a researcher in gaining informed consent and more difficulty in ascertaining participants' identities. This results in increased difficulty in ascertaining ethically correct approaches because of the greater diversity of research venues and because of the global reach of the media involved.

CLOSE

There are many but the following are a useful starting point:

Ess, C. and AoIR Ethics Working Committee (2002) Ethical decision-making and internet research: recommendations form the AoIR ethics working committee.
( http://www.aoir.org/reports/ethics.pdf).

Bruckman, A. (2002a) Ethical Guidelines for Research online
( http://www.cc.gatech.edu/~asb/ethics/).

American Association for the Advancement of Science Ethical and Legal Aspects of Human Subjects Research in Cyberspace
( http://www.aaas.org/spp/sfrl/projects/intres/main.htm)

John Suler (2000) Ethics in cyberspace research
( http://www.rider.edu/~suler/psycyber/ethics.html)

See also the 'Further resources' section for other examples.

CLOSE

There is some debate over this issue but generally speaking for private or semi-private sources (mail, closed chat rooms) informed consent is considered essential, whereas in open access forum (newsgroups/bulletin boards), it is suggested that informed consent may not be so essential. Ess and the AoIR Ethics Working Committee (2002, 5) recommend that the greater the acknowledged publicity of the venue, the less obligation there may be to protect individual privacy, confidentiality and the right to informed consent.

CLOSE

Data security can be improved by the use of web-based questionnaires rather than email questionnaires, or the respondent can be encouraged to complete the questionnaire on an anonymous machine in a library or internet café and then print it off and post it to the researcher. Encryption can ensure email messages can only be encrypted by the intended recipient but equally it may complicate a project because all participants must use email software that shares the same encryption capability and the researcher and participants must have the technology in order to use the software. Additionally, encryption is illegal in some countries and may be viewed suspiciously by governments. Also a general way to increase data security is to regularly back up research data and store it in the most secure location possible.

CLOSE

Yes! Expectations of privacy are the important issue and different venues may have different expectations. Many social messages exchanged through the internet can foster the illusion of privacy because correspondents do not see the numerous people reading their messages, including lurkers to sites, so individuals often believe they are communicating with a small group rather than a large audience. So a key issue facing the online researcher is whether the individual or group considers their correspondence to be public or private. According to Ess and the AoIR Ethics Working Committee (2002, 7) if the participants of the research believe that their communications are made in private, or if they are understood as subjects participating in private exchanges via chat rooms/MUDs or MOOs, then there may be a greater obligation for the researcher to protect individual privacy. But if the research focuses on publicly accessible archives and if inter/actions by authors/agents are public and performative (for example e-mail postings to large listservs or USENET groups, or production of web logs and home pages), then there may be less obligation to protect individual privacy.

CLOSE

Generally, yes, although this will of course depend on the underlying methodological approach to the research and the research topic. Chen et al. (2004, 171) argue that debriefing should include the sharing of research results, so that the online community is made aware of the information that has been gathered from them. This sharing of research results can promote more egalitarian research relationships and can result in corrections to the researcher's analysis and interpretation of data. In this manner, sharing research results 'repel the feeling of being used by the researcher for selfish gains' (Chen et al. 2004, 172).

CLOSE

Based on their research with newsgroups, Hall et al. (2004, 244-247) recognize 6 issues of importance where netiquette is concerned.

1. The importance of the subject header used in any posting to a newsgroup, to assure no misunderstandings between the researcher and newsgroup members occur.
2. Self-identification and self-presentation of the researcher are critical, as readers will form their evaluations about the credibility of the research and the researcher based on this. A formal verifiable, disclosed identity of the researcher, for example through a link to an institutional website, can increase the credibility of the researchers claimed identity see (Madge and O’Connor 2002) and shows respect and courtesy to members of the newsgroup.
3. The researcher must be familiar with the common language used on the specific newsgroup, including jargon, abbreviations, acronyms, emoticons and common grammatical rules. The ability to 'speak' the newsgroups 'language' shows respect to the rules and conventions of the group.
4. The researcher should always ask appropriate questions, not ones that could have been answered by a library or archive search, and to do this the researcher should acquaint themselves on the subject matter before asking for help.
5. The specific culture of the newsgroup should be absorbed through online acclimation or reading FAQs and archives prior to 'jumping in', in order to understand the nuances of group interactions.
6. The researcher has an obligation to be 'up front' about the purpose, nature, procedures and risks of the research.

CLOSE

The following are a good starting point but there are many more:

Digital Divide Network
http://www.digitaldivide.net/
An online community for educators, activists, policy makers and concerned citizens working to bridge the digital divide. Users can build their own online community, publish a blog, share documents and discussions with colleagues, and post news, events and articles.

Bridges.org
http://www.bridges.org/
An international non-profit organisation that promotes the effective use of ICT in the developing world to reduce poverty and improve people's lives.

The World Internet Project
http://www.worldinternetproject.net
A project which originated at the University of California, Los Angeles Center for Communication Policy and which has set out to investigate and document the impact of the spread of internet usage.

World Summit on the Information Society
http://www.itu.int/wsis/.
Website reporting on the World Summit on the Information Society (WSIS), the first phase of which took place in Geneva hosted by the Government of Switzerland from 10 to 12 December 2003. The second phase took place in Tunis hosted by the Government of Tunisia, from 16 to 18 November 2005.

id21 viewpoints: World Summit on the Information Society. What did it achieve for ICTs and Development? What did it ignore?
http://www.id21.org/ viewpoints/ WSISNov05.html.
Reflections from Richard Heeks of the University of Manchester on the 2005 World Summit on the Information Society (WSIS).

See also the 'Further resources' section of the 'Ethics' module for other examples.

CLOSE

The simple answer is yes. There are a number of ways that you can set up a questionnaire. You do not need to be an expert in computers, but you may need to be prepared to spend some time learning some new skills.

If you do not want to spend a great deal of time doing this, producing the questionnaire yourself is not likely to be the best option. If you are working in an institution, it is a good idea to check what support is available to you. It may be that there are systems in place which you can use to get a questionnaire online.

Alternatively you could use one of the many online questionnaire software and hosting services that are available. These aim to make it possible to get a questionnaire online and gather results with little or no technical skills. They generally use a forms-based interface to take you through the whole process of developing and implementing the questionnaire. There are a wide range of options available for different budgets and with different features, and you may need to spend some time working out which of the services to choose. The 'Choosing software' section of the 'Technical guide' module includes an activity designed to help you to work out what features you will need for your questionnaire. It will allow you to develop a checklist of features that you can use when comparing products. The 'Using software' section also provides an outline of the typical procedure for creating and administering a questionnaire using these products.

The 'Introduction to online questionnaire production: Overview and options' section of the 'Technical guide' module provides a detailed overview of the different methods you can use to implement a questionnaire and the technical skills and knowledge that will be required depending on the method you choose.

CLOSE

From a technical perspective, this very much depends on what your experience of web design is at the outset, what support is available, and what method you choose to get the questionnaire implemented.

If you are developing a questionnaire yourself and you have little experience of web design, you will need to spend some time learning about technologies such as HTML and CSS in order to set up a web form and related pages such as an informed consent page. You can also use WYSIWYG (What You See Is What You Get) software such as Macromedia Dreamweaver or Microsoft FrontPage to get the questionnaire developed although an understanding of HTML will help you with this. You should set aside around 10-20 hours to develop the skills to develop the questionnaire and ensure it is designed effectively. If you plan to add extra features to your questionnaire such as validation or the gathering of information about participants' computers, you should expect to spend at least this time again to learn how this is done. Finally, if you do not have support to help you to upload the pages to a server and add server-side processes to validate and deal with the data collected, this is likely to take another 15-20 hours.

Of course any institutional support that may be available (such as an automated service to email results to the researcher) will reduce this time, but in this case you are likely to need to spend a couple of hours working through the details of the system to ensure that the questionnaire you create is suitable and meets any conditions that may be imposed.

If you use an 'off-the-shelf' software and hosting service you should expect to spend an hour or so finding the right service for your needs and budget and then to spend some time working out how the software is used to develop the questionnaire and how to use the different options available. You should have a fully working questionnaire in a few hours.

CLOSE

The minimum equipment you will need to design the web pages for the questionnaire is a simple text editor, such as notepad for windows, and a browser such as Microsoft Explorer or Mozilla Firefox in which you can test your pages. WYSIWYG (What You See Is What You Get) software such as Macromedia Dreamweaver or Microsoft FrontPage can also be used to aid the design of the questionnaire.

You will also need access to a server on which you can upload your questionnaire to make it available online. Depending on how you choose to process the data, you may also need access to a database on the server (Alternatively, you may choose to email the results). In most cases, a simple text editor is the only thing required for the creation of server-side processes to deal with the questionnaire data. Depending on the server and the server-side technologies available to you, however, you may also choose to download software which will help you to do this. See the 'Server-side processing' section of the 'Technical guide' module for more information about the different technologies and software that can be used.

CLOSE

If you are working within an institution such as a university, a government department, an NHS trust, or a charitable organisation, technical support may be available which may even extend to the bespoke conversion of a paper-based questionnaire to a web version. If this is not available, there may also be standard procedures in place for implementing web forms once they have been created by the researcher. This may, for example, involve a mailing facility whereby the data is automatically formatted and delivered by email when the form is submitted. It may also involve a facility allowing the data to be downloaded in an appropriate format for importing into a statistical analysis, database or spreadsheet package. The institution may also have purchased a site licence for an off-the shelf survey creation and administration package which may be suitable for the purposes of the research project and, in educational institutions in particular, site licences may also have been purchased for assessment software which may provide the facilities required for the implementation of a basic questionnaire.

The first step in finding out about these systems is usually to check the institution intranet or to contact computer services.

CLOSE

In HTML pages, there are five basic forms of form controls for inputting data as follows:

Of course, these can be organised in different ways to create different question types. For example, tables can be used to group radio buttons into grids for Likert scales or semantic differential questions, as in the following example:

See the 'Web forms' section of the 'Technical Guide' module for further details.

CLOSE

There is no simple answer to this question as this is likely to depend on the context of the research. An activity designed to explore the length of time needed to complete different types of questions is available as follows:

CLOSE

For check boxes, radio buttons and select boxes, it is possible to specify which of the options is specified by default as shown below:

In the case of radio buttons and check boxes it is a good idea not to set a default option as it will be impossible to establish whether or not the option was actively selected by the participant or whether the question was unanswered. For select boxes, if the default answer is 'Choose an option' as in the example above, this will not be a problem as this answer will clearly indicate non-response.

CLOSE

One-page questionnaires are generally more straightforward to implement as they consist of a single web form which can easily be processed through the submission of this one form. Where the questionnaire is relatively short and straightforward in terms of structure, this is likely to be the best option.

However, with longer or more complex forms, attempting to present the entire questionnaire in one page may lead to problems such as the following:

1. Presenting all questions at the same time may give an impression of greater length which may discourage participants from proceeding.
2. Opportunities to validate individual questions or smaller groups of questions as the participant progresses through the questionnaire may be reduced (see the 'Form validation' section of the 'Technical guide' module). In turn, this may lead to frustration if all questions are validated at once at the end of the questionnaire.
3. Although skip patterns can be introduced through linking to anchors further down a page or through instructing participants to skip a question by scrolling to the next, this may not be the most effective or intuitive method of delivering the questions.
4. If a participant drops out mid-questionnaire, all data will be lost and there will be no opportunity for collection of partially-completed questionnaires or for identification of questions that may be precipitating drop out.

For longer questionnaires, the use of multiple pages can add to the effectiveness of question delivery, providing clearer routes through the questions and offering the opportunity for a more sophisticated presentation of skip patterns. For example, links to different sections can be added which participants can be prompted to select according to the answer to key questions. However, because all the questions are not made visible, submitted and processed at the same time, a number of extra aspects must be considered

1.  An indication of progress through the questionnaire must be given, either through the use of a progress bar (see 'Key design issues' section of the 'Technical guide' module) or through structuring the questionnaire into different sections and indicating the nature of this structure to respondents (see the information about questionnaire length in the 'Design of online questionnaires 2: Content' section of the 'Online questionnaires' module). If this is not done effectively, uncertainty over progress or a realisation that the indicators of progress are inaccurate, may lead to frustration and drop out.
2. A decision must be made on whether data should be submitted for processing at the end of each page, or at the end of the questionnaire. If it is done at the end of each page, this will allow partially-completed questionnaires to be collected and any problem questions to be identified, but measures must be taken to identify or prevent multiple submission of any sections. If it is done at the end of the questionnaire, it will be necessary to pass information entered by participants up to a given point in the questionnaire from page to page.
3. As participants progress through the questionnaire, they may wish to return to a previous page to review and change answers. Unless measures are taken to ensure that the data they have already entered is still available when they do this, it is important to inform them that answers already entered may be no longer available if they go back. It may also be important to add instructions not to go back through a questionnaire if each page is to be submitted individually, or to add validation routines preventing submission a second time.

CLOSE

There are a number of measures that can be taken to ensure that a questionnaire is suitable for older browsers and equipment. These include maintaining a straightforward design, avoiding the use of third-party plug-ins such as Macromedia Flash, ensuring the questionnaire is designed for accessibility (see question below) and validating the HTML and or CSS used to create it. This should ensure suitability for older and or less powerful computers. However differences in aspects such as the size and appearance of form elements and tables may remain when viewed on different systems.

It is thus good practice to test pages on as many different browsers and systems as possible. Friends and acquaintances with different systems (e.g. AppleMacs or PCs) and older versions of browsers can also be called on to test for any problems. Where design problems are found in particular systems and browsers, an attempt can then be made to change the design to best accommodate them.

It is also possible to collect information about the user's computer and browser alongside the data from the questionnaire (see the 'Gathering information about participants' section of the 'Technical guide' module). This can allow an overview of the technologies available to respondents to be gained. If it becomes clear, for example, that older or less common browsers are being used to access the questionnaire, it can be tested on these browsers and redesigned if necessary.

It is also a good idea to test the questionnaire on different screen size settings to ensure that questions are visible in their entirety and do not require scrolling, to use web-safe colours which will display consistently on different monitors and to use common fonts and 'font-families' to ensure that the fonts used will display consistently on different systems.

See the information on consistency in the 'Key design issues' section of the 'Technical guide' module for further information.

CLOSE

It is important to design pages to be as accessible as possible to members of the target group who may have disabilities. There are a number of simple steps that can be taken to increase the accessibility of an online questionnaire and its associated web pages. These can ensure that the contents are accessible to users with a range of user-agents including text-only and screen reading browsers and other assistive technologies.

Designing the site to be compliant with standards set out by the World Wide Web Consortium (W3C) is an important step in ensuring accessibility. The 'Resources' section of the 'Technical guide module includes a link to W3C's validation tools which allow web pages to be checked for standards compliance. By uploading a page or entering the URL, the tools will run automated tests and report on any pieces of invalid markup in the pages.

It is also good practice to separate content from presentation in web pages by using Cascading Style Sheets (CSS) to add design features (See the 'Introduction to CSS' section of the 'Technical guide' module). Although it should be remembered that this may lead to increased inconsistencies in display on older browsers, this allows participants to control how the site should be presented. They can override style information to allow presentational features such as text size, font, colour and layout to be changed according to need.

Beyond this, the W3C Web Accessibility Initiative guidelines (WAI) includes a wide range of measures which should be taken to ensure that a website is accessible (http://www.w3.org/ WAI/intro/wcag.php). The guidelines divide these measures into Priority 1, 2 or 3 according to how essential they are to accessibility. Some key measures that should be taken for accessibility are shown in the 'Key design issues' section of the 'Technical guide' module.

CLOSE

It should be possible to access a well-designed questionnaire on a wide range of web-enabled devices including TVs, mobile phones and Personal Digital Assistants (PDAs). By designing for accessibility and separating content from presentation (see question above), you will be able to make sure that the pages are as accessible as possible to devices such as these as well as to text-only and screen-reading browsers and other assistive technologies.

CLOSE