Introduction to online questionnaire production: Overview and options

How did you implement the questionnaire?

The development process was relatively straightforward and the questionnaire was programmed by one of our technical staff.

Did you experience any problems at all?

We did experience some database issues during implementation (sometimes buttons didn't submit information properly), but nothing that couldn't be resolved fairly quickly.

Were there any issues around response rates?

We mailed a printed version of the questionnaire to everyone who didn't respond to the two online requests, which increased our response rate significantly. My experience has been that printed surveys are needed to motivate large numbers of non respondents to participate, who otherwise would ignore the Web survey.

What was the context of the research?

The research investigated factors that influence geography faculty members’ participation in activities that serve to internationalize college and university campuses, including international collaborative teaching and research.

Why did you choose to carry out the study using an online questionnaire in tandem with a paper-based version rather than a paper-based version or an online version only?

To lower data collection costs, an online instrument was developed. The response rate from the online questionnaire was not as high as anticipated, so paper-based versions of the instrument were sent to online survey non-respondents.

Was this decision made before or after the development of the questionnaire?

The online questionnaire was developed first, followed by the paper-version of the survey. From the beginning of data collection, the paper-based version of the survey was made available online for respondents who for personal preference or accessibility reasons did not want to complete the online version.

Why did you choose to have the questionnaire produced and hosted in-house?

Staff and resources were available to produce and host the questionnaire in-house.

Did you consider any other options for implementing the questionnaire (e.g. off-the shelf software and/or hosting solutions?) If so, why were they rejected?

To my knowledge, no other options for implementing the questionnaire were considered.

How involved were you in the technical implementation?

I was not involved in the technical implementation, other than as a trouble-shooter for the beta version of the online survey.

What did you actually have to do to get the questionnaire online (e.g. hand over a paper version / liaise over design and development / specify technical features required such as access control and format of resulting data)?

I provided the programmers with a paper version of the survey and the programmers took care of the design and development of the questionnaire. I did not provide the programmers with format requirements for the resulting data. As a result, I assigned numerical codes to the data prior to analysis. This step could have been eliminated if I had provided the programmers with format requirements.

Would you recommend your methods of implementing the questionnaire to other researchers? What were the advantages and disadvantages?

I would recommend my methods to other researchers, although I had the benefit of a staff of programmers that were responsible for the technical development of the online questionnaire which made an online questionnaire viable for my research. There were many advantages of an online questionnaire, including (a) the ease of contacting survey participants in far away places; (b) the low cost of using online methods versus paper/mail-based methods; (c) time for data-entry was lessened; and (d) errors resulting from data-entry were lessened, as well. The main disadvantage is that the anticipated response rate was higher than the actual response rate. So it was important that we had the resources to mail paper-versions of the questionnaire to survey non-respondents.

What advice would you give to people in working with technical support staff to get a questionnaire implemented?

In working with technical support staff, I would suggest that other questionnaires are reviewed so that there is a clear understanding of what format would best suit the aims of the research. Then, the finalized questionnaire items, the array of possible responses, and numerical codes for the resulting data should be provided to the technical support staff. Creating beta versions of the survey is important, so that the survey can undergo revisions.

Were there any issues that emerged through the process that you hadn't expected?

Technical issues arose that the support staff resolved without my involvement. The expected response rates were lower than the anticipated response rates, which resulted in delaying data analysis until paper-based versions of the questionnaire could be received from survey non-respondents.

What experience of web page production had you had prior to setting up these pages?

I had very little web page production prior to participating in this research.

Did you have to learn any new technical skills?

No, I did not have to learn any new technical skills.

What was the context of the research?

We were a team of non-technical academics from social science and psychology disciplines. None had experience of setting up online questionnaires, though all were proficient in questionnaire design. The survey was of Leeds University academic staff (all grades) regarding work-life balance. The questionnaire comprised four sections (three research areas plus a demographic) and was quite long.

Why did you decide to use an online questionnaire?

Really, our driving force was cost (so the online bit was a relatively cheap option saving on photocopying and return postage) and ease of data entry. We were looking to have the responses migratable to a format compatible with SPSS 'by magic'!! Which worked! The questionnaire was in html and dropped all responses into a comma delineated format, which could be opened in Excel and copied across to SPSS.

When did you make the decision?

Actually, (as academics) we were more concerned with the questionnaire itself - which took around 1 year to develop - and the issues around making it available online were addressed proportionately quite late on.

Why did you use the university to host the questionnaire (as opposed to a commercial hosting service)?

Never occurred to us to pay outside people!! We had obtained a little funding, which was just enough to pay an internal computer guy within one of our departments plus incentives for recipients...

Were there many issues that emerged through the process that you hadn't expected?

All the points below came about as a result of conversation with other colleagues and the technical staff member involved in setting up the website. None were issues we had consciously considered prior to embarking upon an online questionnaire

• Password protecting the website, to prevent people completing the questionnaire from some other 'sample population'.
• We knew we wanted respondents to view the three research sections in a random order (to prevent fatigue on one particular section) and this was done automatically as people accessed the questionnaire
• We indicated the rough time duration for each section and colour coded the sections for clarity.
• To prevent people missing questions by mistake, a check was performed on missing answers for each section and the respondent informed of any omissions (highlighted in red). If they still wished to skip the question, the submit button would take them to the next page the second time around.
• Logic pathways had to be very clear for programming purposes (e.g. If answering X to question Y, go to section Z etc).
• A lot of attention had to be paid to the coding, both in SPSS and Excel, as well as the main data file and a few dry runs performed to make sure that data wouldn't be lost.
• The questionnaire was designed in Netscape and the layout appeared distorted initially when running in Explorer.

Why did you use web-based questionnaires?

I used web based questionnaires not because they are innovative, but because they were wholly appropriate. They offered a dynamic alternative to paper based questionnaires, with the prospect of a higher return rate as there is no work involved other than the responses themselves. Also the automatic coding of answers meant that responses could be interpreted quickly and saved into databases and separate files relating to each respondent.

What experience of web page production had you had prior to setting up these pages?

I had no experience of web programming before creating the first version of the online questionnaire. HTML was easy to pick up and so I was able to teach myself the majority of what I needed to know, but support was available.

Did you use a WYSIWYG editor like Macromedia Dreamweaver or Microsoft FrontPage or code them by hand?

The web-form was created using a combination of WYSIWYG and hand coding – I think there are pros and cons to simply using WSYIWYG as it is important to understand what you are doing and how you are doing it to avoid confusion further down the line!

How did you go about implementing your questionnaire?

Firstly, I read up about basic HTML programming and learnt the basics. This allowed me to develop the skills needed to produce my questionnaire. I then took advantage of the University’s facilities to host the questionnaire and to link it to server-side scripts allowing the data to be automatically emailed to me.

What kind of support did you receive?

The computer centre at the university offered a good level of support in setting up initial web-form and the server-side scripts for submitting results. Subsequently they helped iron out minor problems with the format of the questionnaire and the way in which results were received via server-based emails.

What exactly were these problems and how did you solve them?

Firstly I had problems with the frames in the webpage. By playing with the page, I managed to get the optimal viewing resolution lower so it appeared on even the lowest resolution and was easier to read. The problem with the automated replies was firstly with the coding. Initially the responses were uncoded and were received in a format that made it difficult to deal with the data. Both of these issues were resolved through trial and error and with the advice of the computer centre until I was receiving the results in the most user-friendly manner.

Why didn't you use off-the-shelf questionnaire software?

Using my limited and self taught knowledge of HTML programming I used Microsoft FrontPage to produce a clear and simple web-based form, designed using university corporate logos and colours. This was preferred over using off the shelf software because of the ability to build the web-form to my exact specification, also using widely available software meant teething problems were solved with the computer centre. At the initial time of conception I was also unaware of any appropriate software for designing questionnaires, and on further enquiry did not have the resources to purchase such software.

What did you do to check the data you received?

All of the respondents to the questionnaire were invited. With name and contact details on the questionnaire itself, responses were verified and constituted the basis for determining whether respondents would subsequently be interviewed. None of the data collected was explicitly presented within the research statistically or otherwise, but informed the interview process. The fact that the questionnaire was sent to invited respondents meant that security was not really an issue so I avoided many of the problems involved with ensuring the integrity of results and respondents.

Would you use the same system again if you were to do a similar study?

The system was easy to use and worked for me on this occasion. I am not the greatest fan of using surveys or questionnaires; however this was a lot easier to manage than a traditional paper based survey at every stage. The ability to modify questions and responses that did not work or were inappropriate after the pilot study was easy to do at any time when the initial questionnaire was set-up. If I did need to use a survey again I would use online over traditional every time!

What advice would you give to people in using systems like these?

I'd say to be persistent! If you have an idea, the chances are that it is possible but you just aren't sure how to do it at the moment! As a social scientist my forte is not designing websites or programming, but there are people out there who can help - the challenge is finding them and learning something new!

1. Why did you decide to use a commercial internet hosting service rather than using the services offered by your university?

As I am doing my PhD at a number of different universities (Cambridge, Cardiff, Freiburg, Berlin), I did not want to depend on any university services for running my studies (I often find it hard to solve problems, when I am not actually there). I am now using a commercial provider (rather cheap) which offers php-support (I understand that universities often do not offer this support or need time to check any scripts). As long as no reaction time measures are involved, this seems to me a very good solution.

2. Why didn’t you use some of the off-the-shelf questionnaire software which is available online?

I did not use any software package to produce my studies, because I like to have full control over what is displayed, how the data is stored, etc. (often, for example, software packages rely on cookies to temporarily store the data, which I don't like). At least for questionnaires and simple experiments, the programming seems to be manageable (and I am really not a computing specialist).

3. What kind of technologies did you use to implement your questionnaires, and have they been successful from a technical perspective?

My studies use nearly exclusively php (and mysql for saving the data) and have been working perfectly well. For questionnaire-based studies without reaction time measures, I think server-side programming really is the way to go and I am confident in saying that pretty much anyone with a Web browser can see and use my pages.

4. How did you achieve this level of accessibility?

JavaScript is only used for setting cookies (in order to check for multiple participation) and checking whether answers have been provided in order to remind participants that they might have forgotten to fill in an answer (both only in one of the studies). So if anyone has switched JavaScript or cookies off, he can still participate.

5. What did you do to check the data you received?

For fraud detection, I look at IP-addresses, cookies that I set and display a message if a cookie is set. Also, it seems a good idea to receive all the submissions by e-mail as well as saving them to a file. This way, one quickly gets a feel for "unusual" patterns of data submission (e.g. very many in a very short time) and can check the respective data sets more carefully.

6. What experience of web page production had you had prior to setting up these pages?

I had no experience before starting online studies (well, my first Website was actually a signup page for lab participants including a questionnaire measure).

7. Did you use a WYSIWYG editor like Macromedia Dreamweaver or Microsoft FrontPage, or code them by hand, and did you have to learn many new skills to do this?

I code my pages by hand using a freeware tool called HTMLkit. I had to learn to understand what html does (which is really fairly quick) and how pages can be formatted using Style Sheets. Learning this really does not take that long (maybe 15-20 hours).

8. Did you write your own php scripts to write to the database and send the email, or did you use or adapt an existing one? Again, did you have to learn many new skills?

I used and adapted existing scripts. Although I understand the basic principles of programming, I find it very hard to put it all exactly the right way and there are lots of scripts freely available online (many of the bits one needs for a questionnaire study - like setting cookies, retaining referrer URL, retaining IP-address are fairly standard and only short pieces of scripts). I have to say, though, that for one study that involves multiple randomisations and lots of different conditions, I had quite a bit of help by a friend who does computing. So if I had to do such a study all by myself, I might either have to put in more time to learn how to properly program or find a tool that I could use. But this does not apply to simple questionnaires.

9. Presumably you developed your questionnaires on a test server (apache?) on your home computer, and then uploaded them to the commercial provider (FTP?). Is this correct?

Yes, I installed foxserv (includes apache and mysql). Often, however, I upload it directly and test it online.

10. How did you choose which provider to go with?

There are lists of the 10 or so best Web providers for each country (my provider is in Germany). I knew I needed php- and mysql-support, so then I compared the prices for the ones that offered these and was either lucky or they are all good. So far, I never noticed any interruption of service.

11. Finally, would you recommend your methods to other researchers?

Yes, although, thinking about it, I might not have done things the most efficient way. The reason being that I did not mind so much investing some time - just out of curiosity.