top

Q1: I want to analyse a weighted survey, but I don't have any special survey software. Is it OK just to make sure that my weights add to the achieved sample size and then use a simple weighted analysis, such as you would get by setting a weight variable in SPSS?

Congratulations. You are obviously a careful analyst and you notice that your results make sense. What you propose will not be too bad, expecially if your survey is a simple one with weighting as its only design feature.

You will always get the right estimates from a weighted analysis, no matter what kind of weights you use. But to get the right standard errors you ought to use probability weights (see section 3.9 of the P|E|A|S site). The usual weights you get in packages like SPSS are frequency or analytical weights. The formula for the standard errors for probability weights is quite different from the other kind. But it will not be the wrong order of magnitude if your weights add to the sample size, as can happen if you don't scale your weights. See exemplar 1 for a practical illustration of this.

If the survey you are analysing has other features like stratification and clustering, a statistical package without these features will not make allowance for this.

top

Q3: My superviser says I should use multi-level modelling for clustered samples. Is this the same as analysing a survey and setting a cluster option in a survey package? Is multi-level modelling better?

Your superviser is quite right. Multilevel modelling is a possible approach to clustered data. It is not the same as using methods for clustered surveys. The two methods were devised for very different sorts of application.

Multi-level modelling was devised for situations where clusters were of interest in theor own right, for example schools, hospitals. It allows you to learn a lot about how your outcomes vary within and between clusters. In survey methodology the clusters are not usually of interest, but are only used for the convenience of the interviewers. A clustered survey analysis will not tell you about the differences between the clusters.

You should discuss with your supervisor whether the clusters in your data are of intrinsic interest, or jsut a nuisance. If they are of interest then you should use multi-level modelling. If they are just a nuisance, use one of the survey packages suggested on this site. The choice of a clustered survey analysis will have the advantage of being free from the modelling assumptions that are needed for a multi-level analysis.

top

Q4: Obviously, it isn’t a good idea to use weighting if your values vary enormously (for example, you have 96 males and four females, and you are attempting to balance it to 50:50. When is it unadvisable to use weighting e.g. is it closer to a 75/25 or 50/50 spilt on values?

I think the situation you refer to is most likely to happen when your weights are the result of non-response, and thus not likely to have been chosen so as to improve the precision of your results. The section on non response weighting has sections that discuss the effect of weighting on precision of estimates. In response to your question I have also added a paragraph to the end of section 5.7.