For voluntary surveys the greatest threat to the accuracy of the survey estimates is non-response. Different surveys achieve different response rates, the surveys with the highest response rates tending to be those that ask questions that seem relevant and interesting to respondents. But, even with ‘popular’ surveys, response rates have been declining in recent years, and, as a direct consequence, worries about survey bias have been increasing.

Non-response is only a problem if the non-respondents are a non-random sample of the total sample. Unfortunately, this seems almost always to be the case. In household surveys, for instance, there is lots of evidence that non-respondents are younger than respondents, and that men are harder to persuade to take part than women. Response rates also tend to be lower than average in cities and in deprived areas.

The result of these patterns is that the achieved samples for surveys often don’t reflect the population they are meant to represent very well. Surveys typically over-represent women, and those over the age of 30. And they often under-represent those living in cities and deprived areas.

Most non-response weighting schemes involve ‘post-stratification ’. This is essentially a two-step procedure:

(i) identify a set of ‘control totals’ for the population that the survey ought to match;

(ii) calculate weights to adjust the sample totals to the control totals.

As a very simple illustration, suppose the adult population distribution by age and sex is:

To poststratify the sample, weights would be calculated that bring the sample distribution into line with the population. The weight applied to men aged 16-39 would be 22.1/20.2; the weight applied to men aged 40-59 would be 14.9/13.8; and so on.

top

There are alternative methods to use in the setting where the full n-way table for the population is not available, but the marginal distributions are. (If, for example, you know the sex distribution and the age distribution, but not the sex by age distribution.) The usual approach is to use a method called raking or iterative proportional fitting. Exemplar 1 uses the Family Resources Survey which uses this method.

Another post-stratification approach that is rapidly gaining ground is ‘calibration weighting’. Calibration is now becoming the standard for surveys of individuals selected via households, especially in the particular case where more than one individual is selected per household. The reasoning is that standard post-stratification will tend to give a different non-response weight to each member of a household and this can create difficulties for household-level analyses. Calibration uses an iterative procedure to create weights that bring individual level survey data into line with the population, but with the constraint that all individuals within a household must have the same weight. The underlying assumption is that non-response is primarily a household decision rather than an individual decision (in the sense that non-response is usually determined on the doorstep before any individuals are selected) and it is household level non-response that creates most of the discrepancy between the sample and population distributions.

Regression models can also be used in simpler situations (See exemplar 5) as an alternative to simple post-stratification . They are most useful when a lot of information is available from the population so that the cells for post-stratification would be too small and might result in very variable weights that have a lot of sampling error. The regression models effectively smooth the weights so as to get more stable estimates.

The most underrepresented group (urban, not publicly funded) gets the highest weight to bring the sample into line with the sampling frame.

When the sampling frame contains more detailed information about the non-responders the factors that influence non-response are often investigated via logistic regression. The resulting model is then used to calculate the probability of response at step (i) above, and the subsequent steps are carried out in the same way as above.

When this type of regression model is used we need to strike a balance between having a powerful model to predict non-reponse (and so reduce bias) and the introduction of extreme weights that will affect precision see weighting section 3.7. Models are often simplified at the final stage to avoid extreme weights (either large or small). Another practice that some surveys employ is to cap the weights, for example by replacing all weights above 2.5 with the value of 2.5.

Survey statisticians claim that non-response weighting should reduce bias,
but they acknowledge it will not eliminate bias. The reasoning is:

- The reasons individuals choose to take part in surveys are complex, and depend upon lots of factors specific to individuals. As a result, the non-response biases in surveys are likely to be complex;
- Post-stratification works on the assumption that, by aligning the survey to the population along a small number of dimensions (such as age and sex), many of these complex biases will reduce. But it would be misleading to suggest they will be eliminated.

To illustrate the point: In the example above, where a survey is post-stratified to the population for age and sex, for non-response bias to be removed we have to make the assumption that the men aged 16-39 who responded to the survey are representative of men aged 16-39 in the population. The same assumption has to be made for the other age and sex groups too. But it is easy to imagine scenarios where this is not the case: for instance, amongst men aged 16-39 it could be that those who live alone were less likely to take part in the survey that men who lived with others, or it could be that men aged 16-39 in professional occupations were less likely to take part than others. Or more intangibly, but realistically, it could be that men aged 16-39 who are uncomfortable about talking to strangers about themselves are less likely to take part than others. Post-stratification by age and sex will not eliminate survey bias such as these.

This is an illustration of the idea of the missing respondents not fulfilling the requirement of being missing at random (MAR - see imputation section 6.3)

the main motivation for post-startification is to remove bias. But poststratification should, if done well, reduce the standard errors of most survey estimates (relative to only applying inverse selection weights). The exceptions are:

(i) if the variables used as control totals are unrelated to the survey variables;
(ii) if there are small numbers of extremely large weights.

In most cases you should be able to assume that the survey organisation responsible for the calculation of the weights will have checked for both of these. (The standard practice is, for instance, to trim very large weights.) However, it is good practice to check the distribution of the weights, and if there are some very large weights, to make sure you understand how and why they have arisen. Chances are they are there for a reason but errors do happen.

But post-stratification also brings problems.

• It relies on the totals being correct.
• If post-stratification cuts accross strata correct standard errors require more complicated methods of analysis. Replication methods are the usual way this is dealt with and they are complicated to use. The alternative of using a calibration method, is only available in one of the packages we include (R, Survey package). It seems to give results very close to those from the replication methods.

Additionally, the benefits of post-stratification (like stratification) are largely confined to whole sample analyses. WIth subgroups that cut accross strata they both of these procedures may have much less benefit (see exemplar 1 section 1.6 )