top

1.1 Introduction

1.2 Getting Started

From links in this section you can:- Downlaod or open the data files Analyze them with any of the 4 packages you have available View the code (with comments) and the ouput, even if you don't have the software. To start, click the mini guide for the statistical package you want to use to analyse Exemplar 1. For additional help click on the appropriate novice guide. For details of the data set see below.

Mini Guides R Stata SPSS SAS Novice Guides R Stata SPSS SAS

top

1.3 Different answers for the mean income and its standard error.

Comments

Weighting corrects the bias in the income estimates and increases the mean weekly income by about ₤13, and somewhat increases its standard error (B).

The theory section explains different kinds of weighting (design, frequency and analytical). Wrongly defining frequency weights (C1) (the default in SPSS) gives much too small a standard error here because the weights are grossing up weights and this makes the program think you have an enormous sample. Analytical weights or euqivalently rescaling the weights so they add to 1 gets an answer in right ball-park, but not what it should be for probability weights.

The correct standard error for this design is D, which also allows for clustering into 320 clusters. This increases the standard error of the estimates even more.

Post-stratification to match two margins (raking) changes the weights so that they match the Scottish population totals. It changes the mean income just a little, decreasing the mean income by about ₤3.50. This adjustment seems mainly to be due to the population having a higher proportion of privately renting households than the original weighted data. When it is analysed correctly it also improved the precision of estimation. In this example, the advantage of post-stratification regains the precision lost by the clustering.

In order to recognise this improvement in precision in reducing standard errors you need to analyse the data using a method that takes the post-stratification into account. Just adjusting the weights by itself will not do this. In fact, in this case, it makes it look as though the change in weighting has had no effect on the standard errors (D after). But analysing it with allowance for the post-stratification using either a calibration method (implemented in R) or a jackknife method shows that it has improved things substantially (E),

Is it worth the bother of post-stratifying?

It depends if you believe the totals are correct. If you can accept this, then the effective sample size for the whole sample would be increased by a factor of almost 2 for this analysis. This has to represent a BIG saving. But if you want to analyse it with correct methods it brings some overheads.. At present that means using either a method that allows for it by calibration (only R of the packages we feature) or you need will need to use replication methods (R or Stata or SAS, with more effort). Replication methods are more work, but Stata and especially R have very easy to use routines for them. See the very concise code that you need for R for this exemplar. But the advantage will be less for subgroups (see belowsection 1.6).

top

1.4 Pitfalls:

When a survey is post -stratified members of a cluster can fall into different strata. This means that you cannot simply define your post-strata as though they were strata. Splitting PSUs would be wrong as it would imply that the clusters consisted of lots of small clusters, for the bits that fall inside each PSU.Some method that makes specific allowance for the post-stratification needs to be used. Two such methods are available in the packages we are featuring on this site. The easiest to use is a calibration method available in R, and the alternative is to use replication methods. The figures in this table are calculated by replication, but calibration gave almost identical answers with much less effort.

When you define the post-stratification as though it were just ordinary disproportionate stratification (as carried out at the design stage) the packages do different things. The problem is that this divides PSUs between strata. This results in a split PSU problem.

• The R survey package prints a warning. If you check the help you will see
  that the option to, split the PSUs is there and it is suggested in the warning.JOINT BEST BUY *****
• SAS prints a warning in the log file and suggests that the option of
  splitting the PSUs between strata be considered. JOINT BEST BUY *****
• Stata splits the PSUs with no warnings unless you notice that you now
  have lots of PSU's instead of just 320.
• SPSS splits the PSUs but does not appear to give any output that shows how many PSUs you have, so no warnings.

top

1.5 Percentiles of the distribution

Table 1.4 Percentiles of the distribution and their S.E.S

The results below show how the standard error of the upper percentiles increases as the distribution of incomes becomes more sparse.

Note also that the post-stratification has given slightly more precise estimates of the percentiles and reduced the estimates a little.

top

Among the 4695 households surveyed, just 334 were lone parent households. We want to investigate the income in these families.

In order to carry out analyses by sub groups for survey data it is NOT CORRECT to make a
new data set consisting of just the subset of the data and analysing it as if it were a complete survey. Instead a special routine to look at subgroups needs to be used. The reason for this is that, when there are strata or post-strata, the subgroup of the survey will no longer balance to the stratum totals.

All the packages considered provide this facility in one form or another. The table below gives the results for an appropriate method (with and without post-stratification . Notice that the advantage of getting a smaller standard error due to the post-stratification in A has now disappeared.

Table 1.5 Mean and standard error of income for single parent households in Scotland analysed by different methods.

When you check the output from some of the programs you will see that you get a different design effect. What we have quoted is a design effect compared to having taken a simple random sample of 334 single parent households over all Scotland. Another way to define a design effect is compared to the number of single parents we would have obtained had we taken a simple random sample of all households in Scotland. This would have given us a different number of households in our sample, so it gives a different answer (0.84 for the design effect). This method of calculation is the default in Stata and the only option in SPSS.

top

1.7 Details of the data set and survey design

We have constructed a data set for this analysis with just those variables we will use. The process by which the data set has been constructed and the programmes used to make it from data at UK Data Archive are explained in detail here The variables in the data file are:

NAME	DESCRIPTION
HHINC	Gross household weekly income (₤s)
DEPCHLDH	Dependent children in household
ADULTH	Number of adults in the household
PSU (PSU)	primary sampling unit here the post code sector- renumbered to prevent identification
CT BAND	council tax band used for post-stratification to Scottish totals. (1 to 8 for A to H and 9=not separately assessed)
TENTYPE	type of household tenure used for poststratification to Scottish totals. 1= owned 2 = LA rented 3 = other public rented 4= private rented or rent free
GROSS2	weight provided by the survey contractors calculated from all aspects of the design including poststratification to UK totals, but not to the Scottish totals.

Further details of the survey can be found here and in particular the methodology section. Questionnaires and further details are available from the ESRC supported Question Bank.

Sample Selection

The sampling frame is the post code address file (PAF) a list from the post office of all addresses in the UK. We are using data from the financial year 2002-/03
for Scotland only.

PSU's / Clustering

The FRS sample is a clustered sample where the primary sampling units (PSU's) are postcode sectors with up to 25 respondents from one sector. The data used has 4695 households in 320 PSU's.

Weighting

Figure 1.1: Box plots of weights by number of adults in the household

The weights in the file are grossing-up weights that add to the population size. There is fairly large range of weights with some very large ones, relative to others. The range of weights by adults in the household is shown in Figure 1.1. The median weight is 466. As this is a survey of households, not people, single person households are not down-weighted. The major factors in weighting probably relate to the post-stratification carried out at the UK level.

Stratification

Stratification was used in selecting the sample based on eight area based socioeconomic groups for the PSU's within regions where Scotland is one region. These design variables were not available with the data and have been ignored in these analyses.

Post-stratification

The FRS data were post-stratified at the UK level, using the method of raking, to make the total households match national totals by tenure and council tax band. In this exemplar we illustrate how to carry out post-stratification for the Scottish data alone. Total numbers of Scottish Households by Council tax band are available froma Scottish Economic Statistics publication. The 2001 census was used to get data on housing tenure in four broad bands.

Table 1.7 compares these outside data with the weighted sample totals. We can see that differences between the sample and the population percentages are small, probably due to the raking carried out a t a UK level.

NOTE: This table has French headings because it is based on the output from the SAS CALMAR macro, featured in the SAS analyses, which is written in French.