This exemplar is based on data from WERS98 to look at the relationship between workplace composition and equal opportunities policies. The example was motivated by analysis reported in the Department of Trade and Industry (DTI) report ‘Equal Opportunities policies and practices at the workplace: secondary analysis of WERS98. Anderson, T., Millward, N., and Forth, J. DTI Employment Relations Research Series No. 30).

The exemplar aims to illustrate analysis of survey data where a key feature of the design is that the sampling fractions (and hence the survey weights) are very different across strata.

The data for this exemplar were obtained from the ESRC data archive, but they have been altered to prevent the disclosure of individual workplaces using the principles described here. The way in which the data provided here relates to the data from the archive can be viewed here.

In addition the sample was stratified by the Standard Industrial Classification (SIC), although within size-bands, the sampling fraction by SIC group was kept roughly constant. WERS98 can be thought of as an extreme example of a sample design using stratification with unequal probabilities of selection. The strata are ‘size by SIC’ categories.

To derive unbiased estimates for workplaces the very large mismatch between the sample distribution and the population distribution has to be addressed. This is achieved by weighting the sample data, the weights being calculated as the inverse of the probability of selection.

WERS98 also included a sample of employees but we have not made use of that dataset here.

The analysis carried out here demonstrates the impact that survey weights can have on survey estimates and their standard errors.

The example looks at the proportion of workplaces that have an equal opportunities policy, and the relationship between this proportion and other workplace characteristics, in particular the percentage of employees that are female, disabled, or from a minority ethnic group.

We can see that adding the survey weights greatly reduces the estimate of the percentage of the workplace with an equal opportunities policy. This happens because there is a very wide range of weights (see below). Also the weighting ‘weights down’ large workplaces and ‘weights up’ smaller workplaces, and having an equal opportunities policy is very strongly related to workplace size.

The standard error is more than doubled in the weighted analysis. We can see in Table 4.4 that the largest workplaces nearly all have an equal oportunities policy. For this outcome the optimum weighting strategy would have been to 'weight down' these strata because they are very homogeneous. But the opposite is true for the design of this survey. Of course this conclusion only applies to this outcome. See the theory section for a discussion of this.

We now compare other workplace characteristics, such as the percentage of employees that are female, disabled, or from a minority ethnic groups between workplaces with and without an equal opportunities (eo) policy. We start with bivariate analyses, that simply check whether the percentage of employees in each of these groups differs by whether or not there is an equal opportunities policy. The unweighted and weighted results are shown in the table below.

We see from this that the survey weights have rather a large impact on the differences in means. Most strikingly, without weights the mean difference in the percentage of women between workplaces with and without an equal opportunities policy is just 11%. But with weights the difference is 17%. It is also striking that adding the weights also increases the standard errors of the differences quite considerably.

We can also see that the unweighted analysis suggests that there are differences between 'eo' and 'no-eo' workplaces in the percentage of disabled empoyees. But the weighted anlysis, adjusting for the survey design does not confirm this. This difference was also seen when a chi-squared test was used. The unweighted chi-squared test gave a p-value of 0.000000005 for the association between 'eo' and the grouped variable for disabled employees. Whereas the weighted chi square test, adjusted for the design (theory link ) gives a p-value of 0.20 (not significant).

The natural next step is to include the three employee characteristic variables in a single regression model, with ‘equal opportunity policy’ as the dependent variable. This suggests a logistic regression model. Again we compare unweighted and weighted model coefficients to check for the impact of the weights.

The table below gives two different logistic regressions to predict the log-odds of having an equal opportunities policy. The first is unweighted and makes no use of the survey design. The second is a weighted analysis and adjusts for the survey design.

Comparing the two columns from this table we see, again, that the survey weights has changed the estimate. The coefficient for the group 'more than 3% disabled', which changes from positive to negative after weighting the data. But neither coefficient is significantly different to zero, so this is probably less important than the fact that the weighted estimates for ‘female’ and ‘minority ethnic’ increase by about 50% after applying the weights.

Note again that the standard errors are larger for the weighted than for the unweighted estimates.

The reason the weights have such a large impact on the survey estimates for WERS98 is that the weights adjust for the very large skew in the sample towards larger workplaces. In other words, the weights give greater ‘weight’ to smaller establishments that are under-represented in the survey. A natural question that arises from this is, if we control for workplace size in the regression models, will the weights still make a difference. And, if weights no longer make a difference, is it legitimate to use unweighted data and to profit from the smaller standard errors?

To test this we can run the same logistic regression model as we did above but now adding a ‘workplace size’ provided as a set of six groups as an extra predictor. This gives the following co-efficients.

Comparing the two columns from this table we see, again, that weighting has changed the estimates. The coefficient for the group 'more than 3% disabled', which changes from positive to negative after weighting the data. But neither coefficient is significantly different to zero, so this is probably less important than the fact that the weighted estimates for ‘female’ and ‘minority ethnic’ increase substantially after applying the weights. Notice also the larger standard errors associated with the weighted estimates.

This may appear to contradict the comments on the effect of weighting on regressions (theory section section 4.8). In this section we claimed that weighting should not be necessary if a model was fully specified, so that the residuals no longer correlated with the weights. Here we have included the major factor affecting the weights (size of workplace) in the model. So we might expect that this would make the residuals uncorrelated with size of workplace, and hence with the weights. There are two reasons why this conclusion might not follow here:-

• Our measure of workplace size is not identical to that used in the design. Some workplaces were found to have a different number of employees than was recorded on the sampling frame. Also, to protect the identity of respondents, our data set does not differntiate between the largest sub-groups used in the startification.
• We are fitting a non-linear logistic model.

The second of these is probably the more important here. The model is very highly non-linear, as we illustrate in Figure 1. This shows the fitted values for a further model that dropped out the disabled variable and fitted the percentage of female employees as a category.

The figure shows the fitted values by workplace size for workplaces with different proportions of female employees. We can see that there are substantial differences between the effects of the gender of the workforce between the weighted and unweighted analysis. There are several places with fitted values close to 100% where the residuals are not symmetric. The guideline about a fully specified model only applies when the model is a correctly specified linear model.

In the analyses presented above we have been ignoring the finite population correction (FPC) (see theory section 8.1). The sampling fraction in some of the strata in this survey is quite large. The highest sampling fraction is 58% . So if we want to make inferences for the workplaces existing in the UK in 1998, we should consider the effect of calculating standard errors with allowance for the FPC.

An example of this might be the calculation of the percentage of workplaces with an equal opportunities policy. We might wish to estimate this in order to know how many firms might need to be targeted in order to improve this situation. Applying the FPC to these estimates we obtain the figures below for the estimates and their standard errors.

We can see from the table above that the FPC makes very little difference to the standard errors. As expected the effect is greatest in the large workplaces that are more likely to be in the strata with the larger sampling fractions (see theory section 9.3). But even in these strata the effect is very small.

The design effects calculated by Stata for this exemplar (see Stata output last analyses carried out) illustrate the different ways that a design effect can be calculated for subgroups. This is discussed in the theory section on subgroups.

This was a survey of workplaces taken from a business register. Thus there was no clustering by the addresses of the workplaces. The whole survey incorporated a cross-sectional survey of employees, a longitudinal survey and a survey of the employees within the workplaces. A technical report describes the design in detail.

The sample was selected from strata formed by classifying workplaces by industry and by the estimated numbers in the workforce (as recorded on the business register). When the final sample was asembled a few strata had obtained results from only one workplace. This could have caused problems at analysis and so these were polled with other similar strata. This resulted in 71 strata .