Weighting Survey Results: Why Do It?
By: Scott Young, Ph.D., Principal, syoung@valtera.com
When administering an organizational survey, the ultimate goal is to understand the attitudes of the entire organization. However, participation rates rarely approach 100%. As a result, an incomplete or inaccurate picture of the larger group or organization may emerge when only a sample of the organization completes the survey. Item weighting is one method of addressing this potential problem.
Weighting for data analysis and reporting may be appropriate whenever the proportions of survey respondents for your groups of interest (e.g. departments, regions, or functions) do not match their proportions in the overall populations of interest (i.e., all employees in the departments, regions, or functions being surveyed).
There are two primary reasons why the proportion of respondents might differ from proportions in the overall population.
1) Sampling techniques may require different proportions from certain groups. When data are to be reported for different groups (e.g., job levels, departments, etc.), it is important to ensure that these group-level results can be reported with an acceptable level of precision. Higher participation rates are needed in smaller groups than larger groups to obtain a given level of precision. Thus, when the survey is being sent only to a sample of an organization, smaller groups will often be over-sampled. While this helps ensure that those smaller groups can receive a report, it also results in those groups being over-represented in the report that combines all respondents.
2) Different groups may have different return rates. Even in the case where equal percentages of employees from each group are invited to participate (e.g., a census survey), actual group response rates may differ for many reasons. For example, one department may have an unusually high response rate because its management communicates strong support for the survey process. In addition, survey administration methods may differ between departments, with some methods yielding greater participation than others.
Weighting helps to compensate for some groups being over- or under-represented in the overall results by giving greater weight to the response of an individual from an under-represented group than a response from an individual from an over-represented group.
So why don’t we always apply weights to survey data? At first glance it appears that data should be weighted whenever the sample proportions do not perfectly represent the organization as a whole. However, there are some circumstances where weighting is not advisable. For example, when return rates are highly disproportionate and very low for certain groups, weighting the data can result in a severe reduction in the precision of the final results. This is because results for extremely under-represented groups are the most unreliable, and therefore giving more weight to data that is the least reliable results in lower precision in the overall results. As a general rule, it is best to apply weights to a group only when the sample from the group is large enough to ensure an accurate representation of the group’s opinions.
In addition, weighting will only have a meaningful impact on results and conclusions if there are significant differences in the results of the over- and under-represented groups. Given that weighting can result in some confusion among consumers of the data, it’s best to use weighting when there are large differences in the representation and the results of different groups.
For more on weighting survey results, follow next week's blogs and download this whitepaper:
