Study with the several resources on Docsity
Earn points by helping other students or get them with a premium plan
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Insights into best practices for conducting business tendency surveys, focusing on sample design in manufacturing. It covers sources of frame lists, sampling methods, response rates, missing data, and quality assurance frameworks in EU15 countries. The report includes figures and tables to illustrate key concepts.
Typology: Exercises
1 / 62
This page cannot be seen from the preview
Don't miss anything!
Study on
November 2004
Laurent Donzé, Richard Etter, Nora Sydow, Oliver Zellweger
Swiss Federal Institute of Technology (ETH) Swiss Institute for Business Cycle Research (KOF) ETH Zentrum WEH CH-8092 Zürich, Switzerland
Study on Sample Design for Industry Surveys (ECFIN/2003/A3-03) – Final Report ii
Study on Sample Design for Industry Surveys (ECFIN/2003/A3-03) – Final Report iii
table 1: Synoptic summary ......................................................................................... 4 table 2: Some key statistics ...................................................................................... 23 table 3: Overview, selected countries ....................................................................... 29
But for the aggregation of the data to results for the European Union as a whole, the requirements are higher. For higher comparability of the results, questions, sampling procedures, weighting schemes and imputation techniques should be harmonised.
Until now, the sample design of the BTS in the EU differs from country to country. The decentralised mode of sample design has the advantage of allowing institutes to take, in a best possible way, country-specific conditions into account. On the other hand, a multitude of sampling schemes can affect transparency and the international comparability. Moreover, it could lead to quality variations across countries as different sampling and non-sampling errors may affect the survey results.
The purpose of this study is to make general recommendations concerning “best practices” in conducting business tendency surveys; it is focussed on the issue of sample design for surveys in manufacturing. In the frame of the joint EU programme of business and consumer surveys sample design is an area of discretion for the individual collaborating institutes. Institutes were, however, asked to give adequate information about various elements of the sample design.
The paper is structured in four parts. Section 2 compares the different elements of the national survey designs collected by a questionnaire. The questions were divided into several subgroups: Questions on the frame list, the sampling methods and the weighting procedures were key elements. Information on the response rate, handling of missing data and the quality assurance framework completed the questionnaire. Section 3 gives some hints to the design effects in BTS data of the different countries. In section 4, six countries are selected to analyse their sample design in- depth. Individual questionnaires were adapted to the country-specific situation. An individual description and assessment is given. The study finishes with a technical report including recommendations.
It is obvious that a harmonised sample design within the EU countries would improve the comparability of the country results. Moreover, the aggregation of the harmonised survey results of the countries to total EU increases the transparency of the underlying aggregation process and thus helps to interpret the EU results accurately. Up to now a contrasting survey of the sample design of the BTS conducting institutions has been missing. This report is intended to fill this gap.
Relevant information was collected by a postal survey. The questionnaire and a covering letter were sent to the managing directors of the institutes of the 15 countries that made up the EU before enlargement (EU15) and Poland, and also to the heads of the unit in charge of conducting the industrial trend surveys. By the end of the deadline for submissions, we received fifteen questionnaires, the remaining institute was contacted in a follow-up by a reminding letter and a reminding e-mail, without success. In general, the responding institutes were collaborating and ready to detail their answers.
The questions were asked in a general manner and respondents were given the chance to specify their answers. The main purpose of our survey was to get a picture of the common practice applied in the different EU countries; a overview of answers is shown in table 1.
In the following subsections the different parts of the questionnaire and the results are presented using some figures and comments. The sequence of the ensuing subsections are similar to the structure of the questionnaire. After a short introduction to each topic an evaluation of the answers is provided. A copy of our questionnaire can be found in the appendix of this report.
BTS are conducted among firms. Demographic structure variables, addresses and economic activity of a firm have to be known to build up a sample framework. But where does the institutes get their data from? A government register, at its best, collects data about each existing firm in a country and displays the complete survey population (the manufacturing sector, in our case). As it will be shown later, not all institutes are using this kind of register data as the sample frame.
As can be seen in figure 1, the majority of the survey conducting institutes uses information from an administrative source such as business registers or any other type of governmental register. 80% (12) do not cover the whole manufacturing industry mainly because of cut-off limits according to some size measures (see figure 2). 2
figure 1: sources of information entering the frame list
0 2 4 6 8 10
other
company lists
own sources of information
government register
business register
number of respondents
source: question 1.1; several answers possible
(^2) Please note: the total number of institutes is mentioned in brackets.
figure 2: units missing in the frame list
0 2 4 6 8 10
other
specific regions
specific size categories
specific branches
number of respondents
source: question 1.3; several answers possible
Taking the reported data on the actual sample size and the population size (figure 3) in the manufacturing survey leads directly to the sampling fraction.
figure 3: sample size and survey population by country
FR^ IT
ES
PT UK
NL
DE
BEAT
SE DK
PL
0 LU
500
1000
1500
2000
2500
3000
3500
4000
4500
0 50000 100000 150000 200000 250000 300000 survey population
sample size
source: question 1.2 and 3.6; country codes: BE Belgium, DK Denmark, DE Germany, ES Spain, FR France, IT Italy, LU Luxembourg, NL Netherlands, AT Austria, PL Poland, PT Portugal, SE Sweden, UK United Kingdom. Data for both Finland and Greece are missing.
It is tempting to speculate that the sampling fraction is inversely related to the size of the country. This proposition becomes intuitively clear as the sample size depends mainly on precision considerations of estimates and not directly on the population size. Therefore the sampling fraction is determined to a great deal by the sample
A sample is used when surveying the whole population would be too laborious and/or expensive. A sample is drawn in order to represent the structure of the population in a best way. To keep costs low the sample is kept as small as possible. Under certain conditions, statistical inference allows the formation of conclusions about the parameters from a sample taken from a larger population. In BTS a selected sample is kept for a longer period and not redrawn monthly and thus resembles a panel. On the one hand, this constancy will lead to an increasing response rate, but on the other hand, the constant sample may drift away from the true structure of the industrial sector.
To cope with the heterogeneity of the survey population, most of the institutes use stratified sampling with simple random sampling. The answers given to question 2. are summarised in figure 5. Among the answer category “other” one institute reported applying quota stratified sampling, the second one voluntary stratified sampling with some exhaustive strata. The stratification criteria are size and branch (5), gross value added (1), branch (1) and turnover (1). The remaining five institutes ticking “other” are split in those surveying all units on the frame list, the largest companies and one contacts business organisations. The last two did not specify their sampling method.
figure 5: sampling methods
0 1 2 3 4 5 6 7 8
other
cluster sampling with simple random sampling
sampling with probability proportional to size
stratified sampling with simple random sampling
simple random sampling
number of respondents
source: question 2.1; single answer
Only one of the institutes uses some sort of rotation scheme in order to reduce the reporting burden; but some of the remaining have already taken this into account by choosing a cut-off for specific size categories.
Figure 6 shows that on a whole all key factors except sampling fraction are important when deciding on sample size. The results also point to the implicit trade-off between maximising the precision of estimates and the underlying costs. Most countries reporting response rates above 75% rated the precision of estimates very highly. There is an inverse relation between budget constraints and the response rate of a survey. This result supports the intuitive fact that more financial resources may improve the quality of a survey at least to some extent.
figure 6: key factors for sample size
0 1 2 3 4 5
nonresponse
sampling fraction
budget constraints
variability in population
precision of estimates
average importance
source: question 2.4; several answers possible
The panel character of BTS data leads to a higher response rate compared to a recurring survey conducted with a sample drawn for each survey wave. Nevertheless the response rate has to be kept continually on a high level to assure comparability over time.
In principle, one could think of making participation compulsory to increase survey return. Firms in Luxembourg, France, Portugal, Italy, the Netherlands and Spain are in fact required to take part in the survey on manufacturing by legal obligation. Yet response rates in these countries were not affected markedly by this legal requirement, that is, comparing response rates with countries without legal obligation did not show any substantial differences. According to our questionnaire, to be fair, we have no information on the type of imposed sanctions or other measures to enforce regulation. However, a compulsory survey is likely to affect the quality in the
One country conducts its survey with a mixed mode of interview/e-mail. The interviews are hold by phone while the results of the previous survey are handed out by e-mail.
The harmonised EU programme intends to ensure the comparability of the answers given to the BTS. In addition to these questions the institutes are free to ask extra questions. 87% (13) use this opportunity and add further questions to their surveys regularly. On average there are 12 supplementary questions per country and year with different periodicities.
figure 8: adds on in postal survey
0 2 4 6 8 10 12
other
results of previous survey
return envelope
covering letter
number of respondents
source: question 3.3; several answers possible
Every survey inevitably encounters the problem of imposing response burden in terms of time and effort for addressees. Besides others factors, the burden increases steadily with the number of questions. Nonetheless, countries in our survey asking additional questions did not show a deteriorated number of responses. Response rates in these countries exceeded the ones in the countries asking exclusively questions in line with the joint harmonised EU programme.
Hence participating may bring about a broad range of tangible and intangible benefits for the sampled unit stemming from the reputation of the surveying institute, pecuniary incentives or even civic virtue which surpass the respondent’s cost of answering.
The mean sample size contains 1,909 units (1 st^ quartile = 1,000; median = 1,700; third quartile = 3,000) and the institutes are aiming for an average response rate of 77%.
One of the most efficient measures to increase the response rate of the running wave are follow-up activities. This measure is applied by almost all responding institutes (14).
figure 9: follow-up mode
0 2 4 6 8 10 12 14 16
other
postal mail
telephone
number of respondents
source: question 3.8; several answers possible
The different modes used for these activities are shown in figure 9. The category “other” has been specified in the following manner: four institutes are using fax, one of them visits the companies having given erratic replies. All institutes use at least telephone mode to get into contact with nonrespondents.
Altering the survey mode when contacting a nonresponding sample unit is likely to be constructive. According to our survey, the majority of the institutes which contact the units by a different mode in the follow-up phase exhibited considerable increases of response rates. Obviously a change in the mode increases the attention of the surveyed units. Moreover, the result could also be interpreted to suggest that it is desirable to provide a set of modes from which firms may chose periodically in order to ascertain that they are able to answer the questions in the most suitable fashion.
All institutes are assuring confidentiality on their questionnaire. Confidentiality refers to the safeguarding of any information about one party that is known by another. As complete anonymity is virtually impossible, confidentiality is a real concern.
Rewarding participation should affect response rates positively. Measuring the immediate effect and size of the reward scheme, however, is tricky. The uncomfortable conclusion from our survey is that by simply comparing the mean response rate of countries with and without reward schemes we are unable to give evidence on the proposition. Response rates here are also affected by many other factors. Hence our results do not challenge the significance of reward schemes.
In order to set up time series it is important to have a constant answering behaviour for all companies. Both bias and variance of survey estimates may be affected whenever a surveyed unit does not answer at all (unit nonresponse) or deliberately omits some questions (item nonresponse). In addition, a surveyed unit may decide not to answer either regularly or occasionally. In any case, the behaviour should be monitored as continually refusing to answer a question may contain some information. Furthermore inconsistent as well as illegible data is to be removed and has to be treated as missing data.
87% (13) check for item nonresponse, when questionnaires return. 73% (11) of the institutes verify the consistency of the values, whereas all check for invalid data in quantitative replies.
All organisations are confronted with the fact that some units do not answer in (at least) a single wave. Nevertheless, 60% (9) consider this problem in their evaluation. Three of them adjust weights, four imput the missing values with data of a similar unit and the remaining two institutes use a combination of these techniques. Six out of nine institutes can resort to their history of nonresponse.
53% (8) cope with regular nonresponse by replacing the unit with a unit of the frame list with similar characteristics.
47% (7) of the institutes deal with item nonresponse. Three organisations adjust their weights, whereas three impute missing values. The follow-up activity of one institute is such intense that there is no item nonresponse at all.
Sampling weights may improve the precision of estimates compared to a simple random sample under otherwise equal conditions. The Horwitz-Thompson estimator, for example, uses the inverses of the selection probabilities of the sampled units as sampling weights. If auxiliary information is obtainable which is related to the study variables at least to some degree, it can be used to further reduce sampling variance of the estimator. At the same time, auxiliary information can also be useful to tackle the problem of nonresponse and its potentially negative impact on bias and sampling variance of estimates. Sample-based (see question 3.14) and population-based weighting are common practice.
The use of auxiliary information for weighting is prevalent in the participating organisations. The institutes have reported utilising auxiliary information for weighting in some form, except one. Figure 11 shows the most frequently ticked options. Replies suggest that firm size comprises turnover data and/or employment size groups. A third of the organisations use firms’ data on branch, sector and size for weighting.
figure 11: auxiliary information for weighting
0 2 4 6 8 10 12 14
other
branch
sector
firm size
number of respondents
source: question 5.1; several answers possible
In question 5.1 the institutes were asked to give information on their method of weighting when aggregating. At the first step, each reply is usually weighted in relation to the proportion of the firm’s industry/employment size or turnover data. In subsequent steps, the aggregation process is mainly based on total value added or structural data of the economy. A more detailed examination of the aggregation is explored for a number of selected countries in section 4.
