Social Research Today
Series editor Martin Bulmer
1. Analyzing tabular data Nigel Gilbert
2. Doing environmental field research Jacquie Burgess
3. Measuring health and medical outcomes Edited by Crispin Jenkinson
4. Researching the powerful in education Edited by Geoffrey Walford
5. Surveys in social research D. A. de Vaus
7. Information technology for the social scientist Edited by Raymond Lee
8. Feminist social research Mary Maynard
9. Introduction to the philosophy of social research Malcolm Williams & Tim May
Surveys in Social Research
Fourth edition D. A. de Vaus
Australian Institute of Family Studies, Melbourne
UCL
PRESS
1 The nature of surveys
Surveys are a method of social research which, like any other method, has its advocates and critics. The primary aim of this book is to provide guidance on how to do good surveys rather than to provide a detailed defence of surveys. However, it is important to realise that many criticisms of surveys are based on misunderstandings of what surveys can be.
The outline of how to do good surveys takes into account some of these criticisms and shows that there are ways of dealing with them. The solution to criticisms of a method need not be to abandon the method but first to see if it can be improved. The focus of this book then is to show what can be achieved with surveys and how to do it.
In this chapter I shall clarify what survey research is and then briefly outline some of the common criticisms. In my definition of survey research and the following discussion I rely partly on an excellent book by Catherine Marsh (1982) titled The Survey Method: the contribution of surveys to sociological explanation.
What is a survey?
Marsh insists that a survey is not synonymous with a particular technique of collecting information: questionnaires are widely used but other techniques such as structured and in-depth interviews, observation, content analysis and so forth are also appropriate. The distinguishing features of surveys are the form of data collection and the method of analysis.
Form of data collection
Surveys are characterised by a structured or systematic set of data which I will call a variable by case data matrix. All it means is that we collect information about the same variables or characteristics from at least two (normally far more) cases and end up with a data matrix (see Table 1.1).
The nature of surveys 5
In other words, for each case we obtain its attribute on each variable. Put together we end up with a structured or 'rectangular' set of data. However, the technique by which we generate the data need not be highly structured so long as we obtain each case's attribute on each variable. Because questionnaires are the easiest way of ensuring this structured data matrix they are the most common technique used in survey research. But there is no necessary connection.
In the example in Table 1.1 each case was a person but this need not be so. A case (called a unit of analysis) could be a country, a year or virtually anything so long as we collect attributes of that case (see section on units of analysis in Chapter 3). If countries were the cases, a list of countries would be across the top of the table instead of people, and attributes of countries (e.g. population size, area, density, unemployment rate) would be listed down the side. If years were the cases, years (e.g. 1950, 1960, 1970, 1980) would be listed across the top with attributes relevant to years down the side (e.g. inflation rate, divorce rate).
The variable by case matrix is fundamental for survey analysis which is based on comparison of cases. It is this method of analysing data which is the second distinguishing feature of surveys.
Methods of analysis
One function of survey analysis is to describe the characteristics of a set of cases. Thus if we want to describe how a group of people will vote, we need to know how each person in that group intends to vote. A variable by case matrix provides this information.
But survey researchers are also interested in causes of phenomena. The survey analyst tries to locate causes by comparing cases. By looking at how cases vary on some characteristic (e.g. some cases will be progressives and others will be traditionalists), the survey analyst will see if the progressives are systematically different from the traditionalists in some additional way. For example, in Table 1.1 there is variation across cases in how they vote. This is systematically linked to variations in class: the progressives are working class and the traditionalists are middle class. In other words, survey research seeks an understanding of what causes some phenomenon (e.g. vote) by looking at variation in that variable across cases, and looking for other characteristics which are systematically linked with it. As such it aims to draw causal inferences (e.g. class affects vote) by a careful comparison of the various characteristics of cases. It does not end there. The next step is to ask why class affects vote. Survey researchers need to be very careful, however, to avoid mistaken attribution of causal links (simply to demonstrate that two things go
6     Surveys in social research
The nature of surveys 7
together does not prove a causal link).
This style of research and analysis can be contrasted with other methods. For example, the case study method involves data collection about one case. Since there are no other cases for comparison quite different strategies for understanding the behaviour and attitudes of that case have to be employed. The experimental method is similar to the survey method in that data are collected in the variable by case matrix form, but is fundamentally different in that the variation between the attributes of people is created by intervention from an experimenter. Some medical research serves as an example. An experimenter wanting to see if a drug cures a disease would take a group of sufferers and divide them into two similar groups. The drug would be administered to only one group and then the recovery rates of the drug and non-drug groups would be compared. Here the variation between the two groups (i.e. drug/non-drug) has been created by the experimenter. A survey approach would not create the variation but would find 'naturally occurring' variation—that is, find a group of sufferers who did have the drug and compare them with a group of sufferers who did not have the drug. The problem for survey researchers is that they cannot be sure
that the two groups are similar in other respects, whereas the experimenter begins with two similar groups and the only difference (in theory) is that only one group receives the treatment. Therefore any difference in recovery rates must be due to the drug. Apart from the potential ethical problems of experimental research, these different approaches to obtaining variation between groups lead to quite different methods of analysis.
In any particular study a range of methods can be used. For example, a study of causes of strikes could involve a survey of attitudes of management and workers, a case study of a particular strike or a particular factory and an experiment where groups of workers work under different conditions to see if this affects their strike frequency.
The techniques by which data are collected using any of these methods can vary considerably. In a survey we could observe each case, interview them, give them a questionnaire and so on.
In summary, survey research is one method of collecting, organising and analysing data. The relevant data can be collected by a variety of techniques and in many studies it may be appropriate to use a variety of research methods (see Figure 1.1).
Figure 1.1   A range of methods of research and techniques of data collection
Criticisms of surveys
Case study
Questionnaire Interview (structured) In-depth interview Observation Content analysis
Research question
Survey
Questionnaire Interview (structured) In-depth interview Observation Content analysis
Experiment
Questionnaire Interview (structured) In-depth interview Observation Content analysis
The most common criticisms of surveys can be classified into three categories: philosophical, technique based and political. These will simply be mentioned here, but will be taken up again in the final chapter.
Philosophically based criticisms
1 Surveys cannot adequately establish causal connections between variables. For example, even though older people are more conservative than younger people we cannot be certain that growing older causes conservatism. We shall see to what extent survey analysis can overcome this type of objection.
2 Surveys are incapable of getting at the meaningful aspects of social action. Because actions are the actions of conscious people who make choices, have memories, wills, goals and values which motivate behaviour, research must take these into account when developing and evaluating why people behave and think as they do. We shall see that survey research can go a long way towards arriving at such 'meaningful' explanations.
3 Surveys just look at particular aspects of people's beliefs and
8     Surveys in social research
The nature of surveys 9
actions without looking at the context in which they occur. Taken out of context it is easy to misunderstand the meaning of behaviour. For example, if a person goes to church regularly it may mean they are highly religious, but it could mean they are searching for a religious faith, or they cannot avoid going because of social pressure, or they go because of important contacts at church and so on. We shall see that with care survey research need not lead to contextless pieces of information and the consequent misunderstanding of that information. (The problem will always remain of how much and which information is needed about context to avoid misunderstanding.)
4 Surveys seem to assume that human action is determined by external forces and neglect the role of human consciousness, goals, intentions and values as important sources of action. We shall see to what extent this accusation of determinism is a necessary part of surveys or whether it is simply because surveys have been used more by positivist sociologists than by others. (It is worth noting that Marx attempted a survey involving 25000 posted questionnaires and that Weber also used a survey in his study of factories.)
5 Survey research is equated with a sterile, ritualistic and rigid model of science centred around hypothesis testing and significance tests, which involves no imagination or creative thinking. By showing how to do survey research we shall see just how wrong this view is.
6 Survey research is basically empiricist (C.W. Mills). That is, it merely collects a mass of facts and statistics and provides nothing of theoretical value. We shall see throughout this book that theory and interpretation is fundamental to well-conceived survey research and analysis.
7 Some things are not measurable—especially by surveys. For example, a survey researcher would probably have great difficulty in actually measuring the extent to which Rupert Murdoch has power. No claim is made in this book that surveys are the method of social research. Surveys should only be used where they are appropriate and other methods should be used when they are more appropriate.
Technique-based criticisms
8 Surveys are too restricted because they rely on highly structured questionnaires which are necessarily limited. This criticism is based on too narrow an understanding of what techniques can be used in surveys.
9 Surveys are too statistical and reduce interesting questions to totally incomprehensible numbers. While many studies are unnecessarily statistical and sterile, the logic involved in these statistical analyses is important and the same logic is widely used in both statistical analyses and much more qualitative analyses. It is this logic and the role of creative thinking that will be emphasised in this book. Statistics should be the servant rather than the master of the survey analyst.
'Political' criticisms
10 Survey research is intrinsically manipulative and is described by the Frankfurt Marxists as 'scientistic' and 'technistic'. It is seen to be manipulative in two ways. First, the knowledge it provides about the social world gives power to those in control and this can lead to an abuse of power. Second, survey research leads to ideological manipulation. It does not produce knowledge about reality but is an ideological reflection whose acceptance by 'the public' furthers particular interests. (See Marsh 1982 for a full discussion of this criticism of surveys.)
Practice vs ideal types
A basic difficulty when trying to describe how to do research is the gap between textbook accounts of how research should be done and how it actually is done. A number of valuable books have now been published in which some researchers 'come clean' and provide accounts of how they did their research (Hammond, 1964; Bell and Newby, 1977; Bell and Encel, 1978). Like my own experience, theirs does not conform to the textbook models.
What ought to be done in a book like this? To describe an 'ideal-typical' model of survey research, in which each step of research is outlined, is not to describe what researchers do. As such it can mislead. When you actually do some research you will find that you are not doing what you 'should'. So should the book describe the reality? Perhaps. But which one? The course that a piece of research actually takes will be peculiar to that piece of research: it is affected by the research topic, the technique of data collection, the experience and personality of the researcher, the 'politics of the research', the types of people or situation being studied, funding and so on. I could describe my experiences but like an ideal-typical model they would not reflect other people's.
I have decided to do a bit of both. I will outline the key steps which
10     Surveys in social research
a survey researcher must take at some stage and describe the reasoning behind the order in which it is normally suggested they be taken. But I will also point out that in practice some steps are omitted, things are done out of order and we move backwards and forwards between steps. Guidelines that are provided are not meant to be prescriptive. The guidelines I describe are like signposts or a map to provide some direction and give us clues as to where to go when we get lost. As you become more familiar with the territory you can manage more easily without the map and learn short cuts. What I describe will not always reflect your research experience but will provide guidance. You should not try to follow each step slavishly. The prime goal of research should be to gain accurate understanding and as a researcher use methods and techniques which enhance understanding. Use the method: do not let it use you.
Further reading
Catherine Marsh's book The Survey Method (1982) is the best description of the survey method available. Her outline and evaluation of the most substantial criticisms of surveys is direct, clear and stimulating. Denzin's The Research Art (1978) provides a critique of survey research from a symbolic interactionist perspective as does Btumer's paper 'Sociological analysis and the variable' (1956). Chapter 3 of the Sociological Imagination (1959) by C.W. Mills on abstracted empiricism is a well known attack on certain forms of survey research.
2 Theory and social research
We can conduct research and show that in the last 100 years social mobility has increased, religion declined, the structure of the family changed and values have been transformed. These are important changes to describe but why have they occurred? We might show which types of people are most mobile or are least religious, and we can document the character of modern families and describe who holds what values. But why are some mobile while others are not, why are some less religious than others? We know that a large number of people live on or below the poverty line. But why?
Social researchers can try to answer two fundamental questions about society. What is going on (descriptive research) and why is it going on (explanatory research). I believe that the central role of social research is to try to answer both the 'why' and the 'what' questions. The aim is both to describe and understand society. : Sociological theories are attempts to answer these sorts of 'why' questions. They are attempts to explain, and as such the role of sociology is to theorise: it is not just social arithmetic.
The interaction of theory and research
Observations require explanation but equally explanations need to be tested against the facts. It is not enough simply to collect facts. Nor is it sufficient simply to develop explanations without testing them against facts. Fundamentally sociological research involves a constant interplay between observation and explanation, collection of further facts to test the explanation, a refinement of the explanation and so on.
The development of good explanations involves two related processes: theory construction and theory testing. These two processes are not alternative ways of arriving at good theories but represent two stages with different starting points (see Figure 2.1).
11