Work Package 2: General Principles of the Questionnaire’s Design

The design of this project has been guided by several research traditions, examples, and logistical
considerations.

RESEARCH TRADITIONS: First, as we said in our Invitation, in a broad sense this research is rooted
in the cultural studies tradition.  It aims to address issues of pleasure, of meaning, of identity,
and of community.  And it sees media (and in this case film) engagement as deeply interwoven into
people’s everyday lives.  Of course, tapping into the complexities of these with a questionnaire is
always going to be tricky.

But cultural studies has had a weak tradition of audience research, tending instead to create
images of the audience out of textual interrogations.  So, alongside that general cultural studies
impulse lies a strong admiration for a broader European reception studies tradition.  Here,
attempts at using complex research methods have been more common.

Our research design owes much to the recent growth of interest in combining quantitative and
qualitative methods.  Quite a number of studies have attempted this in recent years.  But few of
these have done it in the way we do here: combining the two approaches in a single research
implement, which allows the two kinds of results to be explored directly in relation to each other.

EXAMPLES: Immediately at the back of the project is the 2003-4 study of the audiences for The Lord
of the Rings.  In many ways that was an enormous success – attracting 25,000 completed responses,
and enabling us to explore many kinds of question.  But looking back, we have identified some
weaknesses in its design.  We have tried to retain its most successful elements.  But we want to
correct some mistakes.  And we have some new ambitions, which arise from things that emerged by
accident from that project.

LOGISTICS: In devising our research implement, we have had to think about a lot of further issues:

1)      DEMOGRAPHICS: Trying to attract responses from all parts of the world poses many
challenges.  If for instance we want to gather demographic information across the world, we have to
find ways that are as culture-neutral as we can manage.  How do we ask a question about educational
achievement that will make sense to everyone across the world?  How do we do this for
income/wealth/class?  We believe that we need to try, but we need to share ideas on the best ways
to do these things.

2)      LANGUAGES: Translation is going to be critically important.  But translation is never going
to be a purely matter of technical accuracy.  We have to watch carefully for any cultural
assumptions that we import from our own backgrounds.  Some concepts that we want to use may not
easily translate into all languages and cultures.

3)      EXPLANATIONS: We must take great care how we explain the purposes of the project on the
website.  That, and all our publicity, will obviously aim to make the invitation to take part as
interesting and attractive as we can.  But we may need to defray certain fears.  If we want to ask
for particular bits of demographic information – for instance, about income/wealth/class/ethnicity,
or about educational levels, we may need to explain why we want this, and what we will do with this
information.

4)      LENGTH: People are unlikely to be willing to spend a very long time filling in the
questionnaire.  It must not be too long, overall – and people visiting it must be able to see
clearly how long it is likely to take them, and to get a sense that the kinds of questions being
asked make sense to them.

SAMPLING: A central feature of our questionnaire’s design is that it does not require us to seek a
balanced sample.  (We won’t go over in detail again why this is – we will only repeat here that if
we had required such a sample, the entire project would have been prohibitively expensive.)  Of
course our aim will be not only to recruit large numbers, but also a wide spread of kinds of people
– by age, by sex, by class, by knowledge of the books, by response to the films, and so on.  But
providing in general we get good numbers in each category, that will be sufficient for analysis.

CATEGORIES AND PAIRS: The principle is that, as people complete the questionnaire, they put
themselves into a series of categories by which they can be sorted.  The meanings and implications
of those categories are exactly what our project is then designed to interrogate.  And that leads
to the second central feature: key questions are paired, with a link between a quantitative and a
qualitative question.  So, having asked people to indicate their overall feelings about the films
on a Likert-type (1-5) scale, we will ask them to tell us why they have given that evaluation.
That will allow us to investigate from within the database answers from enthusiasts, and from
critics, to see what sorts of elements come most to their respective attentions.  We can then go
further, and see if there are distinct kinds of enthusiasm or criticism, by tightening the
sampling, say, to those who know the book, or by age of respondent, or by which vernacular
categories they choose.  We can follow hunches, we can experiment, we can turn particular examples
into the basis of searches.  The point is that, providing overall the numbers are high, we have an
almost limitless number of questions we can explore.

COMPARISON: We can also of course simply choose one category that particularly interests us, and
build profiles of them.  For example, one national or one language sample can with ease be
separated out, for separate examination.  Or, perhaps, one of us might be interested in the
responses of people who were read the book as children.  We suspect that the temptation will always
be to do this through comparison, seeing how this group varies from the norm at least.   But the
point is that it is then up to us, together and separately, to use whatever particular interests
and particular skills we have to pose questions and to find ways to try to answer them.

PRACTICAL MATTERS: From the general to the particular
 1. We think that the questionnaire really ought to be not much more than 20 questions, overall,
with half of these as multiple-choice.  Then we can honestly say that it should take no more than
20 minutes to complete the entire thing.  That is going to require real discipline of us all.
 2. We think it is best to start with the most general questions – and the ones on which everyone
is likely to have their own answer.  So, asking in deliberately informal words ‘What did you think
of the films?’, and then asks them simply ‘Why did you say this’, should initiate a feeling that
this is a questionnaire where they are the experts.
 3. Particular questions do have different, and sometimes quite special functions.  For example,
although we propose questions asking about both ‘most memorable’ and ‘most disappointing’ aspects
of the films, our interest is especially in the latter.  This is because over a series of projects
we have found that questions about disappointment have a special quality: they can bring into view
what people hoped for and expected, and through that what their ideal film experience would have
been like.  This question can be crucial for disclosing people’s working evaluative criteria.  (Of
course there are things to be learnt from people’s ‘most memorable’ responses, but they may not
have the same diagnostic quality.)
 4. We think that we need to gather information efficiently on which version of the film people
saw.  Was it in English, subtitled, or dubbed?  Was it in 2D or 3D?  We haven’t yet formulated the
best way to do this, but believe that it needs to be done with a single multiple-choice question.
 5. We have assumed that, just like the Lord of the Rings questionnaire, we will make it necessary
for people to answer all the multiple-choice questions (so that responses can be quantitatively
analysed). But all open-ended questions will be optional – people can say nothing or a lot as they
wish.
 6. We have left for separate discussion three particular questions, about ‘vernacular categories’,
about ‘interpretive communities’, and about class/income/wealth.

Questions to our partners:
 1. Are there any other general principles that you think the questionnaire should take into
account?
 2. Do you agree with the general running order we have proposed in the draft questionnaire?
 3. Do you share our feeling that we should aim to limit the questionnaire to around 20 questions,
with a roughly even split between quantitative and qualitative questions?
 4. Are there any other particular questions you would want to propose?
 5. Do you have any strong views about questions in the draft questionnaire?