18/11/2015
1
18/11/2015 Questionnaire Design - Dr Anke Görzig 1
Questionnaire Design
Overview
General quality criteria of scales
Prepare and select “good” items
Reverse items
Item difficulty (spread)
Inter-item correlation
Determine the quality of your scale
Prepare a presentation of your scale
(Presentation this afternoon)
18/11/2015 Questionnaire Design - Dr Anke Görzig 2
18/11/2015 Questionnaire Design - Dr Anke Görzig 3
Quality criteria of scales
Reliable: measures the quantity or concept in a
consistent or reproducible manner
Valid: measures the quantity or concept that is
supposed to be measured
Unbiased/objective: measures the quantity or concept
in a way that does not systematically under- or
overestimate the true value
Discriminating: can distinguish adequately between
respondents for whom the underlying level of the
quantity or concept is different
Quality criteria of scales: Item properties
18/11/2015 Questionnaire Design - Dr Anke Görzig 4
Quality of your scale: Item properties
Recode reversed items
Transform
“Recode into different variable”
Select reversed items
Add new name
-> “Change”
“Old and new values”
18/11/2015 Questionnaire Design - Dr Anke Görzig 5
Quality of your scale: Item properties
Item difficulty
Proportion of “agreement” to the item across respondents
Average response / number of scale points - 1
SPSS: Data -> Transpose…
Select all (recoded) variables from left into right box
! Do not select the external item !
Transform -> compute variables
Type:
MEAN (var001 thru var020)/
(5-1)
Delete all variables except
difficulty and CASE_LBL
Note items for which difficulty
< .2 or > .8
Eliminate from scale
18/11/2015 Questionnaire Design - Dr Anke Görzig 6
18/11/2015
2
Quality of your scale: Item properties
Inter-item correlation
File -> Open ->
Analyse -> Scale -> Reliability analyses
Select all items of your scale but
Use recoded and not original ones
Exclude items with “bad” difficulties
Look at output file
Note items in “corrected inter-item correlation” column when < 0.3
Eliminate from scale
18/11/2015 Questionnaire Design - Dr Anke Görzig 7
Quality criteria of scales
Reliability
Does the scale measure
anything at all?
Consistency of a measure
Do we get the same result
repeatedly?
Validity
Does the test measure what
we want it to measure?
18/11/2015 Questionnaire Design - Dr Anke Görzig 8
Quality of your scale: Reliability
Inter-item correlation
File -> Open ->
Analyse -> Scale -> Reliability analyses
Select all items of your scale but
Use recoded and not original ones
Exclude items with “bad” difficulties
Exclude when “corrected inter-item correlation” < 0.3
Look at output file
Cronbach’s alpha: range 0 to 1
Insufficient < .5
Satisfactory < .7
Good > .7
18/11/2015 Questionnaire Design - Dr Anke Görzig 9
Quality of your scale: Validity
Correlation with external criteria
Transform -> Compute Variable
Target Variable: name of your scale (e.g. “happiness”)
Numeric Expression: type Mean (..)
Select all items which had good properties, i.e.
difficulties, inter-item correlation
18/11/2015 Questionnaire Design - Dr Anke Görzig 10
Quality of your scale: Validity
Correlation with external criteria
Analyse -> Correlate -> Bivariate
Select scale name and external item
Correlation should be fairly low (<.3) and not statistically
significant
18/11/2015 Questionnaire Design - Dr Anke Görzig 11
Test your research hypothesis
Overall mean
Analyse -> descriptive statistics -> descriptives
Mean by your chosen grouping variable
Analyse -> compare means -> independent sample t-test
Try custom graphs and tables
18/11/2015 Questionnaire Design - Dr Anke Görzig 12
18/11/2015
3
Report on your scales
What was your concept?
What was our research hypothesis?
Which items had to be eliminated and why?
Give some examples
Reliability (alpha), discriminant validity (r)
Overall mean
Mean by your chosen grouping variable
Results of your research (t-test)
Discussion
What went well and what not so well, and why
18/11/2015 Questionnaire Design - Dr Anke Görzig 13