Lessons Learned: Scale Variations and their Effects on Survey Data

Writing a good questionnaire requires thoughtful consideration regarding what questions need to be asked, and exactly how to ask them. Many times, there are precedents set by previous questionnaires, as well as size limitations that can dictate the content and form of the questions.

Because of this, a potentially troubling issue arises: Would your data change if you asked the same question in a different way? Ipsos Insight wanted to determine how much influence certain question formats have on survey results. We were particularly interested in the answers to questions that used some common form of a Five Point Interval 'Agreement' Scale.

We decided to use Ipsos Insta-Vue, our mail omnibus, to administer four versions of a questionnaire, each containing different iterations of the same five-point scale. Each version, which contained questions about dishwashing liquids, went to 2,500 households. There was no need for sample balancing because there were no significant differences in demographic profiles between the cells, and the return rates were all within 5% of each other.

The following question was used to determine whether the format of a scale has an effect on which answer a respondent gives:

Indicate below how much you agree or disagree with the following statement: "Generally speaking, dishwashing liquids are safe for the environment."

The accompanying scale took the following four forms:

Findings

  • When only the ends of the scale are defined – either by 'Anchoring' or by indicating what "1" and "5" mean (Versions B & C) – respondents are significantly more likely to use the 'Top Box" than the "2nd Box" (Versions A&D).
  • The "Fill-In #" (Version C) obtains the lowest ratings and the highest "No Answers".
  • The horizontal format of the fully annotated scale (Version D) led respondents to choose the center point more often than did the vertical format (version A).

Conclusions

A thorough examination of the responses showed that our suspicions were justified, and the form of the question did play a role in response data. But the effects vary and understanding how is an important step in developing an analytically relevant study. Different scale formats elicit different responses to the same question. Therefore, a few simple rules apply:

  • Standardize question format whenever possible.
  • "Comparative Analysis" (same scale/format between cells or to a norm) is more meaningful than "Absolute Analysis" (x% Top Box score).
  • Be aware of the effect of various scales/formats and use them appropriately