Bad Survey Design. Please Stop!
A plea to all design educators out there (and to students as well): please stop using crappy surveys as a substitute for actual research.
Survey design is a craft. If you haven’t studied it, you don’t know how to write a survey well, and the data you get is garbage. Surveys are quantitative tools. They require math to plan (what does your sample size need to be to ensure that your results are valid?) and to analyze (regression analysis (or any other buzzword) anyone?). They are very tough to write. Questions have to be worded correctly and sequenced correctly.
Yet design instructors constantly send their students onto the Internet to “do research.” Students spend about 30 seconds writing open-ended questions about their issues, and then blast the “survey” off to email lists populated by other designers. And so in the spirit of helping a good cause, people might respond. But the questions are vague, hard to answer, and not at all controlled.
Garbage in, garbage out.
Today I received a forward from a colleague who has his pulse on global design issues, passing on a survey request from a graduate supervisor at a prestigious east coast US design school. Doubly-endorsed, then, with an intro by the students
We are one of the thesis research teams from the “Design Management” masters program at REDACTED. We are comprised of four dynamic individuals who bring unique set of skills and expertise that substantiates our team. We are highly motivated and eager to seek out credible information.
The research is focused on “Bottled Water” and its affects [sic] on our planet. In the times when the world is focusing on oil as a momentous energy resource that is on the verge of gaining the status of a deficient commodity, this thesis team is exploring indications that cognize [sic] drinking water as a much more serious and fateful resource. With a pragmatic attitude the team’s primary focus is on the bottled water industry and its impact on life, environment and economies. By 2015 over 60% of the world population will be living in urban areas and the use of bottled water is increasing by 12% per annum.
This survey is conceived and designed by the team to get firsthand information in order to understand the trends, perceptions and know-how of people worldwide. It is critically helpful for the team in securing a better perspective of the thought process, gaps, and awareness levels. The survey will be used as part of the thesis research and one of the pillars to base strategic and sustainable recommendation by using Design Management tools.
The team looks forward to your support and cooperation in reaching its goals. This survey will also create way for the future researchers who would be able to use these finding to elaborate and continue the process of strategic enlightenment and making the planet a better place for the generation to come.
Well. Is that preeningly snooty enough for you (ignoring the typos, of course)? Certainly some high expectations have been established here. So, let’s look at two pages from the online survey that we simply must contribute to.
Oh yuck. This is terrible. After (not shown) a lot of demographic and behavioral data (how old are you, how much money do you make, how much do you spend on bottled water every day, etc.) we get to the opinion and perception questions. Except these are ridiculous leading questions that reveal the opinions of the survey writers, and place the respondent in an awkward situation.
Do you believe that water can be more expensive than oil? gives away the game. Selfishly earnest, but also ineffective.
Maybe, try, something like this (very rough)
For each of the following, compare your expectation of its price to water
Much more costly—more costly—same—cheaper—much cheaper
gasoline
coca-cola
shampoo
orange juice
The question mustn’t reveal the intention. And it must not (as the last 3 questions do) put the person on the defensive, implying that they should be doing something in a certain way. It’s not ethical, but it’s just not effective.
Again, I see design students doing this all the time. It’s really bad research, and it’s sadly being endorsed or encouraged by faculty and others who don’t know or don’t care. “Oh, it’s still useful information” I can hear them saying. But it’s not. The data you get from this is useless, or worse than useless since it’s actually misleading.
This example is more egregious because of the smarmy greener-than-thou effluent it exudes.
Ideally, the kind of perception issues these students are after would be collected in some conversations, where follow-ups and probes and listening all come together to generate some new insight. This is not a good use of a survey, especially in such a ham-fisted manner.