How to create a truly enjoyable survey…
and why this is the key to get deeper and more actionable insights.
Do you remember those hot days in the summer of 2018? At one of those days, we were about to launch a mobile survey with 16 questions, 8 of them being open-ended. My colleague David, who were sweating next to me, said “This will never work! Who wants to respond to 8 open-ended question, on a mobile phone?”. Still, we had no other choice. We even added two more questions, “did you enjoyed this survey?” and “why did you answered this way?”
The outcome was a surprise. On average, about 50% of respondents in conventional surveys indicate they enjoyed the survey. Our mobile survey achieved a 91% enjoyment rate. Wow!
You will ask yourself “why?”. This is exactly what we asked the respondents. The most frequently mentioned reasons were “it was easy”, “these were good questions” and “it was fast”. (actually, the average time was 5 minutes, which is a rather long response period on a mobile phone).
Are you satisfied with these answers? We took them to the test. If a reason is a true reason, it must be possible to use its information to better predict the enjoyment rating (in a multivariate model). This is a ground-truth formulated by the fathers of causality research such as Nobel Prize winner Clive Granger.
This is why we build a flexible (machine learning based) prediction model and discover something quite exciting. None of the frequently mentioned reasons predicts well why respondents enjoyed the survey. Instead, two other reasons are highly predictive.
The text response “the survey was simple” was mentioned by 10% of respondents and was a strong predictor of survey enjoyment. Further, text that can be summarized as “i liked the opportunity to describe my sentiments with my own words” was mentioned by 9% and is an even stronger predictor.
Recent attempts to increase respondents’ engagement in the industry typically circle around varied and entertaining ways of surveying and to use more gamification approaches. But what we now learned is that the key is less about building more sophisticated ways of asking, it is more about to keep it really simple and give respondents the option to communicate in the most authentic way possible: to use their own words.
You may ask “ok, but is coding open-text responses economically even feasible and attractive?” Recent technologies that combine Natural Language Processing and Machine Learning can automatically code large volume of text. The system we use must be trained by a human coder. Then its output has comparable predictive power with human codings.
“Ok, but what does this mean for improving the quality of insights?” you might ask yourself…. good question!
Let’s look at what we have learned. We have learned that we receive the most authentic and high quality information from respondents (e.g. your customers) by asking very simple questions and let people respond in their own words. At the same time, we now have the technology to discover hidden drivers of success in those simple text responses.
Of course, we can not only learn what is truly enjoyable with surveys, we also can learn why customers are loyal or more likely to recommend your brand. We can also learn why they consider certain brands or choose a new product innovation.
We can learn valuable insights that are key to success by using extremely simple and cost effective research methods. What could be more exciting?
If you like to dive deeper … e.g. how to make this work for your NPS program, your Brand Tracking study or whatever you think drives value in your company, just let me know and I’ll be happy to chat.
Yours, Frank
