These are the fundamental principles to consider in order to drive value from your customer’s unstructured feedback
1. Build your own codebook and use the right AI
It’s not enough to just buy a text analytics software or subscription. In most cases it does not fully satisfy the expectations your business partners have today.
What you need is a text AI that you can train.
This training starts by defining the set of themes (codebook) that you want your customer’s feedback gets quantified. You don’t want to outsource this to software.
Be sure the software you are using has the right measures to validate its accuracy. E.g. you do not want to look at hit rates but F1 score.
2. Train it well
Garbage in garbage out: Training itself has some tricks and trades that you can either learn yourself or you can find external partners who are experienced in this.
It’s not enough to use a domain expert. The codebook should be documented well and the person should stay the same over time – or the handover phase should be extensive.
Over time training is changing the way your system categorizes feedback. Either you stop training to maintain consistency (not recommended, as the accuracy will decay over time) or you must rebaseline the past once in a while.
It’s important to communicate this expectation early on: No categorization will ever be perfect.
3. Do NOT interpret text analytics – its just data
The greatest misconception about text analytics is jumping from data to conclusions. Intuitively businesses look at the most often mentioned topics. Because they believe these are the reasons for success or failure.
Even worse: this makes perfect sense as it is the answer to the question “why did you rate that way”. And the customer is telling us why.
However, it turns out that the frequency of mentionings and importance is largely uncorrelated.
In other words, whenever you give your business partners unguided access to frequencies of topics, they will most likely conclude with highly imperfect decisions.
The Microsoft case above shows how this issue must be solved.