“Standing on the shoulders of giants”

The last 12 months or so have seen ground-breaking developments in the world of natural language processing (NLP). Businesses without the resources of the tech giants can “stand on their shoulders” and reap the benefits of these R&D developments. Fast-following is no longer a euphemism for being lazy.

One of the great obstacles to making better informed business decisions is getting access to first-hand customer or employee insight, at scale. If you want to understand why customers are leaking from your sales pipeline you would ideally ask a large number of them “why?” and be able to automatically distil customer decision drivers from their responses. Instead – because of the impracticality of having staff manually plough through hundreds or thousands of free-text responses – companies are forced to conduct customer research using closed questions with pre-prepared answer alternatives. The same applies if you want to understand customer perspectives on brand, product, servicing experience, etc. or employee perspectives on culture, business improvement opportunities, etc. This presupposes that the person asking the question already knows what the answers are – somewhat contradicting the need to ask! – and offers no possibility of discovery.

Developments in NLP over the last 12 months are now making scale access to first-hand “voice of the customer / employee” a reality. Throughout 2018 a series of R&D organisations (the Allen Institute for Artificial Intelligence, Open AI, and Google AI) respectively published new approaches to “vectorising” words – essentially new ways of converting a word into a series of numbers which in some sense encapsulate the meaning of the word.

So what do these developments mean for businesses? Here are our top three things to know:

Scale access to “Voice of the customer / employee” is now possible

Use these new NLP techniques to automatically extract and distil the top thematic answers to any open-ended question you can conceive of.

Integrate these unstructured insights into predictive models to aid “precision decisions”

Now that the qualitative has been rendered quantitative, these insights can form inputs to predictive models so that, for example, the top three self-declared drivers of employee churn (or the top three customer-declared drivers of sales leakage, or …) can be isolated and addressed.

Be knowledgeable about the limitations of NLP

…in particular, be wary of anyone who describes their solution as achieving “natural language understanding”. Notwithstanding the great strides that have been made, we are still a very long way from achieving true natural language understanding. Even one of the best performing AI language models – OpenAI’s GPT-2 model – with its 1.5 billion parameters trained on 8 million web pages, still needs multiple attempts to generate realistic human quality text, and even then fails to achieve world understanding (e.g., “the model sometimes writes about fires happening under water”). Practically speaking, real-world business applications that could come from true natural language understanding (e.g., goal-oriented chatbots) are still some way off.