What Is NLP and Why Does It Matter?

Natural language processing (NLP) is a branch of artificial intelligence (AI) related to how software can 'listen to,' process, and manipulate language. Put simply, it's all about the ability of a computer program to understand human language in the way that it is spoken.

Anyone who has ever used a search engine or spoken to a voice assistant like Alexa is actually already familiar with NLP. It's what makes it possible for AI-powered discovery services to understand written or spoken queries and then deliver answers to them. Natural language processing technically refers to the interpretation of anything said to a computer that isn't in a programming language — but multi-word, long-tail questions are the present disrupter and new frontier for NLP.

Current approaches to NLP are "based on deep learning, a type of AI that examines and uses patterns in data to improve a program's understanding [of language over time]," Margaret Rouse writes for Search Business Analytics. "Deep learning models require massive amounts of labelled data to train on and identify relevant correlations, and assembling this kind of big data set is one of the main hurdles to NLP currently."

As natural language processing improves, so too does the possibility of seamless communication and understanding between people and their devices. And the removal of friction in this area has heightened consumer expectations in search.

Why does NLP matter?

NLP technology continues to improve at an unprecedented rate. 2019 saw especially swift progress, spurred on by the introduction and adoption of BERT — Google's revolutionary new model for natural language understanding. In late 2019, Google implemented BERT into its search algorithm, with the goal of better understanding long, complex search queries.

User behavior is quickly adapting to these advances in NLP. People are increasingly using long, natural language queries in search. Instead of "doctor" or "GP," they'll search for "internal medicine doctor near me who takes Aetna insurance," trusting that today's newer, smarter algorithms will be able to understand their complex query and serve relevant results. Consumers now fully expect to find the answers to these questions in search.

Multi-word, natural language queries are especially important because they express high intent. Using the example above, "doctor" is a vague and competitive keyword, as it is unclear what specifically the user is looking for. "Internal medicine doctor near me who takes Aetna insurance," on the other hand, is highly specific and expresses a clear intent. This user is looking for a particular type of doctor, in a particular city, who takes a particular kind of insurance — and they're clearly closer to booking an appointment. The more words in the query, the greater the intent it expresses.

At these moments of high intent, a consumer's business will go to the company that's ready to provide the answer. Research suggests that 76% of customers believe it's now easier than ever to take their business elsewhere, switching from brand to brand to find an experience that matches their expectations for answers. If your brand can't provide direct, structured answers to customers' natural language questions — both on your website and via third-party search experiences — your customers may look to a competitor, and you'll lose out on retention and revenue.

Learn more about how consumer search experiences are changing inThe Customer Journey Starts With a Question.