All You Need to Know to Build an AI Chatbot With NLP in Python

What is a chatbot?

A chatbot is a computer program that simulates and processes human conversation. It allows users to interact with digital devices in a manner similar to if a human were interacting with them. There are different types of chatbots too, and they vary from being able to answer simple queries to making predictions based on input gathered from users.

Why are chatbots important?

The cost-effectiveness of chatbots has encouraged businesses to develop their own. This has led to a massive reduction in labor cost and increased the efficiency of customer interaction.

Chatbots help businesses to scale up operations by allowing them to reach a large number of customers at the same time as well as provide 24/7 service. They also offer personalized interactions to every customer which makes the experience more engaging.

Some common examples include WhatsApp and Telegram chatbots which are widely used to contact customers for promotional purposes. They also provide important information and updates.

Different types of chatbots

There are several types of chatbots including:

Scripted chatbots

These are the simplest types of chatbots. They require predefined knowledge. Queries have to align with the programming language used to design the chatbots.

NLP chatbots

These chatbots require knowledge of NLP, a branch of artificial Intelligence (AI), to design them. They can answer user queries by understanding the text and finding the most appropriate response.

Service chatbots

Widely used by service providers like airlines, restaurant booking apps, etc., action chatbots ask specific questions from users and act accordingly, based on their responses.

Conversational chatbots

These are the most advanced types of chatbots. Here, the input can either be text or speech and the chatbot acts accordingly. An example is Apple’s Siri which accepts both text and speech as input. For instance, Siri can call or open an app or search for something if asked to do so.

Challenges of developing a chatbot

Despite their popularity, several challenges need to be considered when designing AI-assisted chatbots. These are:

• Misspellings
• Synonyms
• Slangs and abbreviations
• Punctuations
• Homophones
• Sarcasm
• Idioms

By addressing these challenges, we can enhance the accuracy of chatbots and enable them to better interact like human beings.

Understanding NLP AI chatbots

An AI chatbot is built using NLP which deals with enabling computers to understand text and speech the way human beings can. The challenges in natural language, as discussed above, can be resolved using NLP. It breaks down paragraphs into sentences and sentences into words called tokens which makes it easier for machines to understand the context.

NLP has several applications:

Sentiment analysis

NLP is used to extract feelings like sadness, happiness, or neutrality. It is mostly used by companies to gauge the sentiments of their users and customers. By understanding how they feel, companies can improve user/customer service and experience.

Speech recognition

This is also known as speech-to-text recognition as it converts voice data to text which machines use to perform certain tasks. A common example is a voice assistant of a smartphone that carries out tasks like searching for something on the web, calling someone, etc., without manual intervention.

Document summarization

NLP is used to summarize a corpus of data so that large bodies of text can be analyzed in a short period of time. Document summarization yields the most important and useful information.

Machine translation

NLP helps translate text or speech from one language to another. It’s fast, ideal for looking through large chunks of data (whether simple text or technical text), and reduces translation cost.

Apart from the applications above, there are several other areas where natural language processing plays an important role. For example, it is widely used in search engines where a user’s query is compared with content on websites and the most suitable content is recommended.

Popular NLP tools

Some popular tools for implementing NLP tasks are listed below:

Natural Language Toolkit (NLTK)

It is an open-source collection of libraries that is widely used for building NLP programs. It has several libraries for performing tasks like stemming, lemmatization, tokenization, and stop word removal.

spaCy

It is one of the most powerful libraries for performing NLP tasks. It is written in Cython and can perform a variety of tasks like tokenization, stemming, stop word removal, and finding similarities between two documents.

Sentence Transformers

This is the most advanced package developed by Hugging Face. It is used to find similarities between documents or to perform NLP-related tasks. It provides easy access to pre-trained models through an API. It also reduces carbon footprint and computation cost and saves developers time in training the model from scratch.

Steps to create an AI chatbot using Python

The following are the steps for building an AI-powered chatbot.

Import libraries

Begin by importing some essential libraries. These include:

a. Pandas: Used for creating a data frame.

b. NumPy: A Python package used for working with arrays and performing matrix-related operations.

c. JSON: A library for working with JSON (JavaScript Object Notation) data.

d. TensorFlow: Required for creating models that will be used to make predictions.

Create JSON of intent

To build a chatbot, it is important to create a database where all words are stored and classified based on intent. The response will also be included in the JSON where the chatbot will respond to user queries. Whenever the user enters a query, it is compared with all words and the intent is determined, based upon which a response is generated.

An example of JSON is illustrated below:

database = {"intents": [
         {"class": "name",
          "words": ["what’s your name?"],
          "responses": ["I’m Steve, an AI-assisted chatbot!"]
         },
          {"class": "greetings",
          "words": [ "Hi", "Hello", "Hey",”Good morning”],
          "responses": ["Hey", "Hi there!", "Greetings! How can I assist you?"],
         },
          {"class": "ending-conversation",
          "words": [ "bye", "later"],
          "responses": ["goodbye", "see you!"]
         },
         {"class": "payment method",
          "words": ["what’s the most preferred payment option?"],
          "responses": ["We accept MasterCard/ Visa/ PayPal," "Please let us know in case you want to bank transfer!"]
         }

]}

Preprocess data

An important stage, this is where the data is preprocessed before it is sent to train the model. There are several steps involved:

a. Stemming: This involves removing letters from a word, irrespective of the inflections, without knowledge of the context. Note that these root forms may not be actual words.

b. Lemmatization: Like stemming, lemmatization reduces words to their base form. The difference is that a lemma is an actual word. For example, ‘moving’ and ‘movement’ come from the word ‘move’, which can be easily understood by a machine, hence, enabling more accurate predictions.

c. Removal of stop words: Stop words include articles, prepositions, pronouns, conjunctions, etc., which don’t add much information to the text. They are removed in order to focus on important information.

d. Tokenization: In this stage, sentences are tokenized (broken into small chunks) into tokens (or words). These tokens are easily understood by the machine.

Design a neural network model

Machines cannot learn from tokenized words directly. The tokens need to be represented as numbers. They are, therefore, converted to a vector representation using two techniques: bag of words (BoW) and term frequency–inverse document frequency (TF-IDF).

BoW

In this encoding technique, the sentence is first tokenized into words. They are represented in the form of a list of unique tokens and, thus, vocabulary is created. This is then converted into a sparse matrix where each row is a sentence, and the number of columns is equivalent to the number of words in the vocabulary.

This can be explained with the help of an example:

sents = ['coronavirus is a highly infectious disease',
         'coronavirus affects older people the most', 
         'older people are at high risk due to this disease']

The vocabulary for the above sentences will be created as follows:

The sparse matrix is created as follows:

In the above sparse matrix, the number of rows is equivalent to the number of sentences and the number of columns is equivalent to the number of words in the vocabulary. Every member of the matrix represents the frequency of each word present in a sentence.

TF-IDF

This method is also based on frequency. A major advantage of using TF-IDF over BoW is that it does not give much preference to articles, prepositions, and conjunctions. The technique has two parts: term frequency and inverse document frequency.

TF is calculated as follows:

IDF is calculated as:

IDF can also be calculated as the logarithm of inverse of document frequency (DF).

DF is calculated as:

The final TF-IDF score is calculated as follows:

Skip-gram

In this method of embedding, the neural network model iterates over each word in a sentence and tries to predict its neighbor. The input is the word and the output are the words that are closer in context to the target word.

Skip-gram can be illustrated with the help of the diagram below:

Image source: Towards Data Science

Continuous bag of words (CBoW)

It is similar to the skip-gram method but the difference is that the neural network model is used to predict the current word, unlike in skip-gram.

BoW is one of the most commonly used word embedding methods. However, the choice of technique depends upon the type of dataset.

The implementation of BoW encoding is shown below:

train_data= [] # training array 
empty_out= [0] * len(num_classes)
# Bag of Words model
for idx, doc in enumerate(documentX):
    bagOfwords = []
    text = lm.lemmatize(doc.lower())
    for word in newWords:
        bagOfwords.append(1) if word in text else bagOfwords.append(0)
outputRow = list(empty_out)
outputRow[num_classes.index(documentY[idx])] = 1
train_data.append([bagOfwords, outputRow])

random.shuffle(train_data)
train_data= num.array(train_data, dtype=object)# converting our data into an array
x = num.array(list(trainingData[:, 0]))# first training phase
y = num.array(list(trainingData[:, 1]))# second training phase

Once the training data is prepared in vector representation, it can be used to train the model. Model training involves creating a complete neural network where these vectors are given as inputs along with the query vector that the user has entered. The query vector is compared with all the vectors to find the best intent.

Another way to compare is by finding the cosine similarity score of the query vector with all other vectors. The result is the intent that has the highest score.

The four steps underlined in this article are essential to creating AI-assisted chatbots. Thanks to NLP, it has become possible to build AI chatbots that understand natural language and simulate near-human-like conversation. They also enhance customer satisfaction by delivering more customized responses.