Deep Learning: Recurrent Neural Network

In this tutorial, you will see how to build a recurrent neural network (RNN), and how to use it to perform a simple sentiment analysis task.

from tqdm.notebook import trange
import pandas as pd
import torch
import torch.nn as nn
import torch.optim as optim

seed = 314
device = "cpu"
path = "/content/drive/<path to the sentiment dataset>/Sentiment/"

Sentiment analysis

Let’s have a look at the data used for this task. It is a toy dataset that comprehends a set of sentences and a binary label: 0 means that the sentence conveys a negative sentiment, 1 a positive sentiment.

trset = pd.read_csv(path + "train_en.csv")
tsset = pd.read_csv(path + "test_en.csv")
tsset

Vocabulary

Since the dataset contains simple sentences, we can tokenize following the spaces. Then, we can build our vocabulary, with a mapping of the words to an integer number. The mapping is needed to easily create word embeddings.

• join takes a list of strings and glue them together, separating them with the character used to call the function (the space " " in our case).
• split instead takes in input a string and splits according to the character passed to the function (again, the space " ") to generate a collection of strings (the single words of the corpus).
• set is a standard structure that imitate a mathematical set. When passing the splitted words, it takes care of eliminating the duplicates storing only one instance per word.

The vocabulary is then used to create the mapping word/index, using a dictionary comprehension

def mapping(trset):
  sentences = " ".join(s for s in trset["sentence"])
  vocabulary = set(sentences.split(" "))
  voc_size = len(vocabulary)
  voc_idx = {w: i for i, w in enumerate(vocabulary)}
  return voc_idx, voc_size

voc_idx, voc_size = mapping(trset=trset)

One-hot embedding

Next, we prepare the embedding functions. For this simple dataset we can use the one-hot embedding one of the simplest ones. Given a vocabulary size $d$ and an integer number $i \in [1, \dots, d]$, its one-hot embedding is a vector $\vec{x} \in \mathbb{R}^d$, with $x_j = 1$ if $j = i$, otherwise $x_j = 0$. In simple words, the one-hot vector of $i$ has all entries set to $0$ except for the $i$-th entry, set to $1$.

The corpus2onehot function takes in input a collection of $n$ sentences and a vocabulary mapping, and generates a list of tensors of size (l_s, d), with l_s being the number of words of sentence s (each sentence may have a different number of words, thus we need an index s for each sentence).

def sentence2ids(sentence, voc_idx):
  words = sentence.split(" ")
  return torch.tensor([voc_idx[word] for word in words])

def corpus2onehot(corpus, voc_idx):
  data = []
  for sentence in corpus:
    ids = sentence2ids(sentence, voc_idx)
    one_hots = torch.nn.functional.one_hot(ids, len(voc_idx))
    data.append(one_hots)
  return data

trcorpus = corpus2onehot(trset["sentence"], voc_idx)
tscorpus = corpus2onehot(tsset["sentence"], voc_idx)
tscorpus

[tensor([[0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
         [0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0]]),

 ...

 tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0],
         [0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
         [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1],
         [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0]])]

Characteristics of text

Text data is particular in two aspects:

• The word contained in a sentence are not independent, and are interconnected together. For example, a sentence containing the word “bank”, may regards very different topics. Perhaps it is about hidrology, in such case it is probable to find another hidrology-related word, like “river”. Perhaps it is about finance, then we may find finance-related words such as “money”, and so on. In summary our algorithm of choice should be able to learn the interplay of different words
• The aforementioned relations may intercur between very distant words. For example, in the following sentence “The Prime Minister, which has already proved to be competent in other situations, decided to publicly express its concerns” the words “Minister” and “decided” are interrelated together through the relation subject/verb, however they are separated by a rather long subordinate sentence.

Model

Taking into consideration these aspects, Recurrent Neural Network was one of the first proposed models to address Natural Language Processing (NLP) tasks.

An RNN is fed the sentence word-by-word. At each step $t$, it takes an input word embedding $\vec{x}^{(t)}$, it computes an hidden state $\vec{h}^{(t)}$ and an output state $\vec{o}^{(t)}$. $\vec{h}^{(t)}$ is computed with the hidden state $\vec{h}^{(t-1)}$ computed on the previous word embedding $\vec{x}^{(t-1)}$, then $\vec{h}^{(t)}$ is sent as output for the next word embedding. The formula for a single step is the following:

$$ \vec{h}^{(t)} = \tanh(\vec{U}\vec{x}^{(t)} + b_u + \vec{V}\vec{h}^{(t-1)} + b_v) \\ $$ $$ \vec{o}^{(t)} = \sigma(\vec{W}\vec{h}^{(t)} + b_w) $$

Some explanation:

• $\vec{U}$, $\vec{V}$, $\vec{W}$ and the corresponding biases are all learnable parameters.
• $\tanh$ is the hyperbolic tangent function. It has an “S”-like shape, similarly to the sigmoid, however its range is $(-1, 1)$ rather than $(0, 1)$.
• $\sigma$, in our case, is the sigmoid function.
• A dummy hidden state $\vec{h}^{(0)}$ is created first and sent as input with $\vec{x}^{(1)}$ to start the process.
• For sentiment analysis, we just need the last output of the network, and we skip the computation of the previous outputs. In other tasks, like Part-Of-Speech Tagging we need instead all the outputs. These two approaches are respectively called many-to-one and many-to-many.

Following, we show the model we use. PyTorch provides the RNN module, which computes the hidden outputs only. We take care of computing the network output with a Linear + Sigmoid module

class SentimentRNN(nn.Module):
  def __init__(self, voc_size, hidden_size):
    super(SentimentRNN, self).__init__()

    self.rnn = nn.RNN(input_size=voc_size, hidden_size=hidden_size,
                      batch_first=True)
    self.classifier = nn.Linear(hidden_size, 1)
    self.sigmoid = nn.Sigmoid()
  
  def forward(self, sentence):
    h0 = torch.zeros(1, 1, 64)  # initial hidden input is set to zeros values
    _, hn = self.rnn(sentence, h0)
    prob = self.sigmoid(self.classifier(hn))
    return prob

Expected loss

The loss function we will use in this case is one which is more fit for classification problems, and it is called Binary Cross Entropy. The output of our network is a scalar $p$ generated by the sigmoid function. This quantity represents the probability of observation (sentence) $\vec{x}$ to convey a positive sentiment ($y = 1$), that is $p = \mathbb{P}(\hat{y} = 1 | \vec{x}, \vec{\theta})$. The binary cross-entropy, is then computed as follows:

$$ \ell(p, y) = -y\ln(p) - (1 - y)\ln(1 - p) = \begin{cases} -\ln(p) & \text{if } y = 1\\ -\ln(1 - p) & \text{otherwise} \end{cases} $$

We have already seen the function $-ln(p)$ in the previous lessons. It represents a measure of “surprise”, in this case, the amount of surprise generated by the comparison of the ground truth with the (probability for the) prediction. If the prediction is wrong, the loss will be high, if the prediction is correct the loss will be $0$ (remember that $\ln(1) = 0$).

In PyTorch, we can use the binary cross entropy loss with nn.BCELoss.

Training

We are ready to train our model with the usual gradient descent algorithm. Look at the loss decrement epoch-by-epoch, in parallel with the increment of validation accuracy.

def evaluate(y_true, y_pred):
  correct = (y_true == y_pred).sum()
  accuracy = 100. * correct / y_true.shape[0]
  return accuracy.item()  # returns just the value without the tensor wrapper
  
def train(epochs, model, trcorpus, tscorpus):
  BCE = nn.BCELoss()
  optimizer = optim.SGD(model.parameters(), lr=1e-3)
  
  pbar = trange(epochs, desc="E: , ACC: ")
  for epoch in pbar:
    # Training
    avg_loss = 0.
    model.train()
    for sentence, y in zip(trcorpus, trset["sentiment"]):
      sentence = sentence.to(device).to(torch.float).unsqueeze(0)
      prob = model(sentence)
      E = BCE(prob[0], torch.tensor([[y]]).to(device).to(torch.float))
      avg_loss += E.item()
  
      optimizer.zero_grad()
      E.backward()
      optimizer.step()
  
    # Evaluation
    y_true = torch.tensor(tsset["sentiment"].to_list())
    y_pred = []
    model.eval()
    for sentence in tscorpus:
      sentence = sentence.to(torch.float).unsqueeze(0)
      prob = model(sentence)
      y_pred.append(prob >= 0.5)
  
    y_pred = torch.tensor(y_pred)
    pbar.set_description(f"E: {avg_loss / len(trcorpus):.4f}, "
                         f"ACC: {evaluate(y_true, y_pred):.4f}")

model = SentimentRNN(voc_size=voc_size, hidden_size=64).to(device)
train(epochs=2000, trcorpus=trcorpus, tscorpus=tscorpus, model=model)