Mario Meissner’s Blog

The Dissected Attention Network

2020-03-08T00:00:00+00:00

This notebook is based on the Pytorch “Translation with a Sequence to Sequence Network and Attention” tutorial by Sean Robertson. Most of the code is taken from the original notebook, but “dissected”. That means that many functions and classes will be stripped apart, and the internals analyzed. We will build a basic sequence to sequence network, while paying attention to the attention part (no pun intended).

Run it yourself on Colab. I edited my commentary on the mardown version (here) after exporting, so the text of the runnable notebook differs slightly.

This notebook is a byproduct of my learning process. I publish it in the hopes that someone may find some of the bits useful. It focuses on exploring seemingly basic segments that I myself didn’t feel comfortable with, while it may skip over other important steps. Feel free to send me suggestions of improvement or clarification!

This is my first time using PyTorch coming from a Keras background, so I will comment on some of my thoughts while using it.

This deep dive helped me uncover some inaccuracies in the original notebook, so I submitted a pull request.

# Let's import the stuff we will need
from __future__ import unicode_literals, print_function, division
from io import open
import unicodedata
import string
import re
import random
import torch
import torch.nn as nn
from torch import optim
import torch.nn.functional as F
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
%matplotlib inline

Loading data files

We will use French - English sentence pairs, which we can get from here.

# !wget -nc https://download.pytorch.org/tutorial/data.zip
# !unzip -n data.zip

This Lang class keeps track of the word to index dictionaries and word counts.

SOS_token = 0
EOS_token = 1

class Lang:
    def __init__(self, name):
        self.name = name
        self.word2index = {}
        self.word2count = {}
        self.index2word = {0: "SOS", 1: "EOS"}
        self.n_words = 2  # Count SOS and EOS

    def addSentence(self, sentence):
        for word in sentence.split(' '):
            self.addWord(word)

    def addWord(self, word):
        if word not in self.word2index:
            self.word2index[word] = self.n_words
            self.word2count[word] = 1
            self.index2word[self.n_words] = word
            self.n_words += 1
        else:
            self.word2count[word] += 1

Perform unicode -> ascii and a bit of data cleaning: lowercase, stripping, separating punctuation and removing non-letter-or-punctuation characters.

# Turn a Unicode string to plain ASCII, thanks to
# http://stackoverflow.com/a/518232/2809427
def unicodeToAscii(s):
    return ''.join(
        c for c in unicodedata.normalize('NFD', s)
        if unicodedata.category(c) != 'Mn'
    )

# Lowercase, trim, and remove non-letter characters
def normalizeString(s):
    s = unicodeToAscii(s.lower().strip())
    # Separate punctuation from word by adding a space
    s = re.sub(r"([.!?])", r" \1", s)
    # Remove anything that isnt a letter or punctuation
    s = re.sub(r"[^a-zA-Z.!?]+", r" ", s)
    return s

The following function will read from the files and create the Lang objects. For me the confusing part here was the necessity of the reverse flag. Why not just exchange lang1 and lang2 when calling the function? The reason is that the files that we download are all English -> other language, so lang1 here has to be English, and then you swap it.

def readLangs(lang1, lang2, reverse=False):
    print("Reading lines...")

    # Read the file and split into lines
    lines = open('data/%s-%s.txt' % (lang1, lang2), encoding='utf-8').\
        read().strip().split('\n')

    # Split every line into pairs and normalize
    pairs = [[normalizeString(s) for s in l.split('\t')] for l in lines]

    # Reverse pairs, make Lang instances
    if reverse:
        pairs = [list(reversed(p)) for p in pairs]
        input_lang = Lang(lang2)
        output_lang = Lang(lang1)
    else:
        input_lang = Lang(lang1)
        output_lang = Lang(lang2)

    return input_lang, output_lang, pairs

Since there are a lot of example sentences and we want to train something quickly, we’ll trim the data set to only relatively short and simple sentences. Here the maximum length is 10 words (that includes ending punctuation) and we’re filtering to sentences that translate to the form “I am” or “He is” etc. (accounting for apostrophes replaced earlier).

MAX_LENGTH = 10

eng_prefixes = (
    "i am ", "i m ",
    "he is", "he s ",
    "she is", "she s",
    "you are", "you re ",
    "we are", "we re ",
    "they are", "they re "
)

def filterPair(p):
    return len(p[0].split(' ')) < MAX_LENGTH and \
        len(p[1].split(' ')) < MAX_LENGTH and \
        p[1].startswith(eng_prefixes)

def filterPairs(pairs):
    return [pair for pair in pairs if filterPair(pair)]

def prepareData(lang1, lang2, reverse=False):
    input_lang, output_lang, pairs = readLangs(lang1, lang2, reverse)
    print("Read %s sentence pairs" % len(pairs))
    pairs = filterPairs(pairs)
    print("Trimmed to %s sentence pairs" % len(pairs))
    print("Counting words...")
    for pair in pairs:
        input_lang.addSentence(pair[0])
        output_lang.addSentence(pair[1])
    print("Counted words:")
    print(input_lang.name, input_lang.n_words)
    print(output_lang.name, output_lang.n_words)
    return input_lang, output_lang, pairs

input_lang, output_lang, pairs = prepareData('eng', 'fra', True)
print(random.choice(pairs))

Reading lines...
Read 135842 sentence pairs
Trimmed to 10853 sentence pairs
Counting words...
Counted words:
fra 4489
eng 2925
['vous regardez tout le temps la television .', 'you are always watching tv .']

Finally let’s define here some utility functions to create tensors from sentences. Simple enough.

# A few utility functions to transform text into tensors
def indexesFromSentence(lang, sentence):
    return [lang.word2index[word] for word in sentence.split(' ')]

def tensorFromSentence(lang, sentence):
    indexes = indexesFromSentence(lang, sentence)
    indexes.append(EOS_token)
    return torch.tensor(indexes, dtype=torch.long, device=device).view(-1, 1)

def tensorsFromPair(pair):
    input_tensor = tensorFromSentence(input_lang, pair[0])
    target_tensor = tensorFromSentence(output_lang, pair[1])
    return (input_tensor, target_tensor)

Let’s see what they look like. Seems like the tensorFromSentence function is already playing with the tensor shape, unsqueezing an extra empty dim at the end. I suppose this is the embedding dimension that will be populated by the Embedding layer later.

pair = random.choice(pairs)
t = tensorsFromPair(pair)
in_t = t[0]
out_t = t[1]
in_t, out_t, in_t.shape, out_t.shape

(tensor([[ 123],
         [ 298],
         [ 126],
         [ 247],
         [ 557],
         [1256],
         [   5],
         [   1]], device='cuda:0'),
 tensor([[ 77],
         [ 78],
         [148],
         [156],
         [693],
         [  4],
         [  1]], device='cuda:0'),
 torch.Size([8, 1]),
 torch.Size([7, 1]))

The model

Here comes the untouched model code from the original tutorial, which we will then dissect to see what’s going on.

class EncoderRNN(nn.Module):
    def __init__(self, input_size, hidden_size):
        super(EncoderRNN, self).__init__()
        self.hidden_size = hidden_size
        self.embedding = nn.Embedding(input_size, hidden_size)
        self.gru = nn.GRU(hidden_size, hidden_size)

    def forward(self, input, hidden):
        output = self.embedding(input).view(1, 1, -1)
        output, hidden = self.gru(output, hidden)
        return output, hidden

    def initHidden(self):
        return torch.zeros(1, 1, self.hidden_size, device=device)

class AttnDecoderRNN(nn.Module):
    def __init__(self, hidden_size, output_size, dropout_p=0.1, max_length=MAX_LENGTH):
        super(AttnDecoderRNN, self).__init__()
        self.hidden_size = hidden_size
        self.output_size = output_size
        self.dropout_p = dropout_p
        self.max_length = max_length
        self.embedding = nn.Embedding(self.output_size, self.hidden_size)
        self.attn = nn.Linear(self.hidden_size * 2, self.max_length)
        self.attn_combine = nn.Linear(self.hidden_size * 2, self.hidden_size)
        self.dropout = nn.Dropout(self.dropout_p)
        self.gru = nn.GRU(self.hidden_size, self.hidden_size)
        self.out = nn.Linear(self.hidden_size, self.output_size)

    def forward(self, input, hidden, encoder_outputs):
        embedded = self.embedding(input).view(1, 1, -1)
        embedded = self.dropout(embedded)
        attn_weights = F.softmax(self.attn(torch.cat((embedded[0], hidden[0]), 1)), dim=1)
        attn_applied = torch.bmm(attn_weights.unsqueeze(0), encoder_outputs.unsqueeze(0))
        output = torch.cat((embedded[0], attn_applied[0]), 1)
        output = self.attn_combine(output).unsqueeze(0)
        output = F.relu(output)
        output, hidden = self.gru(output, hidden)
        output = F.log_softmax(self.out(output[0]), dim=1)
        return output, hidden, attn_weights

    def initHidden(self):
        return torch.zeros(1, 1, self.hidden_size, device=device)

Encoder Dissection

While reading the encoder code, I came up with two questions that I would like to answer:

Why is the .view(1,1,-1) necessary?
Why do we initialize the hidden state with zeros?

# Parameters for testing
input_size = input_lang.n_words
hidden_size = 64

# Encoder has just two components
embedding = nn.Embedding(input_size, hidden_size)
gru = nn.GRU(hidden_size, hidden_size)

idx = indexesFromSentence(input_lang, pairs[0][0])
idx = torch.tensor(idx)
idx

tensor([2, 3, 4, 5])

embedded = embedding(idx)
embedded.shape

torch.Size([4, 64])

idx = idx.view(1,-1)
embedded = embedding(idx)
embedded.shape

torch.Size([1, 4, 64])

Seems like the embedding layer takes in whatever shape we please, and spits out a tensor with an appended embedding dimension. If we check the documentation, indeed it says: “Input: (\*), LongTensor of arbitrary shape containing the indices to extract”. Neat!

The GRU however is stricter. Checking its doc page, it turns out that PyTorch RNNs want a shape of (seq, batch, H_in), instead of (batch, seq, H_in), which is what I am used to as a Keras user.

# A `(seq, batch, H_in) tensor for the GRU
idx = indexesFromSentence(input_lang, pairs[0][0])
idx = torch.tensor(idx)
print(f"idx shape: {idx.shape}.")
embedded = embedding(idx)
embedded = embedded.unsqueeze(1) # Create batch dim
print(f"embedded shape: {embedded.shape}.")

idx shape: torch.Size([4]).
embedded shape: torch.Size([4, 1, 64]).

out,hidden = gru(embedded)
out.shape, hidden.shape

(torch.Size([4, 1, 64]), torch.Size([1, 1, 64]))

The GRU gives us an output tensor of the same shape as the input (because we gave it the same input and output size during initialization) and its last hidden state, which is why the seq dimension is empty there.

Now that we understand how the shapes work at the encoder, we can answer why we needed the view(1,1,-1) in the original script. It is very likely that during training, the encoder receives as input a rank 0 tensor containing only the index of a single word, which is transformed into a rank 1 tensor (embedding), and finally the view inflates it into a (seq, batch, embedding), where both seq and batch are 1. Seems like an interesting challenge at the end of this script could be to improve this to correctly pass tensors with filled seqs and batches.

Let’s also try to answer why we initialize with zeros by doing a little internet search.

A user on stackexchange highlights that it is important to distinguish between model weights, which will be updated through gradient descent, and states, which don’t. There seem to be alternatives to zero initialization, but it’s the most commonly used one, so let’s stick with it!

Decoder Dissection

On to the decoder! Let’s follow the same pattern of initializing everything and then playing around with it.

hidden_size = 64
out_size = output_lang.n_words

embedding = nn.Embedding(out_size, hidden_size)
attn = nn.Linear(hidden_size * 2, MAX_LENGTH)
attn_combine = nn.Linear(hidden_size * 2, hidden_size)
dropout = nn.Dropout(0.1)
gru = nn.GRU(hidden_size, hidden_size)
out_linear = nn.Linear(hidden_size, out_size)

# Since the decoder also works with just one sequence step,
# lets take just one element here
idx = indexesFromSentence(input_lang, pairs[0][1][0])
idx = torch.tensor(idx)
embedded = embedding(idx).unsqueeze(1)
embedded.shape

torch.Size([1, 1, 64])

So far so good, we now have an embedded sequence. The idea behind attention is that we will create an additional input to our GRU, which is an attention vector, concatenated to target sequence.

# The decoder also assumes a context state with empty `seq`
# This guy comes from our encoder tests above
print(hidden.shape)
print(out.shape)

torch.Size([1, 1, 64])
torch.Size([4, 1, 64])

# The next thing that happens is a concatenation
concat = torch.cat((embedded[0], hidden[0]), 1)
concat.shape

torch.Size([1, 128])

What we have here is our embedded token together with the context vector of our encoder, so (batch, hidden * 2). We will run this through a linear which will compress it into (batch, max_seq). This is the core of the attention component.

The goal is to produce a vector that says “this is how much you should pay attention to each of the outputs of the encoder, during the current decoding step”. The information we let it take into account to judge this is the context vector and the current target token.

Attention module talking: “Looking at this context vector, which tells me more or less what the meaning of the input sentence was, and knowing you are currently at this target token in the prediction sequence, I can tell you that in order to produce the next output token, you better look at this particular part of the encoding sequence.”

attention = attn(concat)
attention.shape

torch.Size([1, 10])

# We need to turn it into a probability distribution,
# so we throw it through a softmax
attn_weights = F.softmax(attention, dim=1)
print(attn_weights.shape)
# We can check it adds up to one now
assert torch.abs(torch.sum(attn_weights.squeeze()) - 1) < 0.001

torch.Size([1, 10])

The decoder expects encoder_outputs to have a shape of (seq, hidden), with seq of MAX_LENGTH, so let’s build a fake one here.

After having a peek at the training loop, this is another thing that makes this code quite messy: the batch dim is now absent here again, although we artificially created one with the view(1,1,-1) at the encoder. The training loop will remove it, and later we will once again add a fake one. Let’s stick to what they give us, for now.

out = torch.zeros([MAX_LENGTH, hidden_size])
out.shape

torch.Size([10, 64])

torch.bmm is a batch matrix product, which basically means it takes rank 3 tensors and interprets the first dim as a batch, and performs torch.mm on the two remaining dims.

Again, to me this line of code looks quite silly since we are adding an empty batch dimension to then use the bmm function. We could have used mm directly, and make it way easier to understand. In a real model where batches are treated with a bit more respect, bmm would be necessary though, which is why I assume they kept it here.

attn_applied = torch.bmm(attn_weights.unsqueeze(0), out.unsqueeze(0))
attn_applied.shape

torch.Size([1, 1, 64])

# Now we build the input for the GRU
gru_input = torch.cat((embedded[0], attn_applied[0]), 1)
gru_input.shape

torch.Size([1, 128])

We could technically use this already to feed it to the gru, but I suppose sticking a linear layer here helps reduce the size of the recurrent kernel.

gru_input = F.relu(attn_combine(gru_input).unsqueeze(0))

Once again, we have unsqueezed to bring back the batch dim, which we need because GRU is grumpy about dimensions.

output, hidden = gru(gru_input, hidden)
output.shape, hidden.shape

(torch.Size([1, 1, 64]), torch.Size([1, 1, 64]))

# A final linear layer and we are done
output = F.log_softmax(out_linear(output[0]), dim=1)
output.shape

torch.Size([1, 2925])

Pfew! Let’s see how the training loop is done, although now that we understand all the shapes, it shouldn’t be much of an issue.

Training Dissection

I will directly break things down here because the original functions are quite long. The goal is not to produce beautiful code but to play around with the internals, so let’s get our knife ready.

# Initialize the models and variables
hidden_size = 256
lr = 0.001
encoder = EncoderRNN(input_lang.n_words, hidden_size).to(device)
decoder = AttnDecoderRNN(hidden_size, output_lang.n_words).to(device)
# Create encoder and decoder optimizers
encoder_optimizer = optim.SGD(encoder.parameters(), lr=lr)
decoder_optimizer = optim.SGD(decoder.parameters(), lr=lr)

We gotta define our loss function. The models already contain a log_softmax function, so we have to pick the loss function that wants -log probabilities as input: NLLLoss. We could have used the CrossentropyLoss, which includes both the log_softmax and final loss calculation.

criterion = nn.NLLLoss()

Rerun from here as many times as you want to manually perform training on your model.

# Grab some data to train with

# Grab random data
# pair = random.choice(pairs)
# input_sentence, output_sentence = pair
# input_tensor, target_tensor = tensorsFromPair(pair)

# Grab the same pair always to see the loss decrease
pair = pairs[88]
input_sentence, output_sentence = pair
input_tensor, target_tensor = tensorsFromPair(pair)

# Create the initial hidden state, shape: (1, 1, hidden_size)
encoder_hidden = encoder.initHidden()
# Reset the optimizer accumulated gradients
encoder_optimizer.zero_grad()
decoder_optimizer.zero_grad()
# Get seq length
input_length = input_tensor.size(0)
target_length = target_tensor.size(0)
# This is where we will keep the results of the encoder
encoder_outputs = torch.zeros(MAX_LENGTH, encoder.hidden_size, device=device)
# Set initial loss
loss = 0

Here we run the encoder manually with a for loop. I think it would have been possible to give the encoder a proper seq dimension and let it handle the loop internally, which would be much more efficient.

# Get the outputs from the encoder
for ei in range(input_length):
    encoder_output, encoder_hidden = encoder(
        input_tensor[ei], encoder_hidden)
    encoder_outputs[ei] = encoder_output[0, 0]

# Tensor that will hold decoder inputs
decoder_input = torch.tensor([[SOS_token]], device=device)

Now, in the code they do a mix of teacher forcing and regular training. Teacher forcing means that you use the true target sequence as input for the decoder, using the outputs only to compute the loss. In regular training, the output of the decoder at each timestep is used as input for the next iteration. Both seem to have advantages and disadvantages, so using fifty-fifty of both results is an interesting compromise. Let’s go with teacher forcing here, though both are very similarly implemented.

# Pass over the context
decoder_hidden = encoder_hidden

Let’s keep track of the attention weights, for later visualizing them.

decoder_attentions = torch.zeros(MAX_LENGTH, MAX_LENGTH)

Let’s also keep track of the decoded words that the decoder predicts.

decoded_words = []

I would need to investigate on how to avoid a for loop for the decoder. That seems to be more difficult since there are a lot of steps to perform between each iteration.

for di in range(target_length):
    decoder_output, decoder_hidden, decoder_attention = decoder(
        decoder_input, decoder_hidden, encoder_outputs)
    loss += criterion(decoder_output, target_tensor[di])
    decoder_attentions[di] = decoder_attention
    decoder_input = target_tensor[di]
    topv, topi = decoder_output.data.topk(1)
    if topi.item() == EOS_token:
        decoded_words.append('<EOS>')
        break
    else:
        decoded_words.append(output_lang.index2word[topi.item()])

Now the loss is accumulated in the loss variable, and we can call the PyTorch backward function to perform a calculation of all our gradients. After calling this function, according to PyTorch docs, the graph that was created during the forwards pass is “freed”, which means we cannot call backward a second time. We first have to reset the gradients and perform another forward pass.

Something interesting that happened here is that, although our loss was initialized as an integer (loss = 0), as soon as we run the line loss += criterion(...), the int will be promoted to a PyTorch tensor, which has the method backward.

loss.backward()
print(loss)

tensor(40.0643, device='cuda:0', grad_fn=<AddBackward0>)

# Perform a weight update
encoder_optimizer.step()
decoder_optimizer.step()

If we run the cells between the cell marked above as “rerun from here” to here, we should see the average loss dropping over time. However, as we are grabbing a random sentence every time, there are big changes in the values each iteration. You can comment out that line and uncomment the one where we take the same value every time. The model will of course overfit on a single sequence, but we will be able to see the loss dropping consistently, which is pleasant to see after we dissected the whole thing and made it run “manually”.

Extra! Visualizing the attention weights

The decoder implemented by the original notebook already returns the attention weights, which we have caught above during training, and can plot with matplotlib.

from matplotlib import pyplot as plt
from matplotlib import ticker

attention = decoder_attentions.detach().cpu().numpy()
attention.shape

(10, 10)

Of course, unless we are using a properly trained model, these will look quite random.

plt.matshow(attention)

<matplotlib.image.AxesImage at 0x1d78d10f588>

For a more pleasant experience, where we label the axis with the actual words that composed the sentence, we can do something like the following.

The predicted sentence would of course again be quite nonsensical since we did not train our model properly.

# Input, output and predicted sentences
print(input_sentence)
print(output_sentence)
print(decoded_words)

je suis saoul .
i m drunk .
['chances', 'runner', 'bear', 'interpreter', 'suddenly']

# Set up figure with colorbar
fig = plt.figure()
ax = fig.add_subplot(111)
cax = ax.matshow(attention, cmap='bone')
fig.colorbar(cax)

# Set up axes
ax.set_xticklabels([''] + input_sentence.split(' ') +
                   ['<EOS>'], rotation=90)
ax.set_yticklabels([''] + decoded_words)

# Show label at every tick
ax.xaxis.set_major_locator(ticker.MultipleLocator(1))
ax.yaxis.set_major_locator(ticker.MultipleLocator(1))

plt.show()

Vulcanus in Japan: Arrival

2020-02-29T00:00:00+00:00

The first of September of 2019. That was the designated date when all participants had to arrive in Tokyo. As I was planning to visit China as well, I decided to casually leave three weeks earlier, and then continue towards Japan on the 31st. After a brief transfer in Guangzhou, I headed towards the promised lands. One year in Japan. My dreams come true.

I had been to Japan before, 10 days and 2 months respectively, and I was dying to be able to go again. It seems like I always multiply the time of my previous stay by 6, so the next time come here, expect me to stay a whole 6 years! 😮 The arrival was magical. I did not know how much I missed Japan until I stepped out of the bus that had taken me from the airport to the center of Tokyo. Neon lights, warm humid air, the magical presence of both constant noise and music, coupled with the blissful calmness and peacefulness that only a serious and respectful culture like this one can offer.

Stepping into a Japanese train is an experience in itself as well. Note that there are two types of trains in Japan, rush-hour-trains and a little-less-rush-hour-trains. Both are full of people and advertisements hanging from the ceiling but are possessed by a strange, deep silence. Everyone is looking at their phones, a few are reading books, and some do nothing at all, but rare is the occasion when someone raises their voice. There is a third type of train though, which only exists in the inaka, the Japanese far-lands. On those trains, there are more ghosts than people on board.

An inaka train in Japan.

We headed to the hotel where all Vulcanusians were to stay the night (the hotel was quite nice!). My roommate was comfortably taking a nap in a yukata until my rude entrance brought him back to reality. A konbini dinner later, most of us went to sleep, as our schedule was full from sunrise on our second day. We were provided with an endless list of things that we should and should not do during our stay in Japan. Register at the town hall. Pay healthcare insurance. Write monthly reports. Do not be late for language school. Learn to “read the air”. Once we were all drowned in information, we were more than ready to meet our supervisors (as ready as one can be with jet-lag, an empty stomach, and the nervousness and excitement of realizing that we will stay 1 year in this alien country).

From here on, the experience of each Vulcanusian was very different. According to the keikaku (note: keikaku means plan), our respective supervisors (the person in charge of us at the company of our internship) was to take us to the town hall of the area where we were going to live. What was awaiting us there was a sea of procedures written in Klingon (otherwise known as Japanese). After that, I was introduced to my dorm, the place I was meant to call home for a year. Having heard that some people got dumped at the entrance of the town hall, left to battle their ways through the documents themselves, I do feel quite lucky that mine stayed with me and helped me with everything, all the way to setting up the apartment (electricity, water, etc).

Yokosuka is quite a cute town I have to say. It looks like a miniature version of a big city, has everything you could ask for, but smaller in size and with fewer people per square meter. I like it!

Yokosuka Chuo.

My room looks quite old and worn out, but is usable and comfortable. Having visited some other Vulcanusians at their dwellings in Edo, the differences were very noticable. They all had fancier rooms and usually more appliances and commodities. I win in square meters and size of my desk though. They also lack a certain charm that mine certainly does have (or at least this is what I tell myself to feel less jealous). I also had the chance to contact the previous Vulcanusian in Yokosuka, and got some essential items from him on day one: kitchen tools and bed sheets being the key here. I do not want to list bed-sheet shopping as my first independent activity in Yokosuka.

Day three was another briefing session with more information being thrust into our little distracted brains. Who can focus on those serious matters they may be talking about, knowing that Akihabara is only a few train-stops away? 🤔

From the fourth day onwards, Vulcanusians started attending language school. Keep an eye on the next episode to learn about our adventures in the famous Naganuma School!

Vulcanus in Japan: Application

2020-01-13T00:00:00+00:00

Vulcanus in Japan is a program organized by the EU-Japan Center for Industrial Cooperation, sending 30 European engineering / science students every year to Japan for a 4-month long intensive language course, followed by an 8-month long internship at a Japanese company.

I got into the program for the 2019-2020 session (I am writing this from my small apartment in Yokosuka, Kanagawa). Thinking back, I wish I had found more information on the internet regarding past experiences of applicants and the application process.

Before getting started, I would like to note that everything I state below are my own thoughts and opinions. They might not be true or accurate, but they reflect my own experience.

Experience and CV

Your background is something that you can’t really change anymore if you are applying now. However, it’s important to highlight the right points to make yourself more appealing to the selection committee.

One of the factors that I think is of utmost importance is to have international experience of some kind, for example an exchange program by your university. This is something that almost all of the 30 of us in my year have in common. If you have been to Japan before or have traveled a lot, make sure to mention it as well.

Other than that, this is a science and engineering focused scholarship, so proving that you have a strong technical background is very valuable. Published papers? Contributed to open source? Mentionable academic / personal projects? Work experience?

Finally, this is not at all a requirement, but I believe it is valuable to know some Japanese. More than half of the selected participants knew close to nothing when they came, but if you have already studied it for a while you can prove to the evaluators that you have a strong interest in the country. Knowing some basics before coming is surely helpful. You will be able to join the advanced classes at the language course and get around the country more easily. I arrived with N2 level and joined a business Japanese class. I had a blast!

As for how to structure and style your CV, there are many guides online. I usually follow a very basic template, you can check my CV in the top bar of this website for inspiration.

Motivation Letter

In your motivation letter you have to hit some basic points, which are mentioned in their “Info about Compulsory Documents”):

Your summarized background (also mention scholarships, awards, etc.)
Your current line of work / focus
Why Japan and why Vulcanus, both personally and academically (make sure to mention how you will learn a lot from the seminars, language course and internship)
What makes you special / a good fit for them

Of course, make sure your English is flawless. Show it to as many people as possible and ask for suggestions. After you feel its ready, wait a few days and read it again. Does it still feel good? Probably you will notice that some sentences can still be polished more. Writing a good statement is a long and tedious procedure. Ganbatte!

After being shortlisted, you have to write another letter to the company you want to apply to. Make sure to adapt it a little to highlight the points that make you a good fit for them, and show them that you did your homework on reading about them and their mission.

If you need an extra pair of eyes, you can send me yours through email, if I have time I will take a look!

Reference Letter

You need one or more strong recommendation letters. Try to find a professor that you are close to and who would write only in your best interest. Vulcanus is not the same as a graduate school, so I believe that more than ranking or prestige of the person who wrote it, honest and convincing words win the game here. Some of the things I believe they value the most are mentions regarding your teamwork skills, motivation and adaptability. The letter that my professor wrote for me had the following structure:

How we met, what interactions we had
My achievements and strong qualities from her point of view, while I was under her supervision
Assertion that I am a good fit for Vulcanus and Japan

Remarks

Applying for vulcanus is a rollercoaster experience. Don’t let it get in the way of your studies or work though! I certainly spent more time than I should researching about where I would live and how my life there would be, even when I didn’t even know if I was shortlisted.

I believe that the first stage (shortlisting) evaluates your general fitness for the programme, so if you have a relatively strong background there is not much that you should fear. Where things get tough is in the second stage. It’s much more random and depends on how well you actually fit to the company that you applied for. Also, every company uses different criteria to evaluate you (many will carry out interviews).

I would suggest choosing a company where you believe your background fits well. For example, I was interested in many of the available positions but chose as my first option one where I satisfied their optional language skills (Spanish and German).

When you get shortlisted, the committee suggests you some good matches according to your background. However, some people got suggestions for positions that didn’t really fit their interests so well. I was very satisfied with mine, so I can’t talk from first hand experience. I would suggest contacting them and asking for the reason why they chose that position for you. Don’t hesitate to change the suggested option for a different one if you feel uncomfortable though.

In the coming months I will write more posts about my experience as a Vulcanus participant, so check back from time to time!

Deep Turing OCR: A Summary of my Bachelor’s Thesis

2019-11-03T00:00:00+00:00

A short introduction to my final thesis project: performing OCR on hand-written Turing Machines with deep neural networks.

The Idea

I was attending a class on Models of Computation when the professor mentioned that it was very tedious to grade the final exams for this subject. One of the tasks (arguably the most important) is to write a Turing Machine by hand, capable of performing a specific operation (such as multiplication, division, exponentiation, etc). Each of these can have between 20 and 50 lines of state transition definitions, which look a bit like the following:

To evaluate the correctness of a Turing Machine, it is necessary to simulate it on a computer. The professor transcribed each Turing Machine manually into text, which could then be read by a simple simulator capable of checking if the machine is correct.

This immediately caught my attention, as I was currently hunting for an interesting thesis project. By using a neural network, it would be possible to convert image into text, which would greatly improve the speed at which one can grade Turing Machines. This project covered all the points I was looking for in a thesis project:

Specific enough to be unique (I haven’t found anybody else working on Turing Machine OCR).
Similar enough to other project (image to text, OCR) to be able to find supportive resources when needed.
Challenging enough for me to learn from it.
Related to my academic interests: Machine Learning / Neural Networks.

And so I ventured into building a system capable of inputting a scan of a Turing Machine, and outputting the text associated to that machine.

The code for this project can be found in this repository.

The Dataset

A scan of a Turing Machine may look like the following:

]

In total, in collaboration with my supervisor, we have gathered more than 30 exams, and about 1500 Turing Machine definition lines. Each of these lines had to be manually labelled. The label for each line is a string, containing around 15 to 20 characters on average (this took a very, very long time 😥).

This is what later will be used to train the OCR model.

The Model

To be able to go from scan to text, there are two steps to overcome. First, localization, which is finding the areas where the transition definitions can be found. Then, on each of those, OCR has to be performed to extract the text. Integrating both into a seamless, end-to-end system would of course be ideal, but such a challenge is out of the range of an undergraduate thesis program. I decided to perform localization using the open source tool Tesseract. It is mainly an OCR tool, but can also be used to predict bounding boxes around regions of interest. I created a simple script that uses Tesseract to spit out the coordinates of the regions of interest of any given Turing scan.

The model that I built performs character recognition on the individual, cropped lines. The structure used is a CRNN, whose details are described in this paper. Basically, its a CNN followed by an RNN. First, the important features of the different characters in the image are extracted, and lined up as a sequence. Then, this sequence can be read by the RNN, which at the same time outputs character predictions. The whole thing is then trained using the CTC loss.

The network layers.

The Complete System

The system should be comfortable to use by people from any background. It should also be quick and assistive. Thus, I decided to wrap the scripts and the model into a backend, which offers an API to a web-based frontend. The user interacts with the model through an intuitive (yet visually rather mundane) interface, which looks like the following:

The user uploads a scan of a Turing Machine definition. Then they press the ‘Get Boxes’ button to obtain bounding boxes around the text. Those boxes can be resized and deleted on demand in case the predictions contained mistakes (Tesseract isn’t exactly flawless when it comes to messy handwritten Turing Machines…). Moving forward, the user can request the model’s OCR predictions. These will be displayed on the right-hand side. When hovering over a box or over the predicted text, the corresponding partner will also be highlighted. Mistakes made by the model can be easily amended by editing the text. Finally, the data can be saved both as a dataset for further training, or as part of a text file for posterior simulation.

A summary diagram of the system, its modules and intercommunication.

The Results

The metric used to track performance was edit distance, which is the number of edits necessary to go from the predicted string to the target string. After training, the average edit distance was 1.2. This means that although some predictions are flawless, others have one or two mistakes. Considering that that the ambiguity of some numbers and symbols is high for some types of handwriting, it’s quite an acceptable result. I am currently working on improving the model and dataset to drive this number down even further. Keep an eye on future posts!

Final Thoughts

It was a very valuable experience and I am very happy to have been able to work on this project. I leaned not only to build and train a rather complex network, but also to embed it into an integrated system. The fact that the end result is of practical use, even if just for a very small niche of people, makes it that much more satisfying. I faced uncountable bugs and errors, which took me an uncountable number of hours to hunt down and fix. I attribute this to poor debugging techniques and poor code structure. This experience has hopefully taught me to be more careful and organized, as well as be more thorough with my debugging.

An RNN that learns to talk like you

2018-11-24T00:00:00+00:00

I would like to share a personal project I am working on, that uses sequence-to-sequence models to reply to messages in a similar way to how I would do it (i.e. a personalized chatbot) by using my personal chat data that I have collected since 2014. If you would like to learn more about this type of model, have a look at this paper. I am working on two versions, one at the character level and one at the word level. The full notebooks with code can be found here: char-level, word-level.

Usually, a sequence-to-sequence model is used for translation tasks. The encoder reads an input in a specific source language, and passes the encoded inner state to the decoder, which then produces text in the target language. I was curious if I could apply this model to message generation. I try to encode an input message and then produce an answer / reply by using the decoder.

I will show important parts of the code here, but if you get lost or don’t know where a variable comes from, please check the notebook.

Word level vs character level

Each one has benefits and drawbacks, and I am studying which one would work best for this task.

Word level allows to increase the amount of information you can put into a sequence. Since each word is one token, you can put up to seq_length words into a sequence, as opposed to only seq_length characters if you are working at character level. It also allows the model to work at a higher level of abstraction. One drawback is that we are ‘accidentally’ generating tokens for all misspelled words in the dataset (I have tons of them), as well as all variarions of similar words. For example, me and my friends frequently use ‘xD’, which can also be ‘xDD’ or ‘xDDD’. Cleaning this is a very difficult task.

If you work at character level, you are considerably reducing the vocabulary size. While I have more than 15.000 different words in my dataset, I have only 290 different characters in use (including emojis! :smiley:). Since we need to one-hot encode our tokens (except if we use embeddings, which I will mention later), this helps to drastically reduce memory usage. Missspelled words are automatically taken care of in this case since they are much less frequent than the correctly spelled versions. The model will thus learn to spell them correctly.

I will follow the word level code, but feel free to look at the notebook for the character level code to see the differences.

The correlation assumption

In order to teach the neural network about how someone usually talks, an assuption needs to be made regarding correlation of the data you feed it. It is assumed that previous messages are related to how you replied. Although this is normally the case, a number of problematic exceptions need to be taken into account:

Someone could be replying to a topic of a previous conversation. In this situation, the immediately previous message is not related and breaks our assumption.
Someone could be introducing a new topic, starting a new “chain” of relationships. Of course, this first message is then only related to succedding messages, and not to the previous ones.
We sometimes reply to very old messages or reply to several things in a row, having two parallel chains.

Here, I decided to ignore these edge cases and hold the assumption.

Preparing the dataset

The chatbot notebook I provided expects a json file, which should contain a list of already separated tokens. So let’s preprocess the data and obtain this json. These preprocessing steps can also be found in my data preprocessing notebook.

First we import the things we need and define some helper functions to load and save files.

import regex as re
import numpy as np
import matplotlib.pyplot as plt
from google.colab import drive
drive.mount('/content/gdrive')
import json

def load_doc(filename):
	# open the file as read only
	file = open(filename, 'r', encoding='utf-8')
	# read all text
	text = file.read()
	# close the file
	file.close()
	return text

# save tokens to file, one dialog per line
def save_doc(lines, filename):
	j = json.dumps(lines)
	file = open(filename, 'w')
	file.write(j)
	file.close()

While using Google Colab, you can access your drive contents by using google.colab.drive. Here I am loading a text file I have stored in mine, so please replace this with your own chat dataset.

text = load_doc('/content/gdrive/My Drive/Projects/datasets/whatsapp_dataset/conversation.txt').lower()

I am working with text that looks like this:

11/16/14, 09:54 - raul: Te tendre que hacer una lista de capítulos no?
11/16/14, 10:01 - mario: De los sueltos?
11/16/14, 10:07 - raul: Si
11/16/14, 10:07 - raul: Hay 12 de la trama principal

So lets create a list of lines and clean it up a bit:

# Split into messages and remove date header
lines = re.split(r'\d+\/\d+\/\d+,\s\d+:\d+\s-\s', text)
lines = lines[1:]
# Replace multiple newlines by just one newline
lines = [re.sub("(\\n)+", '\n', line) for line in lines]
# Delete trailing newlines
lines = [re.sub("\n$", '', line) for line in lines]

You can check your max and average message length, if you are curious.

# Whats the maximum message length?
np.max([len(line) for line in lines])
# And the average message length?
np.mean([len(line) for line in lines])

Now we can separate our data into tokens. The first token of each line is going to be the person to whom the message belongs to. Any non alphanumeric character is considered a unique token (punctuation, emojis, etc). Note that [\p{L}] matches any codepoint of the category ‘letter’. As opposed to [a-z], it will also match non-english characters like letters with accents or completely different alphabets like Japanese (おはよう！). Only consecutive ‘letters’ are grouped together into one word. Something like good-looking will be separated into three tokens. This gives the model some flexibility by giving it smaller building blocks.

# We gotta separate the text into tokens.
# By convention, the first element of each sequence is the name of the person saying it
splitted_lines = []
for i,line in enumerate(lines):
    match = re.match('([a-z]+):', line)
    # Ignore messages without name tag
    if not match: continue
    name = match.group(1)
    line = re.sub('^[a-z]+: ', '', line)
    splitted_line = re.findall('(?:[\p{L}]+)|\S', line, re.UNICODE)
    splitted_lines.append([name, *splitted_line])

Now comes a challenging part. How do we deal with consecutive messages from the same person? There are many ways in which one could solve this. Since I want to teach the model how I reply to a message given an input, I have to group all my messages into one, and consider it a reply. To keep the idea of separation between these now grouped messages, I will add newlines (\n) between them. The neural network will then also learn to put newlines into his responses, and we can manually separate this response into several messages again afterwards.

Onother more sophisticated way could be to separate it into blocks of close in time messages (put messages together that are sent shortly after one another, and separate into a different block those who are more distant, since they could be unrelated). However, these and other methods need to solve a more complex correlation assumption. Your reply will most likely be correlated to all and not just the last message of your partner. Also, your next messages will also likely be correlated to both your previous message and your partners messages. I am open to suggestions on how one could model this correctly.

# Join all consecutive messages from the same person into one big message.
grouped_lines = []

name = splitted_lines[0][0]
grouped_lines.append(splitted_lines[0])

for i in range(1, len(splitted_lines)):
    if splitted_lines[i][0] == name:
      grouped_lines[-1].append('\n')
      grouped_lines[-1].extend(splitted_lines[i][1:])
    else:
      name = splitted_lines[i][0]
      grouped_lines.append(splitted_lines[i])

Finally I separate them into two, one for inputs and one for outputs, and I save the result as json.

# Here i'm splitting the lines into input that I receive and reply that I give
mario_response = [line[1:] for line in grouped_lines if line[0] == 'mario']
mario_input = [grouped_lines[i-1][1:] for i in
               range(len(grouped_lines)) if grouped_lines[i][0] == 'mario']
save_doc(mario_input, '/content/gdrive/My Drive/Projects/datasets/whatsapp_dataset/mario_input.txt')
save_doc(mario_response, '/content/gdrive/My Drive/Projects/datasets/whatsapp_dataset/mario_response.txt')

Chatbot phase 1: getting ready

Now comes the code regarding the chatbot per-se. Lets load the necessary stuff, define hyperparameters and have a peek at out data.

# Import necessary packages
import regex as re
import numpy as np
import matplotlib.pyplot as plt
from google.colab import drive
import keras
from keras.models import Model
from keras.layers import Input, CuDNNLSTM, Dense, Embedding
import itertools
import json

# A helper function to load a file as utf-8 text
def load_doc(filename):
	file = open(filename, 'r', encoding='utf-8')
	text = file.read()
	file.close()
	return text

# Our hyperparameters, can be tuned at liking
batch_size = 64
latent_dim = 256
seq_length = 50
num_lines = 1000 # this is per person, so total is double

drive.mount('/content/gdrive')
input_messages = json.loads(load_doc('/content/gdrive/My Drive/Projects/datasets/whatsapp_dataset/mario_input.txt'))
response_messages = json.loads(load_doc('/content/gdrive/My Drive/Projects/datasets/whatsapp_dataset/mario_response.txt'))

# Check that your data is looking good
print(*input_messages[:3], sep='\n')
print(*response_messages[:3], sep='\n')

It is possible to use generators to solve this issue, but for now we will one-hot encode all our target sequences (more on this later). For this reason, the amount of lines you can work with may be limited by the memory available. Cut your lines accordingly.

# Cut your dataset to include only num_lines lines.
input_messages = input_messages[:num_lines]
response_messages = response_messages[:num_lines]

This neural network needs an upper bound for the length of a message. All messages with length above should be cut accordingly. You can study how long your average message is by running the code below. Although here we take the first seq_length words, some more sophisticaded way to trim the messages could be used. For example, try to use the last seq_length messages for the input. They might have more correlation to your reply than the beginning.

# Check how long your messages are
max = 0
mean = 0
count = 0
for line in input_messages + response_messages:
  mean += len(line)
  if len(line) > seq_length:
    count += 1
  if max < len(line):
    max = len(line)
mean /= len(input_messages + response_messages)
print(f"Your longest message is {max} words long. The mean is {mean}.")
print(f"By using a seq_length of {seq_length}, you are cutting {count*100/(num_lines*2)}% of your messages.")

Trim your lines once you decided the length. Since we will append either a START or an END token to our response lines, we need to make sure we leave space for that. Note how any lines below seq_length will be ignored with this indexing expression. The reason why we need these extra tokens is because we will use a technique called teacher forcing.

Source: https://blog.keras.io/a-ten-minute-introduction-to-sequence-to-sequence-learning-in-keras.html

The decoder will get as input the target we want to generate, including one START token at the beginning. Given this, it must learn to generate the actual first character. Once it did that, we feed it that first character as input and let it generate the second one, etc. Finally, it must learn how to stop by producing the STOP token.

# Trim inputs to seq_length and responses to seq_length - 1
# This way we make space for START or END tokens
for i in range(num_lines):
    input_messages[i] = input_messages[i][:seq_length]
for i in range(num_lines):
    response_messages[i] = response_messages[i][:seq_length - 1]

# We'll use '\t' as START and '\r' as END token, since \n could be part of the message
for i in range(len(response_messages)):
  # Add starting '\t'
  response_messages[i].insert(0, '\t')
  # Trim trim out two more characters (to make space for the new tokens) and add \r
  response_messages[i].append('\r')

Chatbot phase 2: Tokenization

A neural network needs numbers to run, so we need to turn our characters into numbers. Each unique character (even emojis!) will receive a unique number. Since not all lines have seq_length length, we need to fill them in order to obtain a numpy array. This filler will be \v (any other random character is also okay, as long as it’s not already in your dataset).

# Create translation dictionaries
# I will use \v as a filler for lines with less than seq_length words, it will get index 0.
# It was chosen at random, we just need something that is not part of our normal vocabulary.
words = list('\v')
for line in (input_messages + response_messages):
    for word in line:
        if not word in words:
            words.append(word)

word_to_ix = dict((c, i) for i, c in enumerate(words))
ix_to_word = dict((i, c) for i, c in enumerate(words))

vocab_size = len(word_to_ix)

Now we can transform our lines into numpy matrices by using these dictionaries.

# Input of encoder is input lines
encoder_input_data = [[word_to_ix[word] for word in line]
                      for line in input_messages]
encoder_input_data = np.array(list(
    itertools.zip_longest(*encoder_input_data, fillvalue=0)), dtype=np.int16).T

# Input of decoder is response lines without END token ('\r').
decoder_input_data = [[word_to_ix[word] for word in line[:-1]]
                      for line in response_messages]
decoder_input_data = np.array(list(
    itertools.zip_longest(*decoder_input_data, fillvalue=0)), dtype=np.int16).T

# Output of decoder is response lines without START token ('\t').
decoder_target_data = [[word_to_ix[word] for word in line[1:]]
                       for line in response_messages]
decoder_target_data = np.array(list(
    itertools.zip_longest(*decoder_target_data, fillvalue=0)), dtype=np.int16).T

# Only target sequences need to be one-hot encoded, since we are using an embedding
decoder_target_data = keras.utils.to_categorical(decoder_target_data,
                                                 num_classes=vocab_size)

# Sanity check: shapes are looking good.
print("Your data looks like this:")
print(f"encoder_input_data shape: {encoder_input_data.shape}")
print(f"decoder_input_data shape: {decoder_input_data.shape}")
print(f"decoder_target_data shape: {decoder_target_data_h.shape}")

Chatbot phase 3: training

Finally, we can define our model. We will use an embedding because our vocabulary size is very big. This is the reason why we did not need to create one-hot encodings of our data, except for the targets. This embedding will transform our integer values into a vector of latent_dim entries, which should be capable of representing all our different words. Similar words (and thus also typos and similar expressions) will have similar vectors, thus aiding us to clean them out a little bit.

#Encoder
encoder_inputs = Input(shape=(None,))
embedding = Embedding(vocab_size, latent_dim)
embedded_enc_inputs = embedding(encoder_inputs)
encoder = CuDNNLSTM(latent_dim, return_state=True)
_, state_h, state_c = encoder(embedded_enc_inputs)
# This is the encoded information we will pass over to the decoder
encoder_states = [state_h, state_c]

# Decoder
decoder_inputs = Input(shape=(None,))
embedded_dec_inputs = embedding(decoder_inputs)
decoder_lstm =  CuDNNLSTM(latent_dim, return_sequences=True, return_state=True)
decoder_outputs, _, _ = decoder_lstm(embedded_dec_inputs,
                                     initial_state=encoder_states)
decoder_dense = Dense(vocab_size, activation='softmax')
decoder_outputs = decoder_dense(decoder_outputs)

# The final model
model = Model([encoder_inputs, decoder_inputs], decoder_outputs)
model.summary()

With the default values I set above, we are getting around 3 million trainable parameters. This value is quite decent and should allow us to learn quite a few relationsips between sequences of words. You can tune the latent_dim to influence this number and change the capacity of the model.

Finally, let’s train it on your data.

# Run training
generator = DataGenerator()
model.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'],)
model.fit([encoder_input_data, decoder_input_data],
          decoder_target_data,
          epochs=100,
         )

# Save model after training
model.save('/content/gdrive/My Drive/Projects/weights/mario_chatbot_v1.h5')

Chatbot phase 4: playing with our model

In order to sample things from our now trained model, we need to modify a bit the decoder. Don’t worry, we’re just creating a different interface for it, the trained weights will stay.

We need to get rid of the teacher forcing part now, and ‘release’ the state_inputs so that we can feed our own. We also want the decoder to just create one token at a time, so that we can feed in the token it just created as input in the next iteration.

# Define sampling models
encoder_model = Model(encoder_inputs, encoder_states)

decoder_state_input_h = Input(shape=(latent_dim,))
decoder_state_input_c = Input(shape=(latent_dim,))
decoder_states_inputs = [decoder_state_input_h, decoder_state_input_c]
embedded_dec_inputs = embedding(decoder_inputs)
decoder_outputs, state_h, state_c = decoder_lstm(
    embedded_dec_inputs, initial_state=decoder_states_inputs)
decoder_states = [state_h, state_c]
decoder_outputs = decoder_dense(decoder_outputs)
decoder_model = Model(
    [decoder_inputs] + decoder_states_inputs,
    [decoder_outputs] + decoder_states)

Now lets define the function that will run this new decoder interface and create an output sequence given an input sequence. This function is the tool we have to ‘talk’ with out chatbot. As long as you use a sequence of words that are present in the word_to_ix dictionary, he will reply.

def decode_sequence(input_seq, t=None):

    states_value = encoder_model.predict(input_seq)
    target_seq = np.zeros((1, 1))
    # Populate the first character of target sequence with the start character.
    target_seq[0, 0] = word_to_ix['\t']

    # Sampling loop for a batch of sequences
    stop_condition = False
    decoded_sentence = ''
    iteration = 0
    for _ in range(seq_length):
        output_tokens, h, c = decoder_model.predict(
            [target_seq] + states_value)

        # Sample a token
        sampled_token_index = np.argmax(output_tokens[0, -1, :])
        sampled_word = ix_to_word[sampled_token_index]
        decoded_sentence += " " + sampled_word

        # Update the target sequence
        target_seq = np.zeros((1, 1))
        target_seq[0, 0] = sampled_token_index

        # Update states
        states_value = [h, c]

        if sampled_word == '\r': break

    return decoded_sentence

To create the input sequence, you need to turn a string into a vector of numbers. Let’s create a helper function for this.

def create_input_sequence(line):
  input_seq = np.zeros((1, seq_length,), dtype=int)
  splitted_line = re.findall('(?:[\p{L}]+)|\S', line, re.UNICODE)
  print(splitted_line)
  for i in range(seq_length):
    if i < len(splitted_line):
      input_seq[0][i] = word_to_ix[splitted_line[i]]
    else:
      input_seq[0][i] = word_to_ix['\v']
  return input_seq

Finally, you can run the following to get an output from yout bot:

input = "hola"
print(decode_sequence(create_input_sequence(input)))

Thoughts

By running 100 epochs, we are clearly overfitting our data. Also notice that I did not use any validation nor regulation to the model. By overfitting it, we can check if we did things correctly or not. If you feed it an input sequence present in your data set, the model should reply with the corresponding target sequence. If this is not the case, something went wrong and you know you can start bug hunting. Once this is out of our way, we can start playing with regularization. Finding a good balance is crucial since the answers the model gives you can change drastically depending on it.

You can also play around with the probability distribution. For example, you can take a random choice with output_tokens[0, -1, :] probabilities, isntead of taking argmax. This will add some randomness to the model.

Thank you for following me to the end! Feel free to leave a comment.

Bias versus Variance Explained

2018-10-14T00:00:00+00:00

As most things in this blog will be, this entry attempts to explain bias vs variance the way that I found it to be the most clear and enlightening.

Fitting a model on simple data

Let’s start by building an intuition and seeing the relationship between bias/variance with over/underfitting. Machine learning models have an underlying complexity level that allow them to be more or less flexible while trying to represent the dataset. Let’s imagine a regression model trying to predict a dataset with just two values per row, X and Y. Y is the real valued label that we are trying to predict based on the value of X.

Let’s assume that our ground truth (what we are ultimately trying to predict) looks a bit like the following:

Now, most datasets can never truly represent the ground truth perfectly. In our case, we will take a sample of points, use it as a training set, and build our models with it. It could look like the following:

Two predictive models have been applied on the dataset. Each has its own characteristics and fits the data in a different way. Neither the simple nor the complex model are very correct, since one of them generalizes too much while the other is too precise.

Simple model

The simple model is linear, it can only fit data by drawing a line. Its precision on the training data is rather low. If this model could talk, it would say that the data has a lot of noise, and the underlying ground truth is in reality the line. That top right point is just an unlucky outlier. This model is clearly underfitting.

If we take a different sample of points (of the same ground truth) and train again, it would look almost the same. The same line can explain many different samples of the ground truth.

The error on this sample is quite high, since not many points are close to the line. But if we test the model with a test set, the error will be similar. The variance over different samples of the same ground truth is low. It also means the general error the model is a tradeoff to be able afford this low variance. Consequently, the bias of the model is high.

Complex model

The complex model is fitting a very high degree polinomial on the data, perfectly explaining each point. If this model could talk, it would be saying that the ground truth is a quite complex polinomial. This model is clearly overfitting.

If we take a different sample of points and train again, it would look very different. This polinomial perfectly fits the current sample, but doesn’t fit at all other samples of the same ground truth.

The error on this sample is very low, since all points sit on the line. This means its bias is low. But if we test it by using a test set, the error would be very high (even higher than the linear model!). It’s not biased towards some specific distribution or shape (like the line does); it will do whatever it takes to be unbiased and train itself with maximum precision. However, the variance of this model is very high. It needs to change a lot to be able to adapt to different samples.

Perfect model?

The arguably best model for this data would be one that takes a slight curve upwards, following the trend we could see in the first picture. However, in a real world dataset we don’t know what the ground truth looks like, so it’s extremely difficult to know what model works best.

Bias / variance tradeoff

The way people usually deal with this problem when they have to build complex models is to keep an eye on both trainig error and validation/test error.

While they both fall proportionally, the model is still underfitting and could do better. However, once the test error behaves differently than the training error (usually going up again), it’s time to stop training because you will be overfitting from now on.

The source that helped me most was this excellent post by Scott Fortmann.

Please feel free to add comments if you liked this post or you think something is wrong or should be clarified.

How to write an LL(1) parser

2018-06-12T00:00:00+00:00

Let’s have a look at how to build an LL(1) parser and what considerations we need to make when writing an LL(1) grammar.

For this post I assume basic knowledge about grammars and languages, but I try to be gentle.

Motivation

The lack of clear and concise information about this topic lead me to think that it could be useful for many people to have a small guide for it. What are the key constraints that an LL(1) grammar has? How to actually write a parser for a grammar you designed? The answers to these questions are actually quite simple, yet most sources I had looked up seemed to be either unclear, not very well focused, or difficult to find.

A secondary reason is that, having just finished my compiler course, I would like to write down some of my knowledge before I forget it all again after the summer break (summer breaks have this powerful and mysterious brain erasing capability, and I have yet to find a good way to fight against it). LR parsers are more properly covered on the internet and resources about this are easier to find.

LL Grammar and Language

So what is an LL grammar anyway? It’s a subset of deterministic context-free grammars, specifically those that can be parsed by an LL parser. This parser reads the input from Left to right, and constructs the Leftmost derivation of whatever you are parsing. Note how this definition of LL grammar may sound a little bit obvious (“can be parsed by LL parser”). That is because giving a more precise definition is actually quite complex, as it is not easy to pinpoint which languages fulfill this requirement and which not. I won’t go into detail for this, since it’s not the purpose of this post.

An LL language is one that has an LL grammar to define it. You can also use a non-LL grammar to define an LL language, but this is not convenient. Also sometimes you have a language that may not look like LL at first, but with a few modifications to its grammar you may find out that you can make it LL.

Our goal will be to define a grammar for our language that happens to be LL, such that we can benefit from the properties of its parser.

LL and LR Parsers

An LL parser is predictive. That is, if you follow a set of conditions such that your grammar is LL, your parser will be really fast because it never has to do backtracking and is extremely simple. This simplicity is what it will also allow us to code a parser by hand (coding an LR parser by hand can be an extremely tough task, and everyone relies on existing tools to do this). One of the common ways to implement these parsers is by using recursive descent, which we will use in this post.

By the way, the number within parenthesis in LL(k) or LR(k) means the number of lookahead tokens that the parser is able to see to make the next decision. In our case, LL(1) means our parser can only see the next token before parsing it.

The big brother of an LL(1) parser is an LR(1) parser. Let’s set things clear here before we move on:

LR(1) parsers can parse ANY deterministic context free language (DCFL). It might be difficult to find a grammar for it, but it is theoretically possible.
LR(k) parsers, for any k, are just as powerful as LR(1). The grammar needed for it just becomes more complex, verbose or difficult to find.
LR(k) parsers are more powerful because they are aware of all preceding tokens in the parsing procedure, unlike LL(k) which only knows about the current and next token.
LL(k) parsers generally don’t use backtracking; if they did, it would defeat their purpose, which is running fast and being really simple.
LL(1) parsers can ONLY parse LL(1) languages.
LL(2) parsers can parse LL(1) and LL(2) languages.
There is an infinite sequence of subsets of DCFLs as we increase the k in LL(k), none of them really ever reaching the whole set.
Thus, an LL(k+1) parser can parse strictly more grammars than an LL(k) parser, and strictly less than an LR(1) parser.

How to make an LL(I) grammar

If you have a language and want to attempt to make an LL(1) grammar for it (thus demonstrating that what you had is indeed an LL(1) language), there are several constraints that you have to follow when defining it. If all constraints apply, you successfully obtained an LL(1) grammar and you can move on to the next section to build a parser for it.

The constraints are:

If you have several rules \( X \rightarrow Y, X \rightarrow Z \), then \( First(Y) \cap First(Z) = \emptyset \)
If you have several rules \( X \rightarrow Y, X \rightarrow Z \), and \( First(Z) \) contains \( \epsilon \), then \( First(Y) \cap Follow(Z) = \emptyset \)
No left recursions: \( X \rightarrow X… \)
If you have several rules \( X \rightarrow Y, X \rightarrow Z \), then \(Z\) must not be non-false (i.e. must be able to return false)

The intuition behind these rules boils down to allowing the parser to make the correct decision with just one lookahead token. If there are several rules that start the same way and only differenciate themselves in the last token, the parser cannot know which rule it is until it is too late. If it took the wrong rule at the beginning, it can’t go back (thats the point, remember? no backtracking!). So for example, the second of our rules says that the First set of two right hand sides that share the same left hand side must have no common tokens. If they had any token in common, the parser wouldn’t know which of the two rules to use when it sees that token. Similarly, if the first of those rules is empty, then whatever comes after it (Follow set) also has to be different from the first set of the other rule.

As you will see below, each rule will become a function. Whenever a rule has a non-terminal symbol on the right side, we will call it’s function within our function. If we have left recursions, you can see how it will produce an infinite loop of function calls. That’s the reason behind the third condition.

Lastly, the non-false condition is there to avoid some similar situations to the first two.

Some of these conditions can be derived from the others, but for the sake of clarity and simpler identification we will use them.

How to make an LL(1) parser

Let’s suppose we have a grammar that allows words like these:

{a = 1;}
{if (a) then b = 5;}
{a = a + b - c; a = 4;}
{if (a) then if (b) then c = 0;}

Some words that should not belong are:

a = b  # No curly brackets
{a = b}  # No semicolon
{a + b = c;} # Assignments don't have value
{if x then 5 = c;} # Numbers can't be assigned

Now it’s our task to use the conditions defined above to actually write an LL grammar. Turns out one good grammar is the following:

program -> { statement_list } eof
statement_list -> statement statement_list
statement_list -> empty
statement -> id = expr ;
statement -> if ( expr ) then statement
expr -> id expr_tail
expr -> num
expr_tail -> + expr
expr_tail -> - expr
expr_tail -> empty

Lets write a python parser that reads the input and tells us if it belongs to our new fancy language or not.

Each rule will be one function. The function will return true if the rule could be applied, and false if not. Inside the function, the right hand side of the rule will be sequantially parsed. If a non-terminal is encountered, it’s function is called. For terminals, they will be compared with the current lookahead symbol. If the first function or the first terminal symbol comparison is false, then the function returns false. If, however, the first symbol(s) already returned true and we advanced in the rule parsing procedure, a function or comparison returning false will imply a parsing error. Let’s have a look with an example.

The first function that our parser will call is the one we usually call “program”, which basically is a function that will tell us if the word if part of the language or not. It’s the function associated to our first rule in the grammar. It will check that the word starts and ends with a curly bracket, and call statement_list to make sure that the thing inside is indeed a list of statements.

program():
    if not read("{") return False
    if not statement_list() raise ParseError
    if not read("}") raise ParseError
    return True

If the first token is not an opening curly bracket, program() will return false, saying that what comes now cannot be generated with this rule. It could be that our grammar has more than one starting rule, and we would then go ahead and check those.

However, if the first token matches, it means that undeniably this is part of our program rule. If anything goes wrong while we continue parsing, we can safely assume that our input is not part of the language. We can be sure because our grammar is an LL(1) grammar and fulfils the above described conditions. There cannot be any other rule that starts with an opening curly bracket at the same level as the “program” rule.

This is the essence of LL grammars and parsers. Go ahead and read the last two paragraphs again, and let it sink in. Once you understood this concept, writing the other functions becomes trivial.

Let’s have a look at another function.

    def statement_list(self):

        # statement_list -> statement statement_list
        if self.statement():
            if not self.statement_list(): raise ParseError()
            return True

        # statement_list -> empty
        else: return True

Here we are saying that if we find a statement, then we better find a statement_list next, otherwise something is wrong. If we didn’t find a statement in the first place, we return true because our statement list can be empty as well. In this particular case, statement_list will never return false nor raise a ParseError, but the statement function it calls could. We still write everything out for the sake of completeness. All other functions are written in the same manner.

Whenever we have a terminal symbol in our rule, we gotta shift. That’s what the read function is for. It knows where we are with the position attribute, and attempts to shift away the characters it receives as parameter.

The full program is implemented in this github gist here. Try it out by running python3 ll1_parser.py "someword", and it will tell you if its part of the language or not!

I’m starting a blog!

2018-06-07T00:00:00+00:00

So, I decided to start a blog. I always wanted to maintain one but the work that it would involve, together with the apparent low reward, always kept me away from doing so.

There are many reasons that could motivate someone to start a blog. Here I want to list a few that were the most influential for me.

Training your writing skills

Not many people realize how important it is to be able to transmit written information in a clear and effective way. It’s an invaluable skill and it should not be underestimated. If you are very knowledgeable but cannot transmit your thoughts very well, how will people acknowledge your potential?

Persuasion is one of the most powerful tools of a good communicator. Finding the right ideas and conveying them in the right way can get you very far. You can push things the way you want them to move. You can influence your environment. You can walk the path you want to walk.

Making new connections

I am eager to find new people with similar interests to mine. The same way I enjoy commenting on other people’s blogs and engage in stimulating discussions, I would love to someday receive comments from other readers and start to establish a network of valuable people.

Building a curriculum

In your blog you can display your knowledge and experience. Future recruiters and scouters will have a better image if you since you appear to have more knowledge about a certain topic and know how to explain things to others.

Forcing yourself to think more

When you have a new idea or understand a new concept, it always feels as if everything makes complete sense and the concept sits fully in your brain. In reality, this is rarely the case. As you continue thinking about it or attempt to explain it to someone else, you realize how full of holes the idea is.

By writing these things out and explaining them, it forces to think it through and thus helps you to solidify your knowledge. I would like to fill my knowledge wholes and learn more about the things I love by forcing myself to write about them.

Be more organized, improve your lifestyle

Building healthy habits can get you in the mood to keep doing so. There is some kind of momentum effect to healthy habits that seems to be able to carry you through.

The days I do exercise, I immediately feel more motivated to continue doing productive and healthy activities. I will usually jump right into a study or reading session. I am looking forward to getting the same result from blogging.