Finding Similar Text with Machine Learning and Natural Language Processing

I've been working with a client on analyzing some text documents and wanted to share a bit of what has been working for us. I can't share the data or the exact project details but it entails, finding similar text documents from a large collection of other documents a specific query example. Imagine searching a database of company statements, product descriptions, articles, contracts, emails, support/trouble tickets, etc. not by keyword but by 'meaning' and 'similarity'.

For this article I thought about using the Enron email or a financial filings dataset but wanted something with more general interest and relatability so instead found something more fun and accessible on Kaggle. The Wikipedia Movie Plots dataset has ~34k records with movie titles, release year, genre, etc. and most importantly for us 'Plot' descriptions. It seems like the perfect dataset set to experiment with and to use when exploring similarity/semantic search engines, content based recommenders, movie plot classifiers and many other interesting projects. In this example we're going to use the data in the plot field to find other 'similar' plots.

Keep in mind that were going to look for similar plot (write ups) and not similar movie styles, or similar genres, etc. That is, a (simple) plot, "boy meets girl, boy loses girl", could be rendered in many different styles and genres and with many different character archetypes. That said, the plots were written by people after having seen the movie and people will tend to include specific details, such as actors, in the 'plot' so they are not completely independent of other characteristics of the movie ... which can make for some other interesting future experiments such as can we predict the genre or year from the plot, etc.

Business Goal and Metric

Analyzing the plot field highlights the first issue any machine learning project: Identifying the business goal and finding an appropriate metric. This is the heart of every project and an important and often tricky first step. This is particularly challenging in similarity or clustering based (ie. unsupervised) experiments as there is no canonical correct answer and the utility and validity is very context dependent.

The point of this article is to give a high level overview of some techniques and approaches not to propose as specific solution to a specific problem ... so, I'm just going to punt on it here and say that my business goal is to 'explore similarity in various ways' and my metric is 'do I get answers that feel right'. If this were are recommender system we might want to measure if the similar movies were on users' 'favorites' list or how often people added recommended movies to their queue and so on. If you have a more specific business need I'd be happy to discuss it.

Feature Generation

Once you have a goal and metric in mind the next step is to develop ways of generating features that you feel may be useful to achieve those goals. Simply put, the algorithms need numbers, we have text. What numbers are we going to generate from our text? For example, we could use the length of each plot write up, the number of vowels, the number of large words, etc. or more advanced techniques like TF-IDF or neural net transformers. Some of these features will work better than others in different applications so feature selection is part art and part science.

For this example I could have used a classic technique, TF-IDF, and perhaps we'll revisit that in a future article, or a hot new BERT based model. I chose instead to use the standby Universal Sentence Encoder (USE) because it is easy to set up and use and works really well. TF-IDF is probably easier if you don't have any TensorFlow experience and a fine-tuned BERT based model would probably work better if you wanted to invest the time but USE is a great place to start and in a few lines of code you can get a meaningful numeric 512 dimensional representation (embedding) of short text passages.

import tensorflow_hub as hub

embed = hub.load("https://tfhub.dev/google/universal-sentence-encoder-large/5")
embeddings = embed(my_text)

Measuring Similarity with Cosine

Ok, great, so I have a 34886 x 512 matrix now what? Well the idea with turning the text into an embedding vector is that the embedding now represents a point in a semantic space and other points 'near' it represent other similar embedded items (plot descriptions).

The next decision is to formalize what we mean by 'near'. In two or three dimensional spaces we have intimate ingrained knowledge and a firm concept of 'nearness' and we call that the Euclidean distance. Unfortunately Euclidean distance does not work well in high dimensional spaces because of the Curse of dimensionality. The bottom line is that we'll be better off using angular (cosine) distance in high dimensions.

Luckily that is easy to do with Python and scikit-learn. Given a point/embedding represented by v and a the matrix of embeddings we can calculate all the angular distances (similarities actually) and then find the top n similarities with this rough pseudo-code

from sklearn.metrics.pairwise import cosine_similarity

d = cosine_similarity(v, embeddings).ravel() #find similarity and flatten results
a = d.argsort()[::-1] # find the indexes of the sorted similarities and reverse them
return a[:n] # return the top n

Results

Using these embeddings lets take a look at a couple of examples from movies that I have watched and enjoyed 'There Will Be Blood', 'The Big Lebowski', and 'Monty Python and the Holy Grail'. Take a moment to think about how you'd describe their plots.

The tables below give the 10 most similar plots to each of the movies based on the cosine similarity of the USE embeddings. Note the plots here have been truncated to fit in the table. The index isn't too useful if you don't have the exact shuffling of the dataset that I used but it helps me keep track of the movies. Similarity relates to the cosine of the angle between the two embeddings/vectors. An angle of 0 has a cosine of 1 so that is most similar. IOU is intersection over union which I added as a measure of the commonality of the words in the plot descriptions between two movies. Genre is specified in the dataset and is a very messy feature. Though the genre could be useful for some experiments there are ~2265 unique values and would need to be cleaned up. Title is self explanatory and the Plot column has the first few words of the plot description so we can get a feel for the text.

I don't know the other movies but note that most have to do with family, land and greed and there are no romance or space related movies. Do you know the movies? Do you feel they are similar in some ways?

I've seen 'Inside Llewyn Davis' but not the others. The plots seem to be related to 'lazy' or 'unsuccessful' people. Also note the Llewyn Davis is shows up twice even though this is supposed to be a de-duped list. I found several duplicates (not just duplicate titles) while exploring dataset with the embeddings.

These all seem to relate to King Arthur and Camelot and none seem to be (absurdist) comedies and there are no westerns or space adventures.

Dimensionality Reduction

It can often be useful to visualize our collection to get a feeling for how the items 'cluster' or relate to each other. The embedding vectors we used have 512 dimensions so to visualize them we're going to need to reduce them to 2 or 3 dimensions. The challenge is doing that while keeping as much useful information as possible.

Principal Component Analysis (PCA)

A classic approach to dimensionality reduction is Principle Components Analysis (PCA) which we'll use to find two axis through the data that show the most information (variance). Imagine trying to draw a loaf of French bread. You can rotate it to find an angle that shows the most information and you are likely to chose an angle that shows the full length and width which sacrifices the information in the height. You are less likely to draw it directly from the point where you highlight the width and height but not the length. Python has PCA functionality that is fast and easy and can be done in a few lines of code.

from sklearn.decomposition import PCA

pca = PCA(n_components=2)
pca.fit(embeddings)
pX = pca.transform(embeddings)

In the chart of the transformed PCA components we can see 3 or 4 main groups but not a lot of structure. This is not too surprising given what we now know about high dimensional spaces. It turns out we're not trying to layout a loaf of bread but rather a high dimensional basketball. If we inspect the PCA model we see that the first two components explain only about 10.8% of the variance.

Still, lets look at the most similar images for our 3 examples. And since we've now reduced the dimensions we can use the Euclidean distance.

from sklearn.metrics.pairwise import euclidean_distances

v = np.array([x, y])
v = v.reshape(1,-1)
d = euclidean_distances(v, emb).ravel()

A few of the plots seem to deal with death and oil but they don't really seem as relevant now. We may have lost too much information in this transformation. To actually judge that we'd need a more concrete metric and an actual business goal.

The Lebowski movies seem even less related.

Same for the Monty Python movies.

Overall this does not seem like a successful strategy as we have lost too much useful information.

T-Distributed Stochastic Neighbor Embedding (T-SNE)

So, going back to flattening a basketball, or perhaps a globe, you can try to choose a rotation that highlights the features (continents) you are most interested in ... or you can abandon the idea of a linear transformation and try to peel the surface off and lay it out on a flat surface. We all know that causes some distortions (ie. Greenland looks way bigger than it actually is on many maps) but is still a useful technique.

And what if you were free to cut and stretch that surface so that you could keep local information by sacrificing some global information. That is something like what T-SNE and UMAP try to do. And luckily they are both is easy to use from Python.

from sklearn.manifold import TSNE

tsne = TSNE(n_components=2, n_jobs=-1)
tX = tsne.fit_transform(embeddings)

T-SNE attempts to model the items in such away that local distances are close in 2 or 3 dimensions at the expense of global (or far away distances). And we can see in the plot that clusters (groups of highly similar items) start to appear.

The similar movies for There Will Be Blood do seem to be be related to family and land again and we see 'The Kidnappers' and 'Kidnapped' appear on the list again.

For Lebowski we see 'That's My Boy' appear again along with a movie about a pot smoking slacker.

The Monty Python movies make a lot more sense and we see movies about King Arthur, Camelot and Knights.

And just cause I'm curious, I took a look at a small cluster near (45.6, 37.4). These turned out to all be short movies staring The Stooges. This is the strongest indication that T-SNE is doing something interesting we've investigated so far.

Uniform Manifold Approximation and Projection for Dimension Reduction (UMAP)

UMAP is a newer technique and like T-SNE, tries to preserve local structure but also most of the global structure in the data. They both depend a lot on the data and the particular parameters you choose so if you are interested on using it in your application you'll have to explore to find settings that work well for you. These algorithms may not be as straight forward as PCA but they seem to yield more interesting results.

We can run the UMAP algorithm by using the umap-learn package which uses the scikit-learn API of fit and transform. By plotting the transformed points you can see there is a lot more structure and clusters with similar movies.

import umap

um =  umap.UMAP()
mapper = um.fit(embeddings)
uX = um.transform(embeddings)

For There Will Be Blood we're back among the western, family and land themes.

Lebowski's nearest films seem to be about lost and lazy under achievers.

An Python is still among the knight and King Arthur themes. Perhaps this is an 'outlier' movie that is easier to classify.

And just cause I'm curious I looked into another small cluster and found a group of Spike and Tyke animations. Another pleasant surprise.

Conclusions

Wow, this has gotten a lot longer than I ever expected so lets end here for now. I hope you are forgiving of the super lax 'metric' I used and understanding of the reasons why and that it sparked some ideas for your own analysis.

There are still lots of different and interesting experiments and analysis we can do on this data set which I leave for future articles.

If there is a specific use case you'd like to explore please get in touch or reach out on twitter.

Thanks for your time.

Julio