Natural Language Processing in Python

Recently, Microsoft’s Twitter bot “Tay” was released into the wild, and quickly began making racist and hateful statements after learning from other Twitter users. The technology behind this? Natural language processing.

NLP is the use of machine learning algorithms for problems that involve text.

Do you ever wonder why you get much less spam in your inbox these days compared to 10 years ago? What kinds of algorithms are people using to do spam detection? How can they take words in an email and know how to compute whether or not it’s spam? In this book you are going to build your very own spam detector.

Did you know people have already used Twitter to determine the current sentiment about a particular company to decide whether or not they should buy or sell stocks? Having a machine that can decide how people feel about something is immensely useful and immediately applicable to revenue optimization. In this course you are going to build your own sentiment analyzer.

Are you an Internet marketer or are you interested in SEO? Have you ever wanted to know how you can automatically generate content? In this course we are going to take a first crack at building your own article spinner. You’ll learn to write programs that can take an article as input and spit out a similar article with different words as output. This can save you tons of time and thousands of dollars if you’re paying someone to write content for you.

Natural Language Processing, or as it is often abbreviated, NLP - is the use of programming and math to do language-based tasks.

If you have Windows or iOS then you have NLP right in front of you! Cortana and Siri are applications that take what you say and turn it into something meaningful that can be done programmatically.

The key point: NLP is highly practical. NLP is everywhere.

This book is split up into multiple sections based on the various practical tasks that you can do with NLP:

Before we do any real programming exercises we’ll look at common NLP tasks (some of these we will actually code ourselves, the others are mentioned so you at least know they exist). We will then look at common data pre-processing techniques used for text. As you’ll see, this preprocessing is what will actually take up a majority of your time when you’re doing NLP.

The first programming exercise we’ll do is look at how to build a spam detector. Your email inbox uses this, so it’s clearly very useful and it’s been the subject of study for a long time.

Next we’ll look at “sentiment analysis” and you’ll build your own “sentiment analyzer”. This is how a computer can judge how positive or negative some text is based on the words and phrases that are used. This is also immediately practical - some people have analyzed Twitter feeds to predict whether a stock would go up or down.

After that we’ll look at the NLTK library. This is a very popular library that solves a lot of fundamental problems in NLP - and you can use it in conjunction with other libraries in your data analysis pipeline.

Next we’ll look at “latent semantic analysis”. This is basically doing dimensionality reduction on text - and it helps us solve the problem of 2 words having the same meaning. It also helps us interpret our data and save on computation time.

Lastly, we’ll talk about one of the most popular applications of NLP - article spinning. This is very practical for internet marketers and entrepreneurs. As you know, your search rankings in Google and other search engines are affected negatively when you have duplicate content - so it would be great if you could alter an article you wrote just enough, so that you could put it in 2 different places on the web, without being penalized by Google.