Python: Count Words in a String or File

In this tutorial, you’ll learn how to use Python to count the number of words and word frequencies in both a string and a text file. Being able to count words and word frequencies is a useful skill. For example, knowing how to do this can be important in text classification machine learning algorithms.

By the end of this tutorial, you’ll have learned:

• How to count the number of words in a string
• How to count the number of words in a text file
• How to calculate word frequencies using Python

Reading a Text File in Python

The processes to count words and calculate word frequencies shown below are the same for whether you’re considering a string or an entire text file. Because of this, this section will briefly describe how to read a text file in Python.

If you want a more in-depth guide on how to read a text file in Python, check out this tutorial here. Here is a quick piece of code that you can use to load the contents of a text file into a Python string:

# Reading a Text File in Python
file_path = '/Users/datagy/Desktop/sample_text.txt'

with open(file_path) as file:
    text = file.read()

I encourage you to check out the tutorial to learn why and how this approach works. However, if you’re in a hurry, just know that the process opens the file, reads its contents, and then closes the file again.

Count Number of Words In Python Using split()

One of the simplest ways to count the number of words in a Python string is by using the split() function. The split function looks like this:

# Understanding the split() function
str.split(
   sep=None     # The delimiter to split on
   maxsplit=-1  # The number of times to split
)

By default, Python will consider runs of consecutive whitespace to be a single separator. This means that if our string had multiple spaces, they’d only be considered a single delimiter. Let’s see what this method returns:

# Splitting a string with .split()
text = 'Welcome to datagy! Here you will learn Python and data science.'
print(text.split())

# Returns: ['Welcome', 'to', 'datagy!', 'Here', 'you', 'will', 'learn', 'Python', 'and', 'data', 'science.']

We can see that the method now returns a list of items. Because we can use the len() function to count the number of items in a list, we’re able to generate a word count. Let’s see what this looks like:

# Counting words with .split()
text = 'Welcome to datagy! Here you will learn Python and data science.'
print(len(text.split()))

# Returns: 11

Count Number of Words In Python Using Regex

Another simple way to count the number of words in a Python string is to use the regular expressions library, re. The library comes with a function, findall(), which lets you search for different patterns of strings.

Because we can use regular expression to search for patterns, we must first define our pattern. In this case, we want patterns of alphanumeric characters that are separated by whitespace.

For this, we can use the pattern \w+, where \w represents any alphanumeric character and the + denotes one or more occurrences. Once the pattern encounters whitespace, such as a space, it will stop the pattern there.

Let’s see how we can use this method to generate a word count using the regular expressions library, re:

# Counting words with regular expressions
import re
text = 'Welcome to datagy! Here you will learn Python and data science.'
print(len(re.findall(r'\w+', text)))

# Returns: 11

Calculating Word Frequencies in Python

In order to calculate word frequencies, we can use either the defaultdict class or the Counter class. Word frequencies represent how often a given word appears in a piece of text.

Using defaultdict To Calculate Word Frequencies in Python

Let’s see how we can use defaultdict to calculate word frequencies in Python. The defaultdict extend on the regular Python dictionary by providing helpful functions to initialize missing keys.

Because of this, we can loop over a piece of text and count the occurrences of each word. Let’s see how we can use it to create word frequencies for a given string:

# Creating word frequencies with defaultdict
from collections import defaultdict
import re

text = 'welcome to datagy! datagy will teach data. data is fun. data data data!'

counts = defaultdict(int)
for word in re.findall('\w+', text):
    counts[word] += 1

print(counts)

# Returns:
# defaultdict(<class 'int'>, {'welcome': 1, 'to': 1, 'datagy': 2, 'will': 1, 'teach': 1, 'data': 5, 'is': 1, 'fun': 1})

Let’s break down what we did here:

1. We imported both the defaultdict function and the re library
2. We loaded some text and instantiated a defaultdict using the int factory function
3. We then looped over each word in the word list and added one for each time it occurred

Using Counter to Create Word Frequencies in Python

Another way to do this is to use the Counter class. The benefit of this approach is that we can even easily identify the most frequent word. Let’s see how we can use this approach:

# Creating word frequencies with Counter
from collections import Counter
import re

text = 'welcome to datagy! datagy will teach data. data is fun. data data data!'
counts =  Counter(re.findall('\w+', text))
print(counts)

# Returns:
# Counter({'data': 5, 'datagy': 2, 'welcome': 1, 'to': 1, 'will': 1, 'teach': 1, 'is': 1, 'fun': 1})

Let’s break down what we did here:

1. We imported our required libraries and classes
2. We passed the resulting list from the findall() function into the Counter class
3. We printed the result of this class

One of the perks of this is that we can easily find the most common word by using the .most_common() function. The function returns a sorted list of tuples, ordering the items from most common to least common. Because of this, we can simply access the 0th index to find the most common word:

# Finding the Most Common Word
from collections import Counter
import re

text = 'welcome to datagy! datagy will teach data. data is fun. data data data!'
counts =  Counter(re.findall('\w+', text))
print(counts.most_common()[0])

# Returns:
# ('data', 5)

Conclusion

In this tutorial, you learned how to generate word counts and word frequencies using Python. You learned a number of different ways to count words including using the .split() method and the re library. Then, you learned different ways to generate word frequencies using defaultdict and Counter. Using the Counter method, you were able to find the most frequent word in a string.

Additional Resources

To learn more about related topics, check out the tutorials below:

• Python str.split() – Official Documentation
• Python Defaultdict: Overview and Examples
• Python: Count Number of Occurrences in List (6 Ways)
• Python: Count Number of Occurrences in a String (4 Ways!)