Scraping Football Data Using the penaltyblog Python Package

Introduction

If you're interested in football analytics or trying to predict football results and odds for betting markets then you are going to need data. This article will walk through how to use the penaltyblog python package to scrape football (soccer) data from different websites and how to join data from different sources together.

Installing penaltyblog

The first thing you need to do is install the penaltyblog package from pypi using pip. Provided you've got Python 3 and pip already installed then you just need to run the command below in a terminal. If you've not got Python installed yet, then head over to Anaconda and install that first.

pip install penaltyblog

The Scrapers

penaltyblog contains scrapers for a number of different websites, included Understat, ESPN and football-data.co.uk. For consistency, each of the scrapers takes the same arguments when it is created. This is a standardized competition name and season. For example:

import penaltyblog as pb

understat = pb.scrapers.Understat("ENG Premier League", "2021-2022")
fb = pb.scrapers.FootballData("ENG Premier League", "2021-2022")
espn = pb.scrapers.ESPN("ENG Premier League", "2021-2022")

It doesn't matter which data source your are scraping, competition names always comprise the country's three letter code followed by it's name, and the season is start_year-end_year. The scraper then maps this to whatever the data source is calling the competition so you don't need to remember whether you are scraping La Liga, LaLiga, La Liga Primera Division or whatever else a particular website calls it.

You can get a list of available competitions for each scraper by calling its list_competitions() function

import penaltyblog as pb

understat = pb.scrapers.Understat.list_competitions()

['DEU Bundesliga 1',
 'ENG Premier League',
 'ESP La Liga',
 'FRA Ligue 1',
 'ITA Serie A',
 'RUS Premier League']

Fixtures

Let's go ahead and scrape ourselves fixtures from Understat for the English Premier League

import penaltyblog as pb

understat = pb.scrapers.Understat("ENG Premier League", "2021-2022")
under_fixtures = understat.get_fixtures()
under_fixtures.head()

If you look at the table above (you may need to scroll the table horizontally if you're on a small screen) then you'll notice that the column names have a consistent style to them, e.g. all in lowercase, formatted as snake case etc to make them easier to work with.

Where possible, the column names are also consistent between data sources. For example, the column for the home team is always called team_home whatever the site you're scraping. This may seem trivial but it's a huge time saving not having to try and remember what each different data source calls the same things.

Columns are always named as team_home, goals_home, xg_home etc so that if you print out the column names then related columns, such as team_home and team_away, appear next to each other. No more searching through a giant list of columns to try and find the one you want.

The data also comes with an id column as the dataframe's index so every row has a unique key associated with it comprising the timestamp plus the team names.

Merging Data Sources

Combining scraped data from multiple data sources can be tricky but the penaltyblog scrapers can help with this too. As a somewhat contrived example, let's try and combine Understat's xG scores with football-data.co.uk's betting odds for Bet365.

The first thing we need to do is scrape the data from both sites.

import penaltyblog as pb

under = pb.scrapers.Understat("ENG Premier League", "2021-2022")
ufix = under.get_fixtures()
ufix = ufix[["team_home", "team_away", "xg_home", "xg_away"]]

fb = pb.scrapers.FootballData("ENG Premier League", "2021-2022")
fbfix = fb.get_fixtures()
fbfix = fbfix[ ["team_home", "team_away", "b365_h", "b365_d","b365_a"]]

In theory, we should be able to merge the two datasets by joining on the team names. Let's give it a go and see what happens...

merged = ufix.merge(fbfix, on=["team_home", "team_away"], how="inner")
print(merged.shape)

Oh dear, we only have 240 fixtures instead of the 380 we would expect for a full season of Premier League fixtures 😕.

Unfortunately, both data sources use different team names. For example, Understat uses Manchester City whereas football-data uses Man City so we can't join on them since they don't match.

To get around this problem, all of the penaltyblog scrapers come with the ability to remap team names. penaltyblog doesn't have mappings for all the world's football teams (yet..) but there's enough for this example and you can easily extend the mappings with your own team names.

The mappings themselves are just a standard python dictionary, with the key as the team name you want to end up with and the value as a list of possibles choices to remap. The example below maps both Man Utd and Man United to Manchester United

{
  "Manchester United": ["Man Utd", "Man United"],
}

Let's try our join again but with the example team name mappings included. Notice how we can now simply join using the id column since it's unique per fixture and will be identical across both datasets now we've mapped the team names.

import penaltyblog as pb

mappings = pb.scrapers.get_example_team_name_mappings()

under = pb.scrapers.Understat("ENG Premier League", "2021-2022", mappings)
ufix = under.get_fixtures()

fb = pb.scrapers.FootballData("ENG Premier League", "2021-2022", mappings)
fbfix = fb.get_fixtures()

merged = ufix.merge(fbfix, left_index=True, right_index=True)
print(merged.shape)

Success! With just 12 lines of code (including the blank lines) we've scraped both Understat and football-data.co.uk then merged the two data sets together using the unique id the scrapers automatically created for us.

What Else Can the Scrapers Do?

Depending on the data source, the scrapers have additional functions for collecting extra data. For example, the Understat scraper can get you shot data, including the XY coordinates, and the ESPN scraper can get you player and team level data too.

import penaltyblog as pb

under = pb.scrapers.Understat("ENG Premier League", "2021-2022")
shots = under.get_shots("16376")

print(shots.head())

Take a look at the penaltyblog documentation for more details and examples of each scraper.

What's Next?

There's plenty of other websites to add to the scrapers, with FBRef and whoscored next up on the TODO list but let me know if there's any others that should be included too.

After that, it's back to the modelling to add more techniques for predicting football results and betting markets.

Thanks for reading!

Comments

Categories

Chance

Cluster Analysis

Deep Learning

EI

Elo Ratings

Expected Goals

Fantasy Football

Feature Engineering

Massey Ratings

Misc

Opta Pro Forum

PageRank

PlayerRating

Poisson

Pythagorean

Recruitment

Scraping

Shots

TSR

UEFA Champions League

Visualisation

Tags

Get In Touch!

Submit your comments below, and feel free to format them using MarkDown if you want. Comments typically take upto 24 hours to appear on the site and be answered so please be patient.

Thanks!