Simple web analytics with Python and Pandas

We are going to do some analytics with our web visits data. As a simple report we are going to obtain the unique and total visits respect the date and many other paramenters like browser, page wisited, language, operative system...

Requirements

% pip install python-dateutil pandas

Getting and filtering the data

Let's assume the structure of our data like:

The python code:

from datetime import datetime, timedelta
import pandas as pd


source = pd.read_csv('data.csv', index_col='uuid', parse_dates=['date'])

date_1 = datetime.utcnow()
date_0 = date_1 - timedelta(days=30)

data = source[(source['date'] > date_0) & (source['date'] < date_1)]

Aggregating the data

We'll obtain data structured as:

The Python code:

def get_visits(groupby):
    ip_visits = data.groupby(groupby)['ip']
    return pd.DataFrame(
        {'unique_visits': ip_visits.apply(lambda x: len(set(x))),
         'total_visits': ip_visits.apply(len)})

data['date_day'] = data['date'].apply(datetime.date)
visits_by_date = get_visits('date_day')
# Redefine with date index to avoid lack of dates.
visits_by_date = pd.DataFrame(visits_by_date,
                              index=pd.date_range(date_0, date_1)).fillna(0)
# Rest of filtered visits.
visits_by_os = get_visits('os')
visits_by_city = get_visits('city')