How to Spy on Competitors with Python & Data Studio (Advanced SEO)

Whether you’re new to web optimization or a seasoned professional, investigating random drops in key phrase rankings is simply a part of the commerce.

There are many strategies to investigating why your focused key phrases could have dropped, however utilizing Python and Data Studio collectively is a recreation changer!

A fast disclaimer: I didn’t create this script myself. A fellow web optimization and developer, Evan from Architek, collaborated with me to resolve a specific want I had. Evan was the brains behind creating the script.

I at all times wished to have the option to view a big set of Google outcomes at scale, with out having to conduct these searches manually one-by-one.

Evan talked about that Python will be the very best resolution for fixing my web optimization investigation wants.

Why Python Is Relevant to web optimization

Python is an extremely highly effective programing language that may do absolutely anything. One of the extra widespread makes use of for Python is automating each day monotonous duties.

One of the good issues about Python is that there are a number of other ways to accomplish the identical job. However, this additionally provides a brand new stage of problem.

Most of the pattern Python scripts on the market could be a bit outdated, so that you’ll end up doing a number of trial and error.

Python has many purposes for analyzing web optimization information. The secret’s to have the precise concept. If you may have an concept to automate a job, likelihood is that there’s a script that may be created for it.

To keep up to date with the newest use circumstances for Python in web optimization, Hamlet Batista has revealed some superior articles right here at Search Engine Journal.

What This Specific Script Does & Doesn’t Do

Most key phrase rating instruments report a mean rating for a key phrase over a specified timeframe. This Python script runs a single crawl on the time you run it out of your IP tackle. This script shouldn’t be meant to monitor key phrase rankings.

The objective of this script was to resolve a difficulty I used to be having with investigating sudden drops of rankings throughout my shopper and their competitors.

Most key phrase rating instruments will let you know which pages your area is rating for, however not your competitor’s highest rating pages per key phrase.

So Why’s That Important?

In this situation, we aren’t tacking web page efficiency for the long-term. We’re merely attempting to get fast information.

This script permits us to shortly establish tendencies throughout the natural panorama and see which pages are performing greatest.

What You’ll Need to Get Started

If you’re new to Python, I like to recommend testing the official Python tutorial or Automate the Boring Stuff.

For this tutorial I’m utilizing PyCharm CE, however you need to use Sublime Text or no matter your most popular growth surroundings is.

This script is written in Python Three and could also be a bit superior for folk new to this programming language.

If you haven’t discovered an interpreter or arrange your first digital surroundings but, this information may also help you get began.

Once you’re arrange with a brand new digital surroundings, you’ll want the next libraries:

Now that you just’re all arrange, let’s dive into some analysis collectively.

1. Make a List of Keywords to Investigate

We’re going to be utilizing some pattern information to examine some key phrases we’re pretending to monitor.

Let’s faux that you just checked out your key phrase monitoring software program and seen that the next key phrases dropped greater than 5 positions:

  • web optimization Tips
  • Local web optimization Advice
  • Learn web optimization
  • Search Engine Optimization Articles
  • web optimization Blog
  • web optimization Basics

Disclaimer: *Searching for too many key phrases could lead to your IP getting briefly banned. Pinging Google for this many outcomes without delay could seem spammy and can drain their assets. Use with warning and moderation.

The very first thing we are going to do is place these key phrases in a easy textual content file. The key phrases must be separated with a line break, as proven within the screenshot beneath.

Example of keywords to search with Python

2. Run the Python Ranking Investigation Script

The TL;DR of this script is that it does three fundamental features:

  • Locates and opens your searches.txt file.
  • Uses these key phrases and searches the primary web page of Google for every outcome.
  • Creates a brand new CSV file and prints the outcomes (Keyword, URLs, and web page titles).

We’ve made this script straightforward to run, so you’ll be able to simply copy and paste your entire script and run it .

from urllib.parse import urlencode, urlparse, parse_qs

from lxml.html import fromstring
from requests import get
import csv

Next, you’ll have the option to enter the primary operate of this script in a single copy/paste motion. This a part of the script units the precise steps taken, however won’t execute the command till the third step.

def scrape_run():
    with open('searches.txt') as searches:
        for search in searches:
           userQuery = search
           uncooked = get("" + userQuery).textual content
           web page = fromstring(uncooked)
           hyperlinks = web page.cssselect('.r a')
           csvfile = 'information.csv'
           for row in hyperlinks:
               raw_url = row.get('href')
               title = row.text_content()
               if raw_url.startswith("/url?"):
                   url = parse_qs(urlparse(raw_url).question)['q']
                   csvRow = [userQuery, url[0], title]
                   with open(csvfile, 'a') as information:
                       author =

Now you’re prepared to run the command. The ultimate step is to copy/paste the command beneath and click on the return key.


That’s it!

3. Use Data Studio to Analyze the Results

By operating this command, it’s possible you’ll discover {that a} new CSV file has been created known as information.csv. These are your uncooked outcomes, which we’ll want for the ultimate step.

My company has created a useful Data Studio template for analyzing your outcomes. In order to use this free report, you’ll want to paste your outcomes into Google Sheets.

The web page within the hyperlink above has in-depth directions on how to arrange your Data Studio report.

How to Analyze Our Results

Now that you’ve your new Data Studio report applied, it’s time to make sense of all this information.

What we’re on the lookout for are patterns. Yes, you will discover patterns within the uncooked information, however this Data Studio template has a useful characteristic that permits us to shortly establish which pages are rating probably the most ceaselessly for our focused key phrases.

This is useful, as a result of it permits us to see which opponents are performing nicely, and which particular pages are performing nicely.

Data Studio template for analyzing python web scraping data

As you’ll be able to see within the Data Studio screenshot above, Moz and Ahrefs are the highest two opponents rating for our searched key phrases.

However, that doesn’t actually assist us determine precisely what they’re doing to rank for these key phrases.

That’s the place the second chart is useful. This shows every rating web page and what number of occasions they happen for all of our search queries. We’re shortly ready to establish the highest three performing pages for our key phrases.

Need to filter down to a web page or key phrase particular stage?

We’ve included filters on the prime of the Data Studio template to simplify this.

data studio filters

Once you’ve made a listing of the best-performing pages, you’ll be able to conduct additional on-page and off-page evaluation to determine why these pages are performing so nicely.

Getting Stuck?

If you’re getting caught, attain out to the inventor of this script for suggestions or customized programming options.

What’s Your Idea?

Hopefully, this has sparked some artistic concepts on how you need to use Python to assist automate your web optimization processes.

More Resources:

Image Credits

All screenshots taken by writer, May 2019

Tags: , , , , , ,