Internal Tool Development at Blueclaw - Link Gap Analysis Tool

Repetitive tasks and data pulls can eat into any agency members valuable time. The time taken for these tasks varies and depends on the amount of data needed, the number of different sources that data is needed from, and any data wranglings tasks such as merging, cleaning and storing the data in an easy to use way. Automating these processes could save any agency or team thousands of pounds in man-hours.

In this blog I will be showing how I have attempted to automate a link gap analysis between clients and competitors by building a web app.

Note :

• The example domains I will be using are not associated with Blueclaw in any way, they are purely used as examples.
• The tool I will be presenting is still in development here at Blueclaw.

The Idea

A competitor link gap analysis is designed to highlight links that your competitors have that you do not. The idea behind this is that these sites are linking to sites similar to yours and therefore are hopefully more likely to link to your site. We can also take this one step further and look into other things like how fast your competitors are building links and what you need to do to keep up, the types of sites your competitors are building links from, and how your back link profile differs.

Pulling the data for each competitor manually and then throwing it into a excel/google sheet tool could present problems. For a start, when working with back link data the sheer volume of data could become a problem, especially with bigger competitors. If using Google Sheets the 5,000,000 cell limit is easily hit and, even if you don’t quite reach this, the sheet will become extremely sluggish/break when using lots of formulas to actually do something useful with the data in either excel or google sheets. This alone rules out deciding to build a smaller internal tool in Google Sheets/Excel. However there are also other issues like the sheets becoming confusing to follow; storing multiple analysis’ for multiple clients can become cluttered and they can become riddled with mistakes created by accidental movement of cells, amongst other things.

All of these reasons led me to develop this tool in Django. Django is a Python web framework where a number of web development features are provided out of the box. My choice of Django over something like Flask is due to Django having its own ORM out of the box, as well as a few other features, whereas Flask is more flexible. To keep the blog shorter I won’t be going into any python code so if you’re looking to build your own Django app here is the best place to start.

The Link Gap Analysis Tool

The LGA tool itself is part of an on going suite of tools we’re developing over here at Blueclaw and I’ll only be showing a few bits and pieces of it today. There is a whole other page level analysis section I won’t be going over but keep your eyes out for future blog posts on this area.

The domain entry section is just for entering your client and competitor domains and looks like this:

When the analysis is run, a report will be produced and stored.

Reports are split into 5 sections:

• Referring Domain Gap: This highlights the gap between your clients and competitors referring domains. It also calculates how fast the domains are building links and how fast your client needs to build links in order to overtake them. The below screenshot shows an example page with link building velocity and a trust flow breakdown further down the page.

• Link Distributions: This highlights proportion of referring domains and back links that point to the homepage and inner pages. This page also highlights other distributions for each domain such as proportion of no follows, image links, etc.

• Topical Trust Flow: This aims to highlight the proportion and type of referring domains our client and competitor garners.

• Competitor Referring Domains: Aims to highlight the referring domains the competitor domains have that the client domain doesn’t have. This is with the aim that you may be able to easily acquire links from these domains as they link to other sites that are similar to yours.

• Recommendations: This section provides recommended domains to pursue links from and splits them into trust flow buckets that can be clicked on to open the recommendations up.

User access

Whilst developing an app on Django you have access to a local server in order to easily test your work. However for other users to be able to access and use the app you need to host your web app on something like AWS or Azure. With minimal development experience and having used things like AWS lambda and DynamoDB in the past, I chose AWS. Elastic beanstalk is a great choice for someone like me as it handles things like auto-scaling and load balancing automatically. All we have to do is set up an environment.

The technical details of how to deploy to Elastic Beanstalk will, again, be omitted, but one of the best guides on how to do so I found here. This guide takes you through the initial set up right through to deploying your final app.

Conclusions

The tool I outlined today will hopefully assist the SEO team here at Blueclaw and will continue to be developed and improved upon. The suite of tools this belongs to will also be built upon with other tools already being available from within it.

With most tools, Google Sheet or Excel workbook are probably the ideal solution due to how quick and easy they are, but once larger chunks of data are needed then web app development is a safe bet, even if they do take a little longer to build.

I hope this blog gives an insight into how we are automating things here at Blueclaw and how we are using data to drive insight for our clients. To discuss any of the above, or for more information on how we can help automate your digital marketing strategy, get in touch with our senior team for a non-obligatory chat!