Repetitive tasks and data pulls can eat into any agency members valuable time. The time taken for these tasks varies and depends on the amount of data needed, the number of different sources that data is needed from, and any data wranglings tasks such as merging, cleaning and storing the data in an easy to use way. Automating these processes could save any agency or team thousands of pounds in man-hours.
In this blog I will be showing how I have attempted to automate a link gap analysis between clients and competitors by building a web app.
A competitor link gap analysis is designed to highlight links that your competitors have that you do not. The idea behind this is that these sites are linking to sites similar to yours and therefore are hopefully more likely to link to your site. We can also take this one step further and look into other things like how fast your competitors are building links and what you need to do to keep up, the types of sites your competitors are building links from, and how your back link profile differs.
Pulling the data for each competitor manually and then throwing it into a excel/google sheet tool could present problems. For a start, when working with back link data the sheer volume of data could become a problem, especially with bigger competitors. If using Google Sheets the 5,000,000 cell limit is easily hit and, even if you don’t quite reach this, the sheet will become extremely sluggish/break when using lots of formulas to actually do something useful with the data in either excel or google sheets. This alone rules out deciding to build a smaller internal tool in Google Sheets/Excel. However there are also other issues like the sheets becoming confusing to follow; storing multiple analysis’ for multiple clients can become cluttered and they can become riddled with mistakes created by accidental movement of cells, amongst other things.
All of these reasons led me to develop this tool in Django. Django is a Python web framework where a number of web development features are provided out of the box. My choice of Django over something like Flask is due to Django having its own ORM out of the box, as well as a few other features, whereas Flask is more flexible. To keep the blog shorter I won’t be going into any python code so if you’re looking to build your own Django app here is the best place to start.
The LGA tool itself is part of an on going suite of tools we’re developing over here at Blueclaw and I’ll only be showing a few bits and pieces of it today. There is a whole other page level analysis section I won’t be going over but keep your eyes out for future blog posts on this area.
The domain entry section is just for entering your client and competitor domains and looks like this:
When the analysis is run, a report will be produced and stored.
Reports are split into 5 sections:
Whilst developing an app on Django you have access to a local server in order to easily test your work. However for other users to be able to access and use the app you need to host your web app on something like AWS or Azure. With minimal development experience and having used things like AWS lambda and DynamoDB in the past, I chose AWS. Elastic beanstalk is a great choice for someone like me as it handles things like auto-scaling and load balancing automatically. All we have to do is set up an environment.
The technical details of how to deploy to Elastic Beanstalk will, again, be omitted, but one of the best guides on how to do so I found here. This guide takes you through the initial set up right through to deploying your final app.
The tool I outlined today will hopefully assist the SEO team here at Blueclaw and will continue to be developed and improved upon. The suite of tools this belongs to will also be built upon with other tools already being available from within it.
With most tools, Google Sheet or Excel workbook are probably the ideal solution due to how quick and easy they are, but once larger chunks of data are needed then web app development is a safe bet, even if they do take a little longer to build.
I hope this blog gives an insight into how we are automating things here at Blueclaw and how we are using data to drive insight for our clients. To discuss any of the above, or for more information on how we can help automate your digital marketing strategy, get in touch with our senior team for a non-obligatory chat!