March 28, 2017
Their credentials in the SEO tools market are beyond reproach, which made the (relatively) recent launch of their Log File Analyser – now up to version 2.0 – an exciting prospect. Log file analysis is an increasingly well-understood element of a technical SEO audit, helping to paint a complete picture of how GoogleBot and other user agents are discovering, crawling and caching content and pages on a site.
As sites become larger and more complex, the importance of this can’t be overstated. Log analysis is the best way to understand how effectively your crawl budget is being utilised and to track down niggling problems in terms of information architecture and site structure.
Most servers will capture access log files but extracting useful data from them can be a challenging task; whether it’s by using Excel or one of the existing log analysis services such as Splunk – it’s not that intuitive a process.
There’s definitely a gap in the market for a more instinctively parsable log analysis tool. So how close does ScreamingFrog Log File Analyser come to filling that role?
LFA allows you to straight-up import the raw .log file as downloaded directly from the server. No need to worrying about converting or formatting the data first, simply drag and drop or use the browse function.
Additionally, the software allows for importing of URL lists in csv or xls/xlsx (Excel files pre and post 2004) – these can then be used to perform match comparisons with the data from the log file – particularly useful if you have a subset of important URLs that you’re looking to analyse.
Helpfully, since the tool is intended primarily as an SEO tool (most other tools of this type pitch themselves as multi-use) LFA will also automatically identify and segregate those requests made by the GoogleBot for you, ensuring that you always have a dataset purely for what is most likely the important spider crawling your site.
The “Overview” tab is an excellent dashboard with all the major metrics you’re likely to be tracking while performing log file analysis. It also includes handy line graphs that break down the response codes, events and URLs accessed over the period of the log. These can be filtered by time period and also by the particular Bot whose behaviour you’re interested in.
AWay from the overview, LFA breaks down the data onto a series of tabs – each containing a number sortable, filterable, exportable columns and rows, echoing the familiar setup of the SEO Spider Tool. As with the Spider Tool you can also filter this data on the fly by searching any alphanumerical string or, if you prefer, use the Tree view to see a more hierarchical breakdown.
The “URL” view contains the following:
Immediately this gives you a large amount of data that is ripe for analysis. A few examples of how quickly sorting this data by 1 or more of the columns can help you find problem pages:
For more specialised analysis the tool’s other, more focused, tabs offer expanded data:
Each site you analyse is stored in a separate Project file, but chances are you’re going to want to output segments of your data and, thankfully, this is simple and flexible in the ScreamingFrog LFA tool.
Each tab has its own export function, which in the case of the URL and Response Code tabs can be filtered to pull out specifically what you’re looking for.
If you’ve imported URL data you can also select whether this data should be included or not, or to pull out a list of matching/missing rows.
Your data is then kicked out in a nicely formatted CSV or Excel file, ready for further analysis or presentation.
Overall I found ScreamingFrog’s Log File Analyser an excellent way to perform an SEO-focused log file analysis job. The fact that it’s built specifically with a technical SEO user in mind was, to me, a huge boon and I’d recommend it to anybody who is looking to undertake this kind of analysis.
A limited free demo (restricted to a single project of 1,000 rows) and full licensed versions are available at https://www.screamingfrog.co.uk/log-file-analyser/. A single license for the full version is £99 per year.