Your (Free) SEO Toolkit: Crawling and Indexing
In part 1 of our Toolkit series, we explored a few of the best free options for SEO analytics. This time, we’re diving into free tools that improve your indexing and crawling.
In order for a search engine to return your site in the results of a user search, your site needs to be indexed – because it’s really the index that Google (or Yahoo, or Bing, or…) is searching, not the entire internet, when you ask it how to fix your dishwasher or to find you a modern glass coffee table.
But even before Google can index your site, it’s got to crawl it. Think of it this way – there are three building blocks to getting Google to return your site.
Crawling: Google has to find your content
Indexing: Google has to create an index of all the pages they’ve discovered
Ranking: Google has to rank your page within the index
Ultimately, that’s your goal – to RANK on Google. Here’s a good explainer on how search engines work if you want a deeper dive.
So let’s take a look at good free tools for those first two – Crawling & Indexing:
Redirect Path Chrome Extension
This is a great little plug-in that flags 301, 302, 404, and 500 status codes and meta and JavaScript redirects, giving you a better understanding of where your site is missing the mark.
An added bonus, all of these details are easily copied to your clipboard to share in greater detail with your team or add to a technical audit.
Link Redirect Trace
Another Chrome extension, this one helps you to ensure that all your link redirects are actually taking people (and crawlers!) exactly where you want them to go within your site. Think of it as a quick QA tool that helps keep the train on its tracks.
Screaming Frog SEO Spider and Log File Analyzer
Technically, this is two tools – the Spider crawls your site (it’s not creepy, we swear) to allow you to discover things like HTTP header errors, excess HTML, crawl mistakes, duplicated content, JavaScript errors, and more. The Log File Analyzer allows you to upload your log files and then works its magic to tell you that yes, the search engine bots have visits, which URLs they’ve crawled, and helps you to study search bot data in greater detail. Together, they’re hugely helpful in getting your site crawled efficiently and effectively.
Robots.txt Generator
This one is used to tell search engine bots how to crawl your site. As it sounds, it creates a correct robots.txt file and advanced users can even customize their files.
XML Sitemaps
This one allows you to create a site map of up to 500 pages for free, without requiring registration. You’ll download your site map as an XML file or receive it via email, saving time (and probably a good bit of aggravation, as well).
SERP Checker
This one ties into your keyword use – determining potential ranking difficulties with a free tool from Ahrefs. Without using location-specific IP addresses or proxies, you can, from any location, check the top 10 search ranking results using this tool.
A few honorable mentions in this category – which is chock-full of free tools:
SFAIK Screaming Frog Analyzer – a robust visualization of Screaming Frog crawl data.
SEOWL Google Title Rewrite Checker – allowing for deeper Title Tag structural analysis.
Hreflang Tag Generator – so Google knows what language to crawl your page in.
Keep-Alive Validation Tool – for checking URLS in bulk to see if servers support persistent connection (ensuring faster load times).
Where Goes? – to check where shortened and redirect links are going.
Check My Links – a Chrome extension that crawls your page and color codes statuses.
Play around with a few and let us know what’s the most useful for your site. Next up for us is part 3, which delves into keyword research.