How to Run and Review a Crawl in ScreamingFrog

Screaming Frog is a website crawler that can help you and your team understand where you’re rocking the technical SEO game, and where your site could use some TLC. This is a great tool that you can use for free or upgrade to the paid version for more features.Once you’ve downloaded the program onto your computer, you’re ready to run your website crawl and make some changes to improve your site’s technical SEO performance. Let’s hop in.

How to Run a Screaming Frog Website Crawl

1. First, make sure you have the latest version of the Screaming Frog tool downloaded to your computer. You can download the tool by visiting screamingfrog.co.uk and choosing SEO Spider > Download. Once you’ve clicked download, you’ll choose the operating system you’re using and start the download process.
2. Next, open the application on your computer and navigate to the header section. 
3. Then, copy your desired website URL and, in the box at the top that says “Enter URL to spider,” paste the website you wish to crawl.
4. Finally, click the “Start” button to the right of the box, and wait for Screaming Frog to finish your website crawl!

If you’ve done the above steps correctly, you should see information populate the crawler application. Note that if you have more than 500 URLs associated with your website URL, the free version of Screaming Frog will let you know that you need to purchase a license to remove the 500 URL crawling limit that the free version imposes. If you’re not interested in purchasing a license, don’t worry—you can still do some great technical search engine optimizations on your site with the limited URLs.

How to Review a Crawl in Screaming Frog

Running a crawl is fairly simple in Screaming Frog, but what do you do once the crawl hits 100 percent? The Screaming Frog platform can be overwhelming at first, but once you know what you’re looking at and what you’re looking for, you’ll be able to take a deep dive into the data to see where you can improve on your website’s technical SEO. 

Here are some tips for reviewing your Screaming Frog crawl.

Navigating Screaming Frog

If you’ve never used Screaming Frog before, it may seem daunting to review your crawl. There is so much information displayed in one place and so many items to review. 

If you’re a beginner, I recommend starting with the overview section. This section will allow you to see at a glance where your site needs some SEO help. (More on navigating quick wins in the following section.)

If you’re looking to do an SEO audit and make recommendations or major changes to things like URLs, page titles, meta descriptions, or H1s, I recommend exporting your crawl into a .CSV file and adding your recommendations in a new column. This will allow you to easily see what each item currently is, along with what you’d like to change each item to for better SEO practices.

Lastly, if you want to get granular on a specific technical SEO topic, you can dive deep into your pages’ meta descriptions, for example, by clicking through the tabs near the top of the window. Clicking meta descriptions will give you more granular data on each page’s meta description, including how many times it’s used throughout your site, what the full meta description is, the length and the indexability of the page. This allows you to see which meta descriptions may need to be changed, if they are over the recommended 155 character limit.

Quick Technical SEO Wins from Your Screaming Frog Crawl

Improving your technical SEO doesn’t have to be hard. Looking through the overview tab, you can see where your site may need the most help. Here are some quick wins you can start implementing today to give your site a technical SEO boost. 

Simply run the Screaming Frog crawl, and then scroll through the overview tab. Pay close attention to the following, and make note of what you need to change:

• Ensure all URLs are HTTPS
• Redirect any “Client Error (4xx)” pages to working links
• Redirect any “Server Error (5xx)” pages to working links
• Update and redirect any URLs over 115 characters (with the caveat that if there are a lot, you may want to opt not to do this, but rather make sure going forward any new pages have URLs under 115 characters because too many redirects are bad for your site’s technical SEO health).
• Update Page Titles
• Add any missing
• Change any duplicates
• Alter any over 60 characters or under 30 characters
• Update meta descriptions
• Add any missing
• Change any duplicates
• Alter any that are over 155 characters or below 70 characters
• Update meta keywords
• Add any missing
• Change any duplicates
• Update H1s
• Add any missing
• Change any duplicates
• Alter any over 70 characters
• Update H2s
• Add any missing
• Change any duplicates
• Alter any over 70 characters
• Update Content
• Change any exact or near duplicates
• Add content where it makes sense to “Low Content Pages”
• Update Images
• Resize and compress any images over 100 KB
• Add missing alt text
• Rework any alt text that’s over 100 characters

Diagnosing and Solving SEO Issues

You and your team are now better equipped to diagnose and solve the technical SEO issues that your site is facing. Take actionable steps today to improve the SEO health of your site and help it climb the search engine ranks. We’ve put together a 15-point Essential Technical SEO Checklist to help you rank No. 1 on search engine results pages (SERPs).