Raven’s SEO Website Auditor Now Crawls Up to 10,000 Pages Per Site

Written by and published



Site Auditor’s Upgrade Makes You a Superhero SEO

Site Auditor has been a major focus at Raven, and for good reason. Site Auditor has crawled and recrawled hundreds of millions of pages and has discovered billions of SEO issues. These kinds of numbers deserve attention.

So, we’re happy to announce that Raven has rebuilt Site Auditor’s crawlers from the ground up. It now has the ability to crawl up to 10,000 pages per site. That’s 10 times more pages in a single crawl! So, bring out your big sites, and start going after those larger-site clients.

Wait, is This Even Fair?

We all know that Raven’s Site Auditor is powerful because of all of the information it gives you. And combined with Raven’s other SEO tools (Backlink Explorer, Keyword Rankings, Site Finder, and lest we forget: Research Central), you’re gonna be the SEO superhero you’ve always dreamed of being. Who cares about fair, just feed all this data goodness into your Reports and forget about your competition.

So, What’s Really Different About Site Auditor?

new on-page SEO website auditor

Over the past several months, we’ve been quietly pushing out some changes in Site Auditor to get ready for this big reveal. Here’s the rundown of all the changes:

1. Ability to Crawl 10,000 Pages/Site: We used to cap site crawls at 1,000 pages, but with our new crawling engine you can now crawl up to 10,000 pages per site! Now you can crawl that big site with the same clarity and accuracy you’ve come to expect from the Raven Site Auditor.

2. No Daily Page Limit: You saw that right! You can crawl as many pages as you want in a day. We used to limit you to 10,000 pages per day to save room for all the other folks, but we’re opening the gates, so bring all your sites to the party. Remember, we only crawl one site per account at a time, so have patience if you’ve got several whoppers. In place of a daily limit, you can crawl up to 300,000 pages per billing cycle whenever you’d like.

3. Schedule Your Crawl Date: We’ve always let you choose Daily, Weekly or Monthly crawls, but now you can set your start date and the Auditor will schedule out from there, or hit the new “Run Crawl” button anytime. No more logging into Raven on Saturday morning to set up that scheduled crawl. Just visit Tool Options > Customize Settings for the date option.

4. Canonical Link Element Support: We added support for the rel=”canonical” tag, which enables SEOs and Webmasters to communicate to Google (and other search engines) which content you want treated as the original on your site. If Site Auditor finds duplicate content, but that content points to the same canonical tag, Site Auditor will no longer report it as duplicate content.

5. Custom Page Title Length Setting: We all have different standards for what an acceptable title, meta description or site content length is. However, until now, Raven’s dictated the character count for long and short Page Titles. Now you can set your own character count in Tool Options > Customize Settings to match your internal guides.

ICYMI (In Case You Missed It)

These are some other changes that have happened in Site Auditor over the past several months. Some are front-end facing, while some are behind the scenes, but we thought you’d like to know:

  • Increased crawl delay to 2 seconds for all crawls.
  • Replaced YSlow with Google PageSpeed reports for both mobile and desktop
  • Improvements for scalability—bring on all the sites, we can handle them.
  • And coming soon: AJAX website crawling

Great! Now What?

If you’re not yet a user, sign up for your free trial. If you’re already a Raven user, log in to your account and run a crawl in SEO > Site Auditor. Then, set up a new report in the Report Builder, and include your data. The Site Auditor reports make a powerful impression on both prospects and existing clients. They’re an easy way to show what you’ll do for a future client, and how you’ve been improving clients’ visibility and accessibility.

So pull on that superhero cape, start those crawls, and get reporting!

  • Awesome news! I’m currently on trial and the inconsistent crawler (it would crawl a different amount of pages each time) was a big negative.

    I just logged into my trial account and I still see the “0/1000 pages crawled today”?

  • Love it! I’ll be putting it through my tests here shortly with some of our larger page count sites.

    I also appreciate you adding the custom Title length option, and we’ve been using that for the past while. Great stuff there! Any chance that you’ll be doing a similar custom option for the Description field?

    I know, I know….. all these great things, and still I’m pressing for more 😉

    Great job guys!

  • That is great news! I always have issues with ‘www’ and/or non www websites with the crawler, I hope that issue is addressed with this update too.

  • Jenny Jordan

    My website’s only home page is crawled. Few days back 10000 pages were getting crawled by the tool but now it says- 1 page is crawled. Please help!

    • We’re so sorry you’re having trouble, Jenny! Can you please send us an email? Our team is happy to take a closer look! help(@)raventools.com Thank you!