Skip to main content

​
Overview

Crawls are automated scans that discover and analyze every link on your domain. Start crawls manually when you need immediate results, or schedule them to run automatically.

​
Starting a Manual Crawl

1

Select Domain

From your dashboard, click on the domain you want to crawl.
2

Click Start Crawl

Click the Start Crawl button in the domain overview.
3

Configure Options

Select your scan options:
  • SSL Check
  • SEO Analysis
  • Anchor Text Analysis
  • Response Trends
4

Start

Click Start Crawl to begin. You’ll see real-time progress.

​
Crawl Progress

While a crawl is running, the progress view shows:

​
Live Feed

The activity feed shows URLs being processed in real-time:

​
Scheduling Crawls

Automated crawls run on a schedule without manual intervention.

​
Setting Up a Schedule

1

Open Domain Settings

Navigate to your domain and click Settings → Schedule.
2

Enable Scheduling

Toggle Scheduled Crawls to ON.
3

Select Frequency

Choose how often to run crawls:
4

Configure Time

Select when crawls should run:
  • Time: Hour of day (in your timezone)
  • Day: Day of week (for weekly)
5

Save

Click Save Schedule to activate.

​
Schedule Options

Schedule crawls during low-traffic hours to minimize impact on your server.

​
Daily Crawl Limits

Each plan has limits on how many crawls you can run per day:
Scheduled crawls count toward your daily limit. Plan schedules accordingly.

​
Crawl Depth

Crawl depth determines how many “links away” from the start URL the crawler will go:

​
Depth by Plan

Higher depths find more pages but take longer. Start with lower depths to estimate total pages.

​
URL Limits

Each crawl has a maximum number of URLs it will check: When the limit is reached:
  • Crawl completes with partial results
  • Dashboard shows “URL limit reached”
  • Results still include all checked URLs

​
Canceling a Crawl

To stop a running crawl:
  1. Go to the crawl progress page
  2. Click Cancel Crawl
  3. Confirm cancellation
Cancellation:
  • Stops immediately
  • Keeps results for URLs already checked
  • Counts toward daily limit
  • Shows status as “Cancelled”

​
Crawl Best Practices

Start Small

Begin with limited depth to gauge site size before running deep crawls.

Off-Peak Hours

Schedule crawls during low-traffic periods to minimize server impact.

Regular Cadence

Weekly crawls catch issues quickly without excessive server load.

After Deployments

Run a crawl after major changes to catch new broken links.

​
Crawl Queue

When multiple crawls are requested, they queue in order:
Queue behavior:
  • One crawl runs at a time per account
  • Queued crawls start automatically when previous completes
  • You can reorder or cancel queued crawls

​
Handling Large Sites

For sites with 10,000+ pages:
Instead of crawling the entire site:
  1. Set start URL to a section (e.g., /blog)
  2. Crawl each section separately
  3. Combine results for full picture
Use your sitemap as the URL source:
  1. Set start URL to /sitemap.xml
  2. Crawler extracts all URLs from sitemap
  3. More efficient than discovery-based crawling
Only check pages that changed:
  1. Enable incremental mode in settings
  2. Crawls check new/changed pages
  3. Significantly faster for large sites

​
Next Steps

Scan Options

Configure what analyses to run during crawls.

Crawl Results

Learn to interpret and act on crawl data.