Automated Crawls

Overview

Automated crawls let you set it and forget it—configure a schedule for each domain, and we’ll automatically run link health checks at your chosen frequency. You’ll receive email reports with the results, and optionally a public shareable link for your team or clients.

Automation Flow
═══════════════════════════════════════════════════════════

   ┌─────────────────┐
   │  Configure      │
   │  Schedule       │───→  Every 6 hours
   └────────┬────────┘
            │
            ▼
   ┌─────────────────┐      ┌─────────────────┐
   │  Crawl Runs     │─────→│  Email Report   │
   │  Automatically  │      │  Sent to You    │
   └────────┬────────┘      └─────────────────┘
            │
            ▼
   ┌─────────────────┐
   │  Repeat at      │
   │  Next Interval  │
   └─────────────────┘

Available Schedules by Plan

The frequency options available depend on your subscription plan:

Schedule	Free	Solo	Pro	Agency
Off	✓	✓	✓	✓
Weekly	—	✓	✓	✓
Daily	—	—	✓	✓
Every 12 Hours	—	—	—	✓
Every 6 Hours	—	—	—	✓
Hourly	—	—	—	✓

Free plan users can run manual crawls but don’t have access to automation. Upgrade to Solo or higher to enable scheduled crawls.

Setting Up Automation

Go to Settings

Navigate to Settings → Automation tab in your dashboard.

Select a Domain

Find the domain you want to automate. Each verified domain has its own configuration card.

Choose a Schedule

Use the dropdown to select your preferred frequency (based on your plan).

Configure Scan Options

Select which checks to run during each automated crawl.

Set Report Preferences (Agency)

Agency users can customize the email recipient and enable public report links.

Scan Options

When configuring automation, you can enable any combination of these checks:

Option	Description	Available
SSL Certificate	Check certificate expiry and warn about upcoming expirations	All plans
Anchor Text Analysis	Identify generic or non-descriptive link text for SEO	All plans
Response Time Trends	Compare response times with previous crawls	All plans
SEO Analysis	Analyze meta tags, headings, content, images, and markup	All plans
Sitemap & Orphan Pages	Compare sitemap.xml with crawled pages, detect orphan pages	Pro+ only

Enable only the checks you need. More checks mean longer crawl times but more comprehensive reports.

Report Delivery

Email Reports

After each automated crawl completes, you’ll receive an email summary containing:

Total links checked
Number of broken links found
Health score percentage
Link to view full report

Custom Recipients (Agency)

Agency plan users can send reports to any email address—perfect for:

Clients — Send reports directly to your clients
Team members — Route reports to different team members per domain
Stakeholders — Keep project managers informed

Report Delivery Options
═══════════════════════════════════════════════════════════

Solo/Pro Plans:          Agency Plan:
─────────────────        ────────────────────────────────
Reports → Your email     Reports → Any email you specify
                                   │
                                   ├─ client@company.com
                                   ├─ dev-team@agency.com
                                   └─ seo@client.com

Public Report Links (Agency)

Enable Include public report link to:

Automatically make each crawl report publicly accessible
Include the shareable link in the email
Allow anyone with the link to view the report (no login required)

Public links expose your crawl data to anyone with the URL. Only enable this if you need to share reports externally.

How Automation Works

Scheduling Engine

Our automation runs on a background scheduler that:

Checks every minute for domains due for a crawl
Respects your daily limits — automated crawls count toward your plan’s daily crawl limit
Prevents overlaps — won’t start a new crawl if one is already running for that domain
Calculates next run — automatically schedules the next crawl after completion

Scheduling Logic
═══════════════════════════════════════════════════════════

Every minute:
├─ Find domains where next_run_at ≤ now
├─ Filter to verified domains only
├─ Check user hasn't hit daily crawl limit
├─ Skip if crawl already running
└─ Start crawl & calculate next_run_at

Daily Crawl Limits

Automated crawls count toward your daily limit:

Plan	Daily Crawls	Example with Hourly Schedule
Free	2	N/A (no automation)
Solo	5	5 domains × weekly
Pro	10	10 domains × daily
Agency	25	2 domains × every 6h + 1 × hourly

If you’ve used all your daily crawls (manual + automated), scheduled crawls will be skipped until the next day.

Best Practices

Choose the Right Frequency

Site Type	Recommended Schedule	Reasoning
Blog	Weekly	Content changes infrequently
E-commerce	Daily	Products added/removed regularly
News site	Every 6 hours	High content velocity
Corporate	Weekly	Stable content structure
SaaS app	Daily	Docs and links change with releases

Optimize Scan Options

Minimal Check (Fastest):
└─ Just link health check

Standard Check:
├─ SSL Certificate
├─ Response Trends
└─ SEO Analysis

Comprehensive Check (Slowest):
├─ SSL Certificate
├─ Anchor Text Analysis
├─ Response Trends
├─ SEO Analysis
└─ Sitemap & Orphan Pages

Monitor Your Limits

Keep an eye on your daily crawl usage in the dashboard. If you’re frequently hitting limits:

Reduce frequency on less critical domains
Upgrade your plan for higher limits
Stagger schedules across domains

Troubleshooting

Crawl Didn’t Run

Cause	Solution
Domain not verified	Complete DNS verification first
Daily limit reached	Wait until tomorrow or upgrade
Crawl already running	Previous crawl still in progress
Schedule set to “Off”	Enable a schedule frequency

Didn’t Receive Email

Cause	Solution
Check spam folder	Mark as “not spam”
Wrong email	Update report recipient (Agency)
Notifications disabled	Check Settings → Notifications

Wrong Scan Options

Scan options are configured per domain. Make sure you’re looking at the correct domain card in the Automation settings.

API Access

Automation settings can also be managed via API:

# Get automation settings
GET /api/automation

# Update domain schedule
PATCH /api/automation
{
  "domainId": "uuid",
  "schedule": "daily",
  "scanOptions": {
    "sslCheck": true,
    "seoAnalysis": true
  }
}

Full API documentation coming soon. Contact support for early access.

Orphan Pages

Find pages with no internal links pointing to them.

Sharing Results

Learn about sharing crawl reports with others.

Getting Started

Features

Domain Management

Crawling

Plans & Billing

Agency Features

Integrations

Overview

Available Schedules by Plan

Setting Up Automation

Scan Options

Report Delivery

Email Reports

Custom Recipients (Agency)

Public Report Links (Agency)

How Automation Works

Scheduling Engine

Daily Crawl Limits

Best Practices

Choose the Right Frequency

Optimize Scan Options

Monitor Your Limits

Troubleshooting

Crawl Didn’t Run

Didn’t Receive Email

Wrong Scan Options

API Access

Orphan Pages

Sharing Results

​Overview

​Available Schedules by Plan

​Setting Up Automation

​Scan Options

​Report Delivery

​Email Reports

​Custom Recipients (Agency)

​Public Report Links (Agency)

​How Automation Works

​Scheduling Engine

​Daily Crawl Limits

​Best Practices

​Choose the Right Frequency

​Optimize Scan Options

​Monitor Your Limits

​Troubleshooting

​Crawl Didn’t Run

​Didn’t Receive Email

​Wrong Scan Options

​API Access

​Related

Orphan Pages

Sharing Results

Overview

Available Schedules by Plan

Setting Up Automation

Scan Options

Report Delivery

Email Reports

Custom Recipients (Agency)

Public Report Links (Agency)

How Automation Works

Scheduling Engine

Daily Crawl Limits

Best Practices

Choose the Right Frequency

Optimize Scan Options

Monitor Your Limits

Troubleshooting

Crawl Didn’t Run

Didn’t Receive Email

Wrong Scan Options

API Access

Related