Skip to main content

​
Overview

Automated crawls let you set it and forget it—configure a schedule for each domain, and we’ll automatically run link health checks at your chosen frequency. You’ll receive email reports with the results, and optionally a public shareable link for your team or clients. 
Automation Flow
═══════════════════════════════════════════════════════════

   ┌─────────────────┐
   │  Configure      │
   │  Schedule       │───→  Every 6 hours
   └────────┬────────┘
            │
            ▼
   ┌─────────────────┐      ┌─────────────────┐
   │  Crawl Runs     │─────→│  Email Report   │
   │  Automatically  │      │  Sent to You    │
   └────────┬────────┘      └─────────────────┘
            │
            ▼
   ┌─────────────────┐
   │  Repeat at      │
   │  Next Interval  │
   └─────────────────┘

​
Available Schedules by Plan

The frequency options available depend on your subscription plan:
ScheduleFreeSoloProAgency
Off✓✓✓✓
Weekly—✓✓✓
Daily——✓✓
Every 12 Hours———✓
Every 6 Hours———✓
Hourly———✓
Free plan users can run manual crawls but don’t have access to automation. Upgrade to Solo or higher to enable scheduled crawls.

​
Setting Up Automation

1

Go to Settings

Navigate to Settings → Automation tab in your dashboard.
2

Select a Domain

Find the domain you want to automate. Each verified domain has its own configuration card.
3

Choose a Schedule

Use the dropdown to select your preferred frequency (based on your plan).
4

Configure Scan Options

Select which checks to run during each automated crawl.
5

Set Report Preferences (Agency)

Agency users can customize the email recipient and enable public report links.

​
Scan Options

When configuring automation, you can enable any combination of these checks:
OptionDescriptionAvailable
SSL CertificateCheck certificate expiry and warn about upcoming expirationsAll plans
Anchor Text AnalysisIdentify generic or non-descriptive link text for SEOAll plans
Response Time TrendsCompare response times with previous crawlsAll plans
SEO AnalysisAnalyze meta tags, headings, content, images, and markupAll plans
Sitemap & Orphan PagesCompare sitemap.xml with crawled pages, detect orphan pagesPro+ only
Enable only the checks you need. More checks mean longer crawl times but more comprehensive reports.

​
Report Delivery

​
Email Reports

After each automated crawl completes, you’ll receive an email summary containing:
  • Total links checked
  • Number of broken links found
  • Health score percentage
  • Link to view full report

​
Custom Recipients (Agency)

Agency plan users can send reports to any email address—perfect for:
  • Clients — Send reports directly to your clients
  • Team members — Route reports to different team members per domain
  • Stakeholders — Keep project managers informed
Report Delivery Options
═══════════════════════════════════════════════════════════

Solo/Pro Plans:          Agency Plan:
─────────────────        ────────────────────────────────
Reports → Your email     Reports → Any email you specify
                                   │
                                   ├─ client@company.com
                                   ├─ dev-team@agency.com
                                   └─ seo@client.com
Enable Include public report link to:
  1. Automatically make each crawl report publicly accessible
  2. Include the shareable link in the email
  3. Allow anyone with the link to view the report (no login required)
Public links expose your crawl data to anyone with the URL. Only enable this if you need to share reports externally.

​
How Automation Works

​
Scheduling Engine

Our automation runs on a background scheduler that:
  1. Checks every minute for domains due for a crawl
  2. Respects your daily limits — automated crawls count toward your plan’s daily crawl limit
  3. Prevents overlaps — won’t start a new crawl if one is already running for that domain
  4. Calculates next run — automatically schedules the next crawl after completion
Scheduling Logic
═══════════════════════════════════════════════════════════

Every minute:
├─ Find domains where next_run_at ≤ now
├─ Filter to verified domains only
├─ Check user hasn't hit daily crawl limit
├─ Skip if crawl already running
└─ Start crawl & calculate next_run_at

​
Daily Crawl Limits

Automated crawls count toward your daily limit:
PlanDaily CrawlsExample with Hourly Schedule
Free2N/A (no automation)
Solo55 domains × weekly
Pro1010 domains × daily
Agency252 domains × every 6h + 1 × hourly
If you’ve used all your daily crawls (manual + automated), scheduled crawls will be skipped until the next day.

​
Best Practices

​
Choose the Right Frequency

Site TypeRecommended ScheduleReasoning
BlogWeeklyContent changes infrequently
E-commerceDailyProducts added/removed regularly
News siteEvery 6 hoursHigh content velocity
CorporateWeeklyStable content structure
SaaS appDailyDocs and links change with releases

​
Optimize Scan Options

Minimal Check (Fastest):
└─ Just link health check

Standard Check:
├─ SSL Certificate
├─ Response Trends
└─ SEO Analysis

Comprehensive Check (Slowest):
├─ SSL Certificate
├─ Anchor Text Analysis
├─ Response Trends
├─ SEO Analysis
└─ Sitemap & Orphan Pages

​
Monitor Your Limits

Keep an eye on your daily crawl usage in the dashboard. If you’re frequently hitting limits:
  1. Reduce frequency on less critical domains
  2. Upgrade your plan for higher limits
  3. Stagger schedules across domains

​
Troubleshooting

​
Crawl Didn’t Run

CauseSolution
Domain not verifiedComplete DNS verification first
Daily limit reachedWait until tomorrow or upgrade
Crawl already runningPrevious crawl still in progress
Schedule set to “Off”Enable a schedule frequency

​
Didn’t Receive Email

CauseSolution
Check spam folderMark as “not spam”
Wrong emailUpdate report recipient (Agency)
Notifications disabledCheck Settings → Notifications

​
Wrong Scan Options

Scan options are configured per domain. Make sure you’re looking at the correct domain card in the Automation settings.

​
API Access

Automation settings can also be managed via API: 
# Get automation settings
GET /api/automation

# Update domain schedule
PATCH /api/automation
{
  "domainId": "uuid",
  "schedule": "daily",
  "scanOptions": {
    "sslCheck": true,
    "seoAnalysis": true
  }
}
Full API documentation coming soon. Contact support for early access.

Orphan Pages

Find pages with no internal links pointing to them.

Sharing Results

Learn about sharing crawl reports with others.