Overview
Automated crawls let you set it and forget it—configure a schedule for each domain, and we’ll automatically run link health checks at your chosen frequency. You’ll receive email reports with the results, and optionally a public shareable link for your team or clients.
Automation Flow
═══════════════════════════════════════════════════════════
┌─────────────────┐
│ Configure │
│ Schedule │───→ Every 6 hours
└────────┬────────┘
│
▼
┌─────────────────┐ ┌─────────────────┐
│ Crawl Runs │─────→│ Email Report │
│ Automatically │ │ Sent to You │
└────────┬────────┘ └─────────────────┘
│
▼
┌─────────────────┐
│ Repeat at │
│ Next Interval │
└─────────────────┘
Available Schedules by Plan
The frequency options available depend on your subscription plan:
| Schedule | Free | Solo | Pro | Agency |
|---|
| Off | ✓ | ✓ | ✓ | ✓ |
| Weekly | — | ✓ | ✓ | ✓ |
| Daily | — | — | ✓ | ✓ |
| Every 12 Hours | — | — | — | ✓ |
| Every 6 Hours | — | — | — | ✓ |
| Hourly | — | — | — | ✓ |
Free plan users can run manual crawls but don’t have access to automation. Upgrade to Solo or higher to enable scheduled crawls.
Setting Up Automation
Go to Settings
Navigate to Settings → Automation tab in your dashboard.
Select a Domain
Find the domain you want to automate. Each verified domain has its own configuration card.
Choose a Schedule
Use the dropdown to select your preferred frequency (based on your plan).
Configure Scan Options
Select which checks to run during each automated crawl.
Set Report Preferences (Agency)
Agency users can customize the email recipient and enable public report links.
Scan Options
When configuring automation, you can enable any combination of these checks:
| Option | Description | Available |
|---|
| SSL Certificate | Check certificate expiry and warn about upcoming expirations | All plans |
| Anchor Text Analysis | Identify generic or non-descriptive link text for SEO | All plans |
| Response Time Trends | Compare response times with previous crawls | All plans |
| SEO Analysis | Analyze meta tags, headings, content, images, and markup | All plans |
| Sitemap & Orphan Pages | Compare sitemap.xml with crawled pages, detect orphan pages | Pro+ only |
Enable only the checks you need. More checks mean longer crawl times but more comprehensive reports.
Report Delivery
Email Reports
After each automated crawl completes, you’ll receive an email summary containing:
- Total links checked
- Number of broken links found
- Health score percentage
- Link to view full report
Custom Recipients (Agency)
Agency plan users can send reports to any email address—perfect for:
- Clients — Send reports directly to your clients
- Team members — Route reports to different team members per domain
- Stakeholders — Keep project managers informed
Report Delivery Options
═══════════════════════════════════════════════════════════
Solo/Pro Plans: Agency Plan:
───────────────── ────────────────────────────────
Reports → Your email Reports → Any email you specify
│
├─ client@company.com
├─ dev-team@agency.com
└─ seo@client.com
Public Report Links (Agency)
Enable Include public report link to:
- Automatically make each crawl report publicly accessible
- Include the shareable link in the email
- Allow anyone with the link to view the report (no login required)
Public links expose your crawl data to anyone with the URL. Only enable this if you need to share reports externally.
How Automation Works
Scheduling Engine
Our automation runs on a background scheduler that:
- Checks every minute for domains due for a crawl
- Respects your daily limits — automated crawls count toward your plan’s daily crawl limit
- Prevents overlaps — won’t start a new crawl if one is already running for that domain
- Calculates next run — automatically schedules the next crawl after completion
Scheduling Logic
═══════════════════════════════════════════════════════════
Every minute:
├─ Find domains where next_run_at ≤ now
├─ Filter to verified domains only
├─ Check user hasn't hit daily crawl limit
├─ Skip if crawl already running
└─ Start crawl & calculate next_run_at
Daily Crawl Limits
Automated crawls count toward your daily limit:
| Plan | Daily Crawls | Example with Hourly Schedule |
|---|
| Free | 2 | N/A (no automation) |
| Solo | 5 | 5 domains × weekly |
| Pro | 10 | 10 domains × daily |
| Agency | 25 | 2 domains × every 6h + 1 × hourly |
If you’ve used all your daily crawls (manual + automated), scheduled crawls will be skipped until the next day.
Best Practices
Choose the Right Frequency
| Site Type | Recommended Schedule | Reasoning |
|---|
| Blog | Weekly | Content changes infrequently |
| E-commerce | Daily | Products added/removed regularly |
| News site | Every 6 hours | High content velocity |
| Corporate | Weekly | Stable content structure |
| SaaS app | Daily | Docs and links change with releases |
Optimize Scan Options
Minimal Check (Fastest):
└─ Just link health check
Standard Check:
├─ SSL Certificate
├─ Response Trends
└─ SEO Analysis
Comprehensive Check (Slowest):
├─ SSL Certificate
├─ Anchor Text Analysis
├─ Response Trends
├─ SEO Analysis
└─ Sitemap & Orphan Pages
Monitor Your Limits
Keep an eye on your daily crawl usage in the dashboard. If you’re frequently hitting limits:
- Reduce frequency on less critical domains
- Upgrade your plan for higher limits
- Stagger schedules across domains
Troubleshooting
Crawl Didn’t Run
| Cause | Solution |
|---|
| Domain not verified | Complete DNS verification first |
| Daily limit reached | Wait until tomorrow or upgrade |
| Crawl already running | Previous crawl still in progress |
| Schedule set to “Off” | Enable a schedule frequency |
Didn’t Receive Email
| Cause | Solution |
|---|
| Check spam folder | Mark as “not spam” |
| Wrong email | Update report recipient (Agency) |
| Notifications disabled | Check Settings → Notifications |
Wrong Scan Options
Scan options are configured per domain. Make sure you’re looking at the correct domain card in the Automation settings.
API Access
Automation settings can also be managed via API:
# Get automation settings
GET /api/automation
# Update domain schedule
PATCH /api/automation
{
"domainId": "uuid",
"schedule": "daily",
"scanOptions": {
"sslCheck": true,
"seoAnalysis": true
}
}
Full API documentation coming soon. Contact support for early access.
