Scan Options

Overview

Scan options control what analyses run during a crawl. Enable more options for comprehensive reports, or keep it simple for faster crawls.

Available Scan Options

SSL Certificate Check

Verifies SSL/TLS certificate validity and expiration dates.

What It Checks	Details
Certificate validity	Is the certificate valid and trusted?
Expiration date	When does the certificate expire?
Certificate chain	Is the chain complete and valid?
Hostname match	Does cert match the domain?

Available on: All plans Performance impact: Minimal (adds ~50ms per domain, not per page)

Always enable SSL Check—it’s fast and catches critical security issues.

SEO Analysis

Audits on-page SEO factors for every crawled page.

What It Checks	Details
Title tags	Present, length, uniqueness
Meta descriptions	Present, length, uniqueness
H1 headings	Single H1, descriptive content
Heading hierarchy	Proper nesting (H1→H2→H3)
Image alt text	All images have alt attributes
Viewport meta	Mobile-friendly configuration
Language attribute	`lang` on HTML element

Available on: Solo, Pro, Agency Performance impact: Moderate (adds ~200ms per page for parsing)

Anchor Text Analysis

Evaluates link text quality for all discovered links.

What It Checks	Details
Generic text	”Click here”, “Read more”, etc.
Empty anchors	Links with no text content
Image links	Images used as links without alt
Over-optimization	Same anchor text used repeatedly
Descriptiveness	Quality score for anchor relevance

Available on: Pro, Agency Performance impact: Low (adds ~50ms per page)

Response Trends

Tracks response times across multiple crawls for trend analysis.

What It Tracks	Details
TTFB over time	Time to First Byte trends
Total response time	Full download time trends
Performance changes	Regressions or improvements
Slow pages	Consistently slow resources

Available on: Pro, Agency Performance impact: None (uses existing timing data)

Response Trends requires multiple crawls to show meaningful data. Run at least 3-4 crawls before analyzing trends.

Configuring Options

Per-Crawl Configuration

Configure options each time you start a crawl:

Start Crawl

Click Start Crawl on your domain.

Select Options

Check the boxes for desired analyses:

┌─────────────────────────────────────┐
│  Scan Options                       │
│  ─────────────────────────────────  │
│  ☑ SSL Certificate Check            │
│  ☑ SEO Analysis                     │
│  ☐ Anchor Text Analysis             │
│  ☐ Response Trends                  │
└─────────────────────────────────────┘

Start

Click Start Crawl with your selected options.

Default Options

Set default options to apply to all crawls:

Open Domain Settings

Navigate to domain → Settings → Crawl Defaults.

Configure Defaults

Setting	Description
Default Options	Pre-selected scan options
Default Depth	Starting crawl depth
Max URLs	URL limit per crawl

Save

Click Save Defaults.

Advanced Settings

Rate Limiting

Control how fast the crawler makes requests:

Setting	Description	Default
Requests per second	Max requests per second	5
Concurrent connections	Parallel requests	3
Request delay	Minimum delay between requests	200ms

Be respectful of target servers. Aggressive crawling can trigger rate limits or blocks.

Timeouts

Configure how long to wait for responses:

Setting	Description	Default
Connect timeout	Time to establish connection	10s
Response timeout	Time to receive headers	30s
Total timeout	Maximum total request time	60s

User Agent

Customize the crawler’s user agent string:

Default:
SEOCrawler/1.0 (+https://seocrawler.app/bot)

Options:

Default: Identifies as SEO Crawler
Generic: Generic bot user agent
Browser-like: Mimics a standard browser
Custom: Enter your own user agent

Some sites block bot user agents. Use browser-like if you’re seeing unexpected blocks.

URL Filters

Control which URLs are included or excluded:

Filter Type	Example	Effect
Include patterns	`/blog/*`	Only crawl matching URLs
Exclude patterns	`/admin/*`	Skip matching URLs
Exclude extensions	`.pdf, .zip`	Skip certain file types

URL Filters
─────────────────────────────────────────
Include (leave empty for all):
  /blog/*
  /products/*

Exclude:
  /admin/*
  /login/*
  /api/*
  
Skip file types:
  .pdf, .zip, .doc, .xls

External Links

Configure how external links are handled:

Setting	Description
Check external links	Verify external URLs respond
Follow external redirects	Follow redirects on external links
Max external depth	How many hops for external redirects

Enable external link checking to catch broken outbound links, but don’t follow them deeply—that slows crawls significantly.

Option Combinations

Quick Audit

Fast overview of site health:

☑ SSL Certificate Check
☐ SEO Analysis
☐ Anchor Text Analysis
☐ Response Trends

Depth: 2
Max URLs: 100

Best for: Quick health checks, post-deployment verification

SEO Audit

Comprehensive SEO analysis:

☑ SSL Certificate Check
☑ SEO Analysis
☑ Anchor Text Analysis
☐ Response Trends

Depth: 5
Max URLs: 1,000

Best for: Monthly SEO audits, client reports

Full Analysis

Complete site analysis:

☑ SSL Certificate Check
☑ SEO Analysis
☑ Anchor Text Analysis
☑ Response Trends

Depth: 10
Max URLs: 10,000

Best for: Quarterly deep audits, site migrations

Performance Focus

Focus on speed and reliability:

☑ SSL Certificate Check
☐ SEO Analysis
☐ Anchor Text Analysis
☑ Response Trends

Depth: 5
Max URLs: 1,000

Best for: Performance monitoring, server health

Plan Feature Matrix

Option	Free	Solo	Pro	Agency
SSL Check	✅	✅	✅	✅
SEO Analysis	❌	✅	✅	✅
Anchor Text	❌	❌	✅	✅
Response Trends	❌	❌	✅	✅
Custom User Agent	❌	❌	✅	✅
URL Filters	❌	✅	✅	✅

Getting Started

Features

Domain Management

Crawling

Plans & Billing

Agency Features

Integrations

Overview

Available Scan Options

SSL Certificate Check

SSL Certificate Check

SEO Analysis

SEO Analysis

Anchor Text Analysis

Anchor Text Analysis

Response Trends

Response Trends

Configuring Options

Per-Crawl Configuration

Default Options

Advanced Settings

Rate Limiting

Timeouts

User Agent

URL Filters

External Links

Option Combinations

Quick Audit

SEO Audit

Full Analysis

Performance Focus

Plan Feature Matrix

Next Steps

Starting Crawls

Crawl Results

​Overview

​Available Scan Options

​SSL Certificate Check

SSL Certificate Check

​SEO Analysis

SEO Analysis

​Anchor Text Analysis

Anchor Text Analysis

​Response Trends

Response Trends

​Configuring Options

​Per-Crawl Configuration

​Default Options

​Advanced Settings

​Rate Limiting

​Timeouts

​User Agent

​URL Filters

​External Links

​Option Combinations

​Quick Audit

​SEO Audit

​Full Analysis

​Performance Focus

​Plan Feature Matrix

​Next Steps

Starting Crawls

Crawl Results

Overview

Available Scan Options

SSL Certificate Check

SEO Analysis

Anchor Text Analysis

Response Trends

Configuring Options

Per-Crawl Configuration

Default Options

Advanced Settings

Rate Limiting

Timeouts

User Agent

URL Filters

External Links

Option Combinations

Quick Audit

SEO Audit

Full Analysis

Performance Focus

Plan Feature Matrix

Next Steps