Skip to main content

​
Overview

Domain verification ensures you have permission to crawl a website. This protects site owners from unauthorized scanning and ensures you’re the rightful administrator of domains in your account.
Verification is a one-time process. Once verified, a domain remains verified unless you remove it from your account.

​
How DNS Verification Works

DNS verification requires adding a TXT record to your domain’s DNS settings:
FieldValue
TypeTXT
Host/Name_seocrawler-verify
Valueseocrawler-verification=<your-token>
When you click Verify, we query your DNS for this record. If found with the correct token, your domain is verified.

​
Step-by-Step Instructions

1

Get Your Verification Token

From your domain’s verification page, copy the full TXT record value:
seocrawler-verification=abc123xyz789
Each domain has a unique token. Don’t reuse tokens between domains.
2

Access DNS Settings

Log into your domain registrar or DNS provider’s dashboard.
3

Add TXT Record

Create a new TXT record with:
  • Name/Host: _seocrawler-verify
  • Value/Content: Your verification token
  • TTL: 300 or lowest available
4

Wait for Propagation

DNS changes can take up to 48 hours to propagate, though most complete within 15 minutes.
5

Click Verify

Return to SEO Crawler and click Verify Domain.

​
Provider-Specific Instructions

1

Log into Cloudflare

Go to dash.cloudflare.com and select your domain.
2

Open DNS Settings

Click DNS in the sidebar, then Records.
3

Add Record

Click Add record and configure:
FieldValue
TypeTXT
Name_seocrawler-verify
Contentseocrawler-verification=<token>
TTLAuto
Leave the proxy status as DNS only (gray cloud). TXT records aren’t proxied.
4

Save

Click Save to create the record.

​
Verifying Your DNS Record

Before clicking Verify in SEO Crawler, confirm your DNS record is active:

​
Using Command Line

dig TXT _seocrawler-verify.example.com +short
Expected output:
"seocrawler-verification=abc123xyz789"

​
Using Online Tools

Enter _seocrawler-verify.example.com (with your domain) and select TXT record type.

​
DNS Propagation

DNS changes don’t take effect instantly. When you create a TXT record:
  1. Your DNS provider updates their servers
  2. The change propagates to DNS resolvers worldwide
  3. Cached records at intermediate servers expire

​
Typical Propagation Times

ProviderTypical Time
Cloudflare1-5 minutes
GoDaddy15-30 minutes
Namecheap15-30 minutes
Route 535-15 minutes
Google Domains5-15 minutes
While full global propagation can take up to 48 hours, most verifications succeed within 30 minutes.

​
Speeding Up Propagation

  • Use the lowest TTL available when creating the record
  • Clear your local DNS cache before testing
  • Use 8.8.8.8 or 1.1.1.1 as DNS resolver for testing

​
Why Verification is Required

Domain verification serves important purposes:

Security

Prevents unauthorized parties from scanning websites they don’t own.

Resource Protection

Ensures crawl resources are used by legitimate owners.

Privacy

Results and reports are only accessible to verified owners.

Trust

Website owners trust that only they can analyze their sites.

​
Subdomain Verification

For subdomains, add the TXT record at the subdomain level:
DomainTXT Record Host
example.com_seocrawler-verify
blog.example.com_seocrawler-verify.blog
app.example.com_seocrawler-verify.app
If you’ve verified the root domain, you can optionally enable Auto-verify subdomains to skip verification for subdomains.

​
Keeping Verification Active

You can safely remove the TXT record after verification. We don’t re-check DNS after the initial verification.
However, keeping the record provides:
  • Faster re-verification if you remove and re-add the domain
  • Proof of ownership for auditing purposes

​
Next Steps

Verification Troubleshooting

Having issues? Check our troubleshooting guide.

Start Crawling

Your domain is verified—time to crawl!