Overview This step is optional. If your website uses a WAF, CDN protection, or rate limiting, you may see some 403 or 429 errors in your crawl results. For more accurate results, you can whitelist our crawler using one of the methods below.
Identification Methods Our crawler can be identified using two methods:
User-Agent String SEOCrawler/1.0 (https://seocrawler.app)
Our crawler sends this unique header with every request to your site:
X-SEOCrawler-Verify: c0cf648c-fb8b-41f0-b25e-1c99646701e3
Using the verification header is more secure than User-Agent alone, as the token is unique to your domain and cannot be easily spoofed.
Cloudflare
AWS WAF
Nginx
Apache
Cloudflare WAF Create a Cloudflare WAF rule to allow our crawler:
Open WAF Settings
Go to Security → WAF → Custom Rules
Create New Rule
Click Create rule and enter a name like “Allow SEO Crawler”
Add Expression
Use this expression to match our verification header: (http.request.headers["x-seocrawler-verify"] eq "c0cf648c-fb8b-41f0-b25e-1c99646701e3")
Set Action
Set the action to Skip and check all security features you want to bypass.
Deploy
Click Deploy to activate the rule.
Alternative: User-Agent Matching You can also use User-Agent matching if you prefer: (http.user_agent contains "SEOCrawler")
User-Agent matching is less secure as it can be spoofed. We recommend using the verification header when possible.
AWS WAF Create an AWS WAF rule to allow our crawler:
Open Web ACL
Go to AWS WAF → Web ACLs → Select your ACL → Rules
Add Rule
Click Add rules → Add my own rules and rule groups
Configure Rule
Set up the rule with these settings: Setting Value Rule type Regular rule Inspect Single header Header field name X-SEOCrawler-VerifyMatch type Exactly matches string String to match c0cf648c-fb8b-41f0-b25e-1c99646701e3Action Allow
Set Priority
Place this rule before your blocking rules to ensure it takes precedence.
Save
Click Add rule and then Save on the Web ACL.
Make sure the rule priority is set so this allow rule is evaluated before any deny rules.
Nginx Rate Limiting Add this to your Nginx configuration to exempt our crawler from rate limiting: nginx.conf Add this in your http or server block: # Create a map to detect SEO Crawler map $ http_x_seocrawler_verify $ is_seocrawler { "c0cf648c-fb8b-41f0-b25e-1c99646701e3" 1 ; default 0 ; } # Define rate limiting zone limit_req_zone $ binary_remote_addr zone=general:10m rate=10r/s; server { location / { # Skip rate limit for SEO Crawler if ($ is_seocrawler ) { set $ limit_rate 0 ; } limit_req zone=general burst=20 nodelay; # ... your other config } }
Test Configuration After making changes, test and reload: # Test configuration nginx -t # Reload if test passes systemctl reload nginx
Apache / .htaccess Add this to your Apache configuration or .htaccess file: .htaccess or Apache config # Allow SEO Crawler < IfModule mod_rewrite.c > RewriteEngine On # Skip blocking rules for SEO Crawler RewriteCond %{HTTP:X-SEOCrawler-Verify} "^c0cf648c-fb8b-41f0-b25e-1c99646701e3$" RewriteRule .* - [E=SEOCRAWLER: 1 ] </ IfModule > # If using mod_security, add this rule < IfModule mod_security2.c > SecRule REQUEST_HEADERS:X-SEOCrawler-Verify "@streq c0cf648c-fb8b-41f0-b25e-1c99646701e3" \ "id: 1000001 ,phase: 1 ,allow,nolog,msg:' Allow SEO Crawler'" </ IfModule >
Restart Apache After making changes: # Test configuration apachectl configtest # Restart if test passes systemctl restart apache2 # or systemctl restart httpd
In the Sucuri dashboard:
Go to Firewall → Access Control
Under Whitelisted HTTP Headers , add:
Header: X-SEOCrawler-Verify
Value: c0cf648c-fb8b-41f0-b25e-1c99646701e3
Save changes
In WordPress admin:
Go to Wordfence → Firewall → Blocking
Create a new whitelist rule
Set pattern type to Custom Pattern
Allow requests containing header X-SEOCrawler-Verify with value c0cf648c-fb8b-41f0-b25e-1c99646701e3
In the Akamai Control Center:
Go to your property configuration
Add a new rule under Security
Match on header X-SEOCrawler-Verify equals c0cf648c-fb8b-41f0-b25e-1c99646701e3
Set action to Allow
Deploy the configuration
Add this VCL snippet: sub vcl_recv { if (req.http.X-SEOCrawler-Verify == "c0cf648c-fb8b-41f0-b25e-1c99646701e3") { # Skip rate limiting and security checks set req.http.Fastly-Allow = "true"; } }
In the Imperva dashboard:
Go to Security → WAF → Policies
Create a new Whitelist Rule
Set condition: Header X-SEOCrawler-Verify equals c0cf648c-fb8b-41f0-b25e-1c99646701e3
Apply the rule
Verifying the Whitelist After setting up your whitelist, run a new crawl to verify it’s working:
Start a New Crawl
Run a fresh crawl on your domain.
Check Results
Look for a reduction in 403 (Forbidden) and 429 (Too Many Requests) errors.
Compare
Compare the results to your previous crawl to see the improvement.
If you’re still seeing blocked requests after whitelisting, check that:
The rule is enabled and deployed
The rule has higher priority than blocking rules
There are no other security layers blocking requests
Troubleshooting
Verify the whitelist rule is active
Check for multiple security layers (WAF + CDN + origin)
Ensure the header name is exact: X-SEOCrawler-Verify
Check rule priority/order
Rate limiting may be configured at multiple levels
Check both CDN and origin server configurations
Verify the whitelist bypasses rate limiting, not just WAF
Can't find verification header
Security Considerations Keep your verification token secure. While we provide a unique token for your domain, you should treat it like a secret. Don’t share it publicly.
Our approach is secure because:
Unique tokens : Each domain has a unique verification tokenHeader-based : Headers are harder to spoof than User-Agent stringsServer-side verification : The check happens at your server/WAF level
Need Help? 
