Skip to main content

​
Crawls

Programmatically start crawls, check their status, and retrieve results.

​
Start a Crawl

​
/crawls
Start a new crawl for a verified domain. Request Body
FieldTypeRequiredDescription
domain_idstringYesUUID of the domain to crawl
options.scan_imagesbooleanNoCheck image URLs (default: false)
options.scan_scriptsbooleanNoCheck script URLs (default: false)
options.scan_stylesheetsbooleanNoCheck stylesheet URLs (default: false)
options.follow_externalbooleanNoCheck external links (default: false)
curl -X POST https://seocrawler.app/api/v1/crawls \
  -H "Authorization: Bearer sc_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{"domain_id": "550e8400-e29b-41d4-a716-446655440000"}'

​
List Crawls

​
/crawls
Retrieve a paginated list of crawls. Query Parameters
FieldTypeDefaultDescription
pageinteger1Page number
limitinteger20Results per page (max: 100)
domain_idstring—Filter by domain
statusstring—Filter: pending, running, completed, failed
curl "https://seocrawler.app/api/v1/crawls?status=completed&limit=10" \
  -H "Authorization: Bearer sc_your_api_key"

​
Get Crawl Details

​
/crawls/{id}
Retrieve detailed information about a specific crawl. 
curl https://seocrawler.app/api/v1/crawls/7c9e6679-7425-40de-944b-e07fc1f90ae7 \
  -H "Authorization: Bearer sc_your_api_key"
​
/crawls/{id}/links
Retrieve all links discovered during a crawl. Query Parameters
FieldTypeDescription
pageintegerPage number
limitintegerResults per page (max: 100)
status_codeintegerFilter by HTTP status
is_brokenbooleanFilter broken links only
curl "https://seocrawler.app/api/v1/crawls/7c9e6679-7425-40de-944b-e07fc1f90ae7/links?is_broken=true" \
  -H "Authorization: Bearer sc_your_api_key"
​
/crawls/{id}/broken
Convenience endpoint returning only broken links from a crawl. 
curl https://seocrawler.app/api/v1/crawls/7c9e6679-7425-40de-944b-e07fc1f90ae7/broken \
  -H "Authorization: Bearer sc_your_api_key"