Xcrawl Site Url Discovery

---
name: xcrawl-site-url-discovery
description: "Use this skill for XCrawl map tasks, including site URL discovery, regex filtering, scope estimation, and crawl planning before full-site crawling."
category: "Web & Scraping"
author: community
version: "1.0.2"
icon: globe
---

# XCrawl Map

## Overview

This skill uses XCrawl Map API to discover URLs for a site.
Default behavior is raw passthrough: return upstream API response bodies as-is.

## Required Local Config

Before using this skill, the user must create a local config file and write `XCRAWL_API_KEY` into it.

Path: `~/.xcrawl/config.json`

```json
{
  "XCRAWL_API_KEY": "<your_api_key>"
}
```

Read API key from local config file only. Do not require global environment variables.

## Credits and Account Setup

Using XCrawl APIs consumes credits.
If the user does not have an account or available credits, guide them to register at `https://dash.xcrawl.com/`.
After registration, they can activate the free `1000` credits plan before running requests.

## Tool Permission Policy

Request runtime permissions for `curl` and `node` only.
Do not request Python, shell helper scripts, or other runtime permissions.

## API Surface

- Start map task: `POST /v1/map`
- Base URL: `https://run.xcrawl.com`
- Required header: `Authorization: Bearer <XCRAWL_API_KEY>`

## Usage Examples

### cURL

```bash
API_KEY="$(node -e "const fs=require('fs');const p=process.env.HOME+'/.xcrawl/config.json';const k=JSON.parse(fs.readFileSync(p,'utf8')).XCRAWL_API_KEY||'';process.stdout.write(k)")"

curl -sS -X POST "https://run.xcrawl.com/v1/map" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${API_KEY}" \
  -d '{"url":"https://example.com","filter":"/docs/.*","limit":2000,"include_subdomains":true,"ignore_query_parameters":false}'
```

### Node

```bash
node -e '
const fs=require("fs");
const apiKey=JSON.parse(fs.readFileSync(process.env.HOME+"/.xcrawl/config.json","utf8")).XCRAWL_API_KEY;
const body={url:"https://example.com",filter:"/docs/.*",limit:3000,include_subdomains:true,ignore_query_parameters:false};
fetch("https://run.xcrawl.com/v1/map",{
  method:"POST",
  headers:{"Content-Type":"application/json",Authorization:`Bearer ${apiKey}`},
  body:JSON.stringify(body)
}).then(async r=>{console.log(await r.text());});
'
```

## Request Parameters

### Request endpoint and headers

- Endpoint: `POST https://run.xcrawl.com/v1/map`
- Headers:
- `Content-Type: application/json`
- `Authorization: Bearer <api_key>`

### Request body: top-level fields

| Field | Type | Required | Default | Description |
|---|---|---:|---|---|
| `url` | string | Yes | - | Site entry URL |
| `filter` | string | No | - | Regex filter for URLs |
| `limit` | integer | No | `5000` | Max URLs (up to `100000`) |
| `include_subdomains` | boolean | No | `true` | Include subdomains |
| `ignore_query_parameters` | boolean | No | `true` | Ignore URLs with query parameters |

## Response Parameters

| Field | Type | Description |
|---|---|---|
| `map_id` | string | Task ID |
| `endpoint` | string | Always `map` |
| `version` | string | Version |
| `status` | string | `completed` |
| `url` | string | Entry URL |
| `data` | object | URL list data |
| `started_at` | string | Start time (ISO 8601) |
| `ended_at` | string | End time (ISO 8601) |
| `total_credits_used` | integer | Total credits used |

`data` fields:

- `links`: URL list
- `total_links`: URL count
- `credits_used`: credits used
- `credits_detail`: credit breakdown

## Workflow

1. Restate mapping objective.
- Discovery only, selective crawl planning, or structure analysis.

2. Build and execute `POST /v1/map`.
- Keep filters explicit and reproducible.

3. Return raw API response directly.
- Do not synthesize URL-family summaries unless requested.

## Output Contract

Return:

- Endpoint used (`POST /v1/map`)
- `request_payload` used for the request
- Raw response body from map call
- Error details when request fails

Do not generate summaries unless the user explicitly requests a summary.

## Guardrails

- Do not claim full site coverage if `limit` is reached.
- Do not mix inferred URLs with returned URLs.
- Do not hardcode provider-specific tool schemas in core logic.