Get all crawl requests for a dataset

This endpoint is used to get all crawl requests for a dataset.

GET /api/crawl

Try it

Get all crawl requests for a dataset

cURL

curl --request GET \
  --url https://api.trieve.ai/api/crawl \
  --header 'Authorization: <api-key>' \
  --header 'TR-Dataset: <tr-dataset>'

200

[
  {
    "attempt_number": 123,
    "crawl_options": {
      "crawl_options": {
        "allow_external_links": false,
        "boost_titles": true,
        "exclude_tags": [
          "#ad",
          "#footer",
          "header",
          "head",
          "navbar",
          "footer",
          "aside",
          "nav",
          "form"
        ],
        "heading_remove_strings": [
          "Advertisement",
          "Sponsored"
        ],
        "ignore_sitemap": true,
        "include_tags": [],
        "interval": "daily",
        "limit": 50,
        "site_url": "nedzo.ai"
      }
    },
    "crawl_type": "firecrawl",
    "created_at": "2023-11-07T05:31:56Z",
    "dataset_id": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
    "id": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
    "scrape_id": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
    "status": "Pending",
    "url": "<string>",
    "interval": "<string>",
    "next_crawl_at": "2023-11-07T05:31:56Z"
  }
]

Authorizations

Authorization · string · header · required

Headers

TR-Dataset · string<uuid> · required

The dataset id to use for the request

Query Parameters

page · integer<int64> | null

The page number to retrieve

limit · integer<int64> | null

The number of items to retrieve per page

Response

200

application/json

Crawl requests retrieved successfully

attempt_number · integer<int32> · required

crawl_options · object · required

Options for setting up the crawl which will populate the dataset.

Show child attributes

crawl_options. add_chunks_to_dataset · boolean | null

Add chunks to the dataset that the crawl is created for, defaults to true

crawl_options. allow_external_links · boolean | null

Option for allowing the crawl to follow links to external websites.

crawl_options. body_remove_strings · string[] | null

Text strings to remove from body when creating chunks for each page

crawl_options. boost_titles · boolean | null

Boost titles such that keyword matches in titles are prioritized in search results. Strongly recommended to leave this on. Defaults to true.

crawl_options. exclude_paths · string[] | null

URL Patterns to exclude from the crawl

crawl_options. exclude_tags · string[] | null

Specify the HTML tags, classes and ids to exclude from the response.

crawl_options. heading_remove_strings · string[] | null

Text strings to remove from headings when creating chunks for each page

crawl_options. ignore_sitemap · boolean | null

Ignore the website sitemap when crawling, defaults to true.

crawl_options. include_paths · string[] | null

URL Patterns to include in the crawl

crawl_options. include_tags · string[] | null

Specify the HTML tags, classes and ids to include in the response.

crawl_options. interval · enum<string>

Interval at which specified site should be re-scraped

Available options: daily, weekly, monthly

crawl_options. limit · integer<int32> | null

How many pages to crawl, defaults to 1000

crawl_options. scrape_options · CrawlOpenAPIOptions · object

Options for including an openapi spec or shopify settigns

CrawlOpenAPIOptions

Show child attributes

crawl_options.scrape_options. openapi_schema_url · string · required

OpenAPI json schema to be processed alongside the site crawl

crawl_options.scrape_options. openapi_tag · string · required

Tag to look for to determine if a page should create an openapi route chunk instead of chunks from heading-split of the HTML

crawl_options.scrape_options. type · enum<string> · required

Available options: openapi

crawl_options. site_url · string | null

The URL to crawl

crawl_options. tags · string[] | null

Tags to add to the crawl

crawl_options. webhook_metadata · any | null

Metadata to send back with the webhook call for each successful page scrape

crawl_options. webhook_urls · string[] | null

Host to call back on the webhook for each successful page scrape

Example:

{
  "crawl_options": {
    "allow_external_links": false,
    "boost_titles": true,
    "exclude_tags": [
      "#ad",
      "#footer",
      "header",
      "head",
      "navbar",
      "footer",
      "aside",
      "nav",
      "form"
    ],
    "heading_remove_strings": ["Advertisement", "Sponsored"],
    "ignore_sitemap": true,
    "include_tags": [],
    "interval": "daily",
    "limit": 50,
    "site_url": "nedzo.ai"
  }
}

crawl_type · enum<string> · required

Available options: firecrawl, openapi, shopify, youtube

created_at · string<date-time> · required

dataset_id · string<uuid> · required

id · string<uuid> · required

scrape_id · string<uuid> · required

status · required

Available options: Pending

url · string · required

interval · string | null

next_crawl_at · string<date-time> | null