Scroll Chunks

Get paginated chunks from your dataset with filters and custom sorting. If sort by is not specified, the results will sort by the id’s of the chunks in ascending order. Sort by and offset_chunk_id cannot be used together; if you want to scroll with a sort by then you need to use a must_not filter with the ids you have already seen. There is a limit of 1000 id’s in a must_not filter at a time.

POST /api/chunks/scroll

Try it

Scroll Chunks

cURL

curl --request POST \
  --url https://api.trieve.ai/api/chunks/scroll \
  --header 'Authorization: <api-key>' \
  --header 'Content-Type: application/json' \
  --header 'TR-Dataset: <tr-dataset>' \
  --data '{
  "filters": {
    "must": [
      {
        "field": "tag_set",
        "match_all": [
          "A",
          "B"
        ]
      },
      {
        "field": "num_value",
        "range": {
          "gte": 10,
          "lte": 25
        }
      }
    ]
  },
  "offset_chunk_id": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
  "page_size": 1,
  "sort_by": {
    "field": "<string>",
    "direction": "desc",
    "prefetch_amount": 1
  }
}'

200

{
  "chunks": [
    {
      "chunk_html": "<p>Hello, world!</p>",
      "created_at": "2021-01-01 00:00:00.000",
      "dataset_id": "e3e3e3e3-e3e3-e3e3-e3e3-e3e3e3e3e3e3",
      "id": "e3e3e3e3-e3e3-e3e3-e3e3-e3e3e3e3e3e3",
      "link": "https://trieve.ai",
      "metadata": {
        "key": "value"
      },
      "tag_set": "[tag1,tag2]",
      "time_stamp": "2021-01-01 00:00:00.000",
      "tracking_id": "e3e3e3e3-e3e3-e3e3-e3e3-e3e3e3e3e3e3",
      "updated_at": "2021-01-01 00:00:00.000",
      "weight": 0.5
    }
  ]
}

Authorizations

Authorization · string · header · required

Headers

TR-Dataset · string<uuid> · required

The dataset id or tracking_id to use for the request. We assume you intend to use an id if the value is a valid uuid.

Body

application/json

JSON request payload to scroll through chunks (chunks)

filters · object

ChunkFilter is a JSON object which can be used to filter chunks. This is useful for when you want to filter chunks by arbitrary metadata. Unlike with tag filtering, there is a performance hit for filtering on metadata.

Show child attributes

filters. must · object[] | null

All of these field conditions have to match for the chunk to be included in the result set.

Filters can be constructed using either fields on the chunk objects, ids or tracking ids of chunks, and finally ids or tracking ids of groups.

Option 1

Show child attributes

filters.must. field · string · required

Field is the name of the field to filter on. Commonly used fields are timestamp, link, tag_set, location, num_value, group_ids, and group_tracking_ids. The field value will be used to check for an exact substring match on the metadata values for each existing chunk. This is useful for when you want to filter chunks by arbitrary metadata. To access fields inside of the metadata that you provide with the card, prefix the field name with metadata..

filters.must. boolean · boolean | null

Boolean is a true false value for a field. This only works for boolean fields. You can specify this if you want values to be true or false.

filters.must. date_range · object

DateRange is a JSON object which can be used to filter chunks by a range of dates. This leverages the time_stamp field on chunks in your dataset. You can specify this if you want values in a certain range. You must provide ISO 8601 combined date and time without timezone.

Show child attributes

filters.must.date_range. gt · string | null

filters.must.date_range. gte · string | null

filters.must.date_range. lt · string | null

filters.must.date_range. lte · string | null

Example:

{
  "gt": "2021-01-01 00:00:00.000",
  "gte": "2021-01-01 00:00:00.000",
  "lt": "2021-01-01 00:00:00.000",
  "lte": "2021-01-01 00:00:00.000"
}

filters.must. geo_bounding_box · object

Show child attributes

filters.must.geo_bounding_box. bottom_right · object · required

Location that you want to use as the center of the search.

Show child attributes

filters.must.geo_bounding_box.bottom_right. lat · required

filters.must.geo_bounding_box.bottom_right. lon · required

filters.must.geo_bounding_box. top_left · object · required

Location that you want to use as the center of the search.

Show child attributes

filters.must.geo_bounding_box.top_left. lat · required

filters.must.geo_bounding_box.top_left. lon · required

filters.must. geo_polygon · object

Show child attributes

filters.must.geo_polygon. exterior · object[] · required

Show child attributes

filters.must.geo_polygon.exterior. lat · required

filters.must.geo_polygon.exterior. lon · required

filters.must.geo_polygon. interior · object[][] | null

Show child attributes

filters.must.geo_polygon.interior. lat · required

filters.must.geo_polygon.interior. lon · required

filters.must. geo_radius · object

Show child attributes

filters.must.geo_radius. center · object · required

Location that you want to use as the center of the search.

Show child attributes

filters.must.geo_radius.center. lat · required

filters.must.geo_radius.center. lon · required

filters.must.geo_radius. radius · number<double> · required

filters.must. match_all · (string | integer<int64> | number<double>)[] | null

Match all lets you pass in an array of values that will return results if all of the items match. The match value will be used to check for an exact substring match on the metadata values for each existing chunk. If both match_all and match_any are provided, the match_any condition will be used.

filters.must. match_any · (string | integer<int64> | number<double>)[] | null

Match any lets you pass in an array of values that will return results if any of the items match. The match value will be used to check for an exact substring match on the metadata values for each existing chunk. If both match_all and match_any are provided, the match_any condition will be used.

filters.must. range · object

Show child attributes

filters.must.range. gt

filters.must.range. gte

filters.must.range. lt

filters.must.range. lte

Example:

{ "gt": 0, "gte": 0, "lt": 1, "lte": 1 }

Example:

{
  "field": "metadata.key1",
  "match": ["value1", "value2"],
  "range": { "gt": 0, "gte": 0, "lt": 1, "lte": 1 }
}

filters. must_not · object[] | null

None of these field conditions can match for the chunk to be included in the result set.

Filters can be constructed using either fields on the chunk objects, ids or tracking ids of chunks, and finally ids or tracking ids of groups.

Option 1

Show child attributes

filters.must_not. field · string · required

Field is the name of the field to filter on. Commonly used fields are timestamp, link, tag_set, location, num_value, group_ids, and group_tracking_ids. The field value will be used to check for an exact substring match on the metadata values for each existing chunk. This is useful for when you want to filter chunks by arbitrary metadata. To access fields inside of the metadata that you provide with the card, prefix the field name with metadata..

filters.must_not. boolean · boolean | null

Boolean is a true false value for a field. This only works for boolean fields. You can specify this if you want values to be true or false.

filters.must_not. date_range · object

DateRange is a JSON object which can be used to filter chunks by a range of dates. This leverages the time_stamp field on chunks in your dataset. You can specify this if you want values in a certain range. You must provide ISO 8601 combined date and time without timezone.

Show child attributes

filters.must_not.date_range. gt · string | null

filters.must_not.date_range. gte · string | null

filters.must_not.date_range. lt · string | null

filters.must_not.date_range. lte · string | null

Example:

{
  "gt": "2021-01-01 00:00:00.000",
  "gte": "2021-01-01 00:00:00.000",
  "lt": "2021-01-01 00:00:00.000",
  "lte": "2021-01-01 00:00:00.000"
}

filters.must_not. geo_bounding_box · object

Show child attributes

filters.must_not.geo_bounding_box. bottom_right · object · required

Location that you want to use as the center of the search.

Show child attributes

filters.must_not.geo_bounding_box.bottom_right. lat · required

filters.must_not.geo_bounding_box.bottom_right. lon · required

filters.must_not.geo_bounding_box. top_left · object · required

Location that you want to use as the center of the search.

Show child attributes

filters.must_not.geo_bounding_box.top_left. lat · required

filters.must_not.geo_bounding_box.top_left. lon · required

filters.must_not. geo_polygon · object

Show child attributes

filters.must_not.geo_polygon. exterior · object[] · required

Show child attributes

filters.must_not.geo_polygon.exterior. lat · required

filters.must_not.geo_polygon.exterior. lon · required

filters.must_not.geo_polygon. interior · object[][] | null

Show child attributes

filters.must_not.geo_polygon.interior. lat · required

filters.must_not.geo_polygon.interior. lon · required

filters.must_not. geo_radius · object

Show child attributes

filters.must_not.geo_radius. center · object · required

Location that you want to use as the center of the search.

Show child attributes

filters.must_not.geo_radius.center. lat · required

filters.must_not.geo_radius.center. lon · required

filters.must_not.geo_radius. radius · number<double> · required

filters.must_not. match_all · (string | integer<int64> | number<double>)[] | null

Match all lets you pass in an array of values that will return results if all of the items match. The match value will be used to check for an exact substring match on the metadata values for each existing chunk. If both match_all and match_any are provided, the match_any condition will be used.

filters.must_not. match_any · (string | integer<int64> | number<double>)[] | null

Match any lets you pass in an array of values that will return results if any of the items match. The match value will be used to check for an exact substring match on the metadata values for each existing chunk. If both match_all and match_any are provided, the match_any condition will be used.

filters.must_not. range · object

Show child attributes

filters.must_not.range. gt

filters.must_not.range. gte

filters.must_not.range. lt

filters.must_not.range. lte

Example:

{ "gt": 0, "gte": 0, "lt": 1, "lte": 1 }

Example:

{
  "field": "metadata.key1",
  "match": ["value1", "value2"],
  "range": { "gt": 0, "gte": 0, "lt": 1, "lte": 1 }
}

filters. should · object[] | null

Only one of these field conditions has to match for the chunk to be included in the result set.

Filters can be constructed using either fields on the chunk objects, ids or tracking ids of chunks, and finally ids or tracking ids of groups.

Option 1

Show child attributes

filters.should. field · string · required

Field is the name of the field to filter on. Commonly used fields are timestamp, link, tag_set, location, num_value, group_ids, and group_tracking_ids. The field value will be used to check for an exact substring match on the metadata values for each existing chunk. This is useful for when you want to filter chunks by arbitrary metadata. To access fields inside of the metadata that you provide with the card, prefix the field name with metadata..

filters.should. boolean · boolean | null

Boolean is a true false value for a field. This only works for boolean fields. You can specify this if you want values to be true or false.

filters.should. date_range · object

DateRange is a JSON object which can be used to filter chunks by a range of dates. This leverages the time_stamp field on chunks in your dataset. You can specify this if you want values in a certain range. You must provide ISO 8601 combined date and time without timezone.

Show child attributes

filters.should.date_range. gt · string | null

filters.should.date_range. gte · string | null

filters.should.date_range. lt · string | null

filters.should.date_range. lte · string | null

Example:

{
  "gt": "2021-01-01 00:00:00.000",
  "gte": "2021-01-01 00:00:00.000",
  "lt": "2021-01-01 00:00:00.000",
  "lte": "2021-01-01 00:00:00.000"
}

filters.should. geo_bounding_box · object

Show child attributes

filters.should.geo_bounding_box. bottom_right · object · required

Location that you want to use as the center of the search.

Show child attributes

filters.should.geo_bounding_box.bottom_right. lat · required

filters.should.geo_bounding_box.bottom_right. lon · required

filters.should.geo_bounding_box. top_left · object · required

Location that you want to use as the center of the search.

Show child attributes

filters.should.geo_bounding_box.top_left. lat · required

filters.should.geo_bounding_box.top_left. lon · required

filters.should. geo_polygon · object

Show child attributes

filters.should.geo_polygon. exterior · object[] · required

Show child attributes

filters.should.geo_polygon.exterior. lat · required

filters.should.geo_polygon.exterior. lon · required

filters.should.geo_polygon. interior · object[][] | null

Show child attributes

filters.should.geo_polygon.interior. lat · required

filters.should.geo_polygon.interior. lon · required

filters.should. geo_radius · object

Show child attributes

filters.should.geo_radius. center · object · required

Location that you want to use as the center of the search.

Show child attributes

filters.should.geo_radius.center. lat · required

filters.should.geo_radius.center. lon · required

filters.should.geo_radius. radius · number<double> · required

filters.should. match_all · (string | integer<int64> | number<double>)[] | null

Match all lets you pass in an array of values that will return results if all of the items match. The match value will be used to check for an exact substring match on the metadata values for each existing chunk. If both match_all and match_any are provided, the match_any condition will be used.

filters.should. match_any · (string | integer<int64> | number<double>)[] | null

Match any lets you pass in an array of values that will return results if any of the items match. The match value will be used to check for an exact substring match on the metadata values for each existing chunk. If both match_all and match_any are provided, the match_any condition will be used.

filters.should. range · object

Show child attributes

filters.should.range. gt

filters.should.range. gte

filters.should.range. lt

filters.should.range. lte

Example:

{ "gt": 0, "gte": 0, "lt": 1, "lte": 1 }

Example:

{
  "field": "metadata.key1",
  "match": ["value1", "value2"],
  "range": { "gt": 0, "gte": 0, "lt": 1, "lte": 1 }
}

Example:

{
  "must": [
    {
      "field": "tag_set",
      "match_all": ["A", "B"]
    },
    {
      "field": "num_value",
      "range": { "gte": 10, "lte": 25 }
    }
  ]
}

offset_chunk_id · string<uuid> | null

Offset chunk id is the id of the chunk to start the page from. If not specified, this defaults to the first chunk in the dataset sorted by id ascending.

page_size · integer<int64> | null

Page size is the number of chunks to fetch. This can be used to fetch more than 10 chunks at a time.

Required range: x >= 0

sort_by · object

Show child attributes

sort_by. field · string · required

Field to sort by. This has to be a numeric field with a Qdrant Range index on it. i.e. num_value and timestamp

sort_by. direction · enum<string>

Available options: desc, asc

sort_by. prefetch_amount · integer<int64> | null

How many results to pull in before the sort

Required range: x >= 0

Response

200

application/json

Number of chunks equivalent to page_size starting from offset_chunk_id

chunks · V2 · object[] · required

Show child attributes

chunks. created_at · string<date-time> · required

Timestamp of the creation of the chunk

chunks. dataset_id · string<uuid> · required

ID of the dataset which the chunk belongs to

chunks. id · string<uuid> · required

Unique identifier of the chunk, auto-generated uuid created by Trieve

chunks. updated_at · string<date-time> · required

Timestamp of the last update of the chunk

chunks. weight · number<double> · required

Weight of the chunk, can be any float. Used as a multiplier on a chunk's relevance score for ranking purposes.

chunks. chunk_html · string | null

HTML content of the chunk, can also be an arbitrary string which is not HTML

chunks. image_urls · (string | null)[] | null

Image URLs of the chunk, can be any list of strings. Used for image search and RAG.

chunks. link · string | null

Link to the chunk, should be a URL

chunks. location · object

Location that you want to use as the center of the search.

Show child attributes

chunks.location. lat · required

chunks.location. lon · required

chunks. metadata · any | null

Metadata of the chunk, can be any JSON object

chunks. num_value · number<double> | null

Numeric value of the chunk, can be any float. Can represent the most relevant numeric value of the chunk, such as a price, quantity in stock, rating, etc.

chunks. tag_set · (string | null)[] | null

Tag set of the chunk, can be any list of strings. Used for tag-filtered searches.

chunks. time_stamp · string<date-time> | null

Timestamp of the chunk, can be any timestamp. Specified by the user.

chunks. tracking_id · string | null

Tracking ID of the chunk, can be any string, determined by the user. Tracking ID's are unique identifiers for chunks within a dataset. They are designed to match the unique identifier of the chunk in the user's system.