Upload File

Upload a file to S3 bucket attached to your dataset. You can select between a naive chunking strategy where the text is extracted with Apache Tika and split into segments with a target number of segments per chunk OR you can use a vision LLM to convert the file to markdown and create chunks per page. You must specifically use a base64url encoding. Auth’ed user must be an admin or owner of the dataset’s organization to upload a file.

POST /api/file

Upload File

cURL

curl --request POST \
  --url https://api.trieve.ai/api/file \
  --header 'Authorization: <api-key>' \
  --header 'Content-Type: application/json' \
  --header 'TR-Dataset: <tr-dataset>' \
  --data '
{
  "base64_file": "<base64_encoded_file>",
  "create_chunks": true,
  "description": "This is an example file",
  "file_name": "example.pdf",
  "link": "https://example.com",
  "metadata": {
    "key1": "value1",
    "key2": "value2"
  },
  "split_delimiters": [
    ",",
    ".",
    "\n"
  ],
  "tag_set": [
    "tag1",
    "tag2"
  ],
  "target_splits_per_chunk": 20,
  "time_stamp": "2021-01-01 00:00:00.000Z",
  "use_pdf2md_ocr": false
}
'

200

{
  "file_metadata": {
    "created_at": "2021-01-01 00:00:00.000",
    "dataset_id": "e3e3e3e3-e3e3-e3e3-e3e3-e3e3e3e3e3e3",
    "file_name": "file.txt",
    "id": "e3e3e3e3-e3e3-e3e3-e3e3-e3e3e3e3e3e3",
    "link": "https://trieve.ai",
    "metadata": {
      "key": "value"
    },
    "size": 1000,
    "tag_set": "tag1,tag2",
    "time_stamp": "2021-01-01 00:00:00.000",
    "updated_at": "2021-01-01 00:00:00.000"
  }
}

Authorizations

Authorization · string · header · required

Headers

TR-Dataset · string<uuid> · required

The dataset id or tracking_id to use for the request. We assume you intend to use an id if the value is a valid uuid.

Body

application/json

JSON request payload to upload a file

base64_file · string · required

Base64 encoded file.

file_name · string · required

Name of the file being uploaded, including the extension.

chunkr_create_task_req_payload · object | null

Will use chunkr.ai to process the file when this object is defined. See docs.chunkr.ai/api-references/task/create-task for detailed information about what each field on this request payload does.

Show child attributes

create_chunks · boolean | null

Create chunks is a boolean which determines whether or not to create chunks from the file. If false, you can manually chunk the file and send the chunks to the create_chunk endpoint with the file_id to associate chunks with the file. Meant mostly for advanced users.

description · string | null

Description is an optional convience field so you do not have to remember what the file contains or is about. It will be included on the group resulting from the file which will hold its chunk.

group_tracking_id · string | null

Group tracking id is an optional field which allows you to specify the tracking id of the group that is created from the file. Chunks created will be created with the tracking id of group_tracking_id|<index of chunk>

link · string | null

Link to the file. This can also be any string. This can be used to filter when searching for the file's resulting chunks. The link value will not affect embedding creation.

metadata · any | null

Metadata is a JSON object which can be used to filter chunks. This is useful for when you want to filter chunks by arbitrary metadata. Unlike with tag filtering, there is a performance hit for filtering on metadata. Will be passed down to the file's chunks.

pdf2md_options · object | null

We plan to deprecate pdf2md in favor of chunkr.ai. This is a legacy option for using a vision LLM to convert a given file into markdown and then ingest it.

Show child attributes

rebalance_chunks · boolean | null

Rebalance chunks is an optional field which allows you to specify whether or not to rebalance the chunks created from the file. If not specified, the default true is used. If true, Trieve will evenly distribute remainder splits across chunks such that 66 splits with a target_splits_per_chunk of 20 will result in 3 chunks with 22 splits each.

split_avg · boolean | null

Split average will automatically split your file into multiple chunks and average all of the resulting vectors into a single output chunk. Default is false. Explicitly enabling this will cause each file to only produce a single chunk.

split_delimiters · string[] | null

Split delimiters is an optional field which allows you to specify the delimiters to use when splitting the file before chunking the text. If not specified, the default [.!?\n] are used to split into sentences. However, you may want to use spaces or other delimiters.

tag_set · string[] | null

Tag set is a comma separated list of tags which will be passed down to the chunks made from the file. Tags are used to filter chunks when searching. HNSW indices are created for each tag such that there is no performance loss when filtering on them.

target_splits_per_chunk · integer | null

Target splits per chunk. This is an optional field which allows you to specify the number of splits you want per chunk. If not specified, the default 20 is used. However, you may want to use a different number.

Required range: x >= 0

time_stamp · string | null

Time stamp should be an ISO 8601 combined date and time without timezone. Time_stamp is used for time window filtering and recency-biasing search results. Will be passed down to the file's chunks.

webhook_url · string | null

Optional webhook URL to receive notifications for each page processed.

Response

200

application/json

Confirmation that the file is uploading

file_metadata · object · required

Show child attributes

Example:

{
  "created_at": "2021-01-01 00:00:00.000",
  "dataset_id": "e3e3e3e3-e3e3-e3e3-e3e3-e3e3e3e3e3e3",
  "file_name": "file.txt",
  "id": "e3e3e3e3-e3e3-e3e3-e3e3-e3e3e3e3e3e3",
  "link": "https://trieve.ai",
  "metadata": { "key": "value" },
  "size": 1000,
  "tag_set": "tag1,tag2",
  "time_stamp": "2021-01-01 00:00:00.000",
  "updated_at": "2021-01-01 00:00:00.000"
}