Create Presigned CSV/JSONL S3 PUT URL

This route is useful for uploading very large CSV or JSONL files. Once you have completed the upload, chunks will be automatically created from the file for each line in the CSV or JSONL file. The chunks will be indexed and searchable. Auth’ed user must be an admin or owner of the dataset’s organization to upload a file.

POST /api/file/csv_or_jsonl

Create Presigned CSV/JSONL S3 PUT URL

cURL

curl --request POST \
  --url https://api.trieve.ai/api/file/csv_or_jsonl \
  --header 'Authorization: <api-key>' \
  --header 'Content-Type: application/json' \
  --header 'TR-Dataset: <tr-dataset>' \
  --data '
{
  "description": "This is an example file",
  "file_name": "example.pdf",
  "link": "https://example.com",
  "metadata": {
    "key1": "value1",
    "key2": "value2"
  },
  "tag_set": [
    "tag1",
    "tag2"
  ],
  "time_stamp": "2021-01-01 00:00:00.000Z"
}
'

200

{
  "file_metadata": {
    "created_at": "2021-01-01 00:00:00.000",
    "dataset_id": "e3e3e3e3-e3e3-e3e3-e3e3-e3e3e3e3e3e3",
    "file_name": "file.txt",
    "id": "e3e3e3e3-e3e3-e3e3-e3e3-e3e3e3e3e3e3",
    "link": "https://trieve.ai",
    "metadata": {
      "key": "value"
    },
    "size": 1000,
    "tag_set": "tag1,tag2",
    "time_stamp": "2021-01-01 00:00:00.000",
    "updated_at": "2021-01-01 00:00:00.000"
  },
  "presigned_put_url": "<string>"
}

Authorizations

Authorization · string · header · required

Headers

TR-Dataset · string<uuid> · required

The dataset id or tracking_id to use for the request. We assume you intend to use an id if the value is a valid uuid.

Body

application/json

JSON request payload to upload a CSV or JSONL file

file_name · string · required

Name of the file being uploaded, including the extension. Will be used to determine CSV or JSONL for processing.

description · string | null

Description is an optional convience field so you do not have to remember what the file contains or is about. It will be included on the group resulting from the file which will hold its chunk.

fulltext_boost_factor · number<double> | null

Amount to multiplicatevly increase the frequency of the tokens in the boost phrase for each row's chunk by. Applies to fulltext (SPLADE) and keyword (BM25) search.

group_tracking_id · string | null

Group tracking id is an optional field which allows you to specify the tracking id of the group that is created from the file. Chunks created will be created with the tracking id of group_tracking_id|<index of chunk>

link · string | null

Link to the file. This can also be any string. This can be used to filter when searching for the file's resulting chunks. The link value will not affect embedding creation.

mappings · object[] | null

Specify all of the mappings between columns or fields in a CSV or JSONL file and keys in the ChunkReqPayload. Array fields like tag_set, image_urls, and group_tracking_ids can have multiple mappings. Boost phrase can also have multiple mappings which get concatenated. Other fields can only have one mapping and only the last mapping will be used.

Show child attributes

metadata · any | null

Metadata is a JSON object which can be used to filter chunks. This is useful for when you want to filter chunks by arbitrary metadata. Unlike with tag filtering, there is a performance hit for filtering on metadata. Will be passed down to the file's chunks.

semantic_boost_factor · number<double> | null

Arbitrary float (positive or negative) specifying the multiplicate factor to apply before summing the phrase vector with the chunk_html embedding vector. Applies to semantic (embedding model) search.

tag_set · string[] | null

Tag set is a comma separated list of tags which will be passed down to the chunks made from the file. Each tag will be joined with what's creatd per row of the CSV or JSONL file.

time_stamp · string | null

Time stamp should be an ISO 8601 combined date and time without timezone. Time_stamp is used for time window filtering and recency-biasing search results. Will be passed down to the file's chunks.

upsert_by_tracking_id · boolean | null

Upsert by tracking_id. If true, chunks will be upserted by tracking_id. If false, chunks with the same tracking_id as another already existing chunk will be ignored. Defaults to true.

Response

200

application/json

File object information and signed put URL

file_metadata · object · required

Show child attributes

Example:

{
  "created_at": "2021-01-01 00:00:00.000",
  "dataset_id": "e3e3e3e3-e3e3-e3e3-e3e3-e3e3e3e3e3e3",
  "file_name": "file.txt",
  "id": "e3e3e3e3-e3e3-e3e3-e3e3-e3e3e3e3e3e3",
  "link": "https://trieve.ai",
  "metadata": { "key": "value" },
  "size": 1000,
  "tag_set": "tag1,tag2",
  "time_stamp": "2021-01-01 00:00:00.000",
  "updated_at": "2021-01-01 00:00:00.000"
}

presigned_put_url · string · required

Signed URL to upload the file to.