Skip to main content
Version: 3.6.1-rc.2

RAG endpoint API

The RAG (Retrieval-Augmented Generation) Endpoint is a flexible and modular system built to enhance LLM responses with relevant context from your own documents. This system provides a standardized API for document ingestion and context retrieval, with support for temporal-aware search to automatically find the most time-relevant content.

For generated answers, use agentic RAG: give an agent or model access to the RAG as a tool so it can retrieve relevant passages and generate an answer. See Use tools with the Responses API for setup and examples.

Base URL and Authentication​

All requests to the RAG endpoint must include:

  1. Base URL: Your endpoint-specific base URL obtained from the portal

    https://api.example.com/v1/api/{endpoint-id}
  2. Authorization Header: Your API key or signed-in user access token in the Authorization header

    Authorization: Bearer your-api-key-from-portal

Documents protected by source permissions​

An SMB source can enforce file permissions. The caller needs a valid directory SID and matching file permissions. A personal API key uses the SID its owner had when the key was created. Service-account keys have no SID.

  • Search and listing. /context, GET /files, and MCP tools return only readable documents. File totals use the same filtering, including for administrators.
  • Direct reads. A document or chunk the caller cannot read returns 404, just like a missing document.
  • Groups. Only groups with readable documents are listed. A group with no readable documents returns 404.
  • Lookup failures. A required directory lookup failure returns 503 with Source permission lookup is unavailable. Unrelated sources are skipped when explicit source or document filters restrict the request.

API Reference​

Files API​

The /files endpoint provides a RESTful interface for managing files in the system.

Upload a File​

POST {base_url}/files

Headers:

  • Authorization: Bearer your-api-key
  • Content-Type: multipart/form-data

Parameters:

  • file: File data (required)
  • user_id: ID of the user uploading the file (optional, defaults to "system")
  • document_id: Custom ID for the document (optional, auto-generated if not provided)
  • group_ids: List of group IDs to associate with the file (optional)
  • metadata: JSON string of additional metadata (optional)
  • content_timestamp: ISO 8601 timestamp for when the content was created/effective (optional, defaults to upload time)
  • use_ocr: Enable OCR for PDFs and images when an OCR model is configured (optional)

When the endpoint has Delete files left without a group turned on, group_ids is required. An upload without a group answers 400 with the message Group ID is required when auto_delete_files_without_group is enabled. The same rule applies to group_id in Upload Content. See Configure a RAG endpoint to check or change the setting.

Example metadata:

{
"source": "email",
"author": "John Doe",
"tags": ["important", "reference"],
"custom_field": "value"
}

Example with custom timestamp:

# Upload a document with its actual creation date
with open("q3_report_2024.pdf", 'rb') as file:
response = requests.post(
f"{api_base_url}/files",
headers={"Authorization": f"Bearer {api_key}"},
files={"file": file},
data={
"content_timestamp": "2024-09-30T23:59:59Z", # Q3 end date
"metadata": json.dumps({"quarter": "Q3", "year": 2024})
}
)

Response:

{
"id": "document_id",
"filename": "example.pdf",
"created_at": 1677409068,
"status": "processed"
}

Which files can be indexed​

The RAG reads the text out of the file and indexes that text.

  • Documents: .pdf, .docx, .pptx, .xlsx, .xls, .epub, and Outlook email messages (.msg)
  • Text: .txt, .text, .md, .markdown, .json, .jsonl, .csv, .html, .htm, and .ipynb
  • Zip archives (.zip): unpacked, and each file inside is read by the rules above

A text file is read in the character encoding it was saved in. It does not have to be UTF-8.

An image (.jpg, .jpeg, .png) is indexed when the upload sets use_ocr and an OCR model is configured.

Video files are not indexed, and neither are the audio formats .m4a, .aac, .wma, .opus, .aiff, .aif and .amr. The upload answers 415 with a message that names the format, such as File format is not supported: .mp4. The file is not read.

A file in any other format the RAG cannot read is still read first. That upload answers 400 with the message Failed to parse file <filename>.

Audio files​

An audio file in .wav, .mp3, .flac, .ogg or .oga format is transcribed, and the transcript is indexed as the text of the document.

The endpoint needs a transcription model for this. Choose one on the RAG's page in the portal, as described in Configure a RAG endpoint. Without one, the upload answers 412 with the message File <filename> requires a configured transcription model.

If the transcription model cannot decode the recording, the upload answers 415 with the message Audio format of file <filename> is not supported. This also happens when the file name ends in a supported extension but the contents are in another format or are damaged. Save the recording as .wav, .mp3, .flac, .ogg or .oga and upload it again. If the transcription fails for another reason, the upload answers 400 with the message Failed to transcribe file <filename>. If the recording holds no speech, the upload answers 400 with the message No text content found in file <filename>.

Each piece of the transcript also keeps the time in the recording where its speech starts and ends. Search results carry these times, so an answer can point back to the right moment of a call or a meeting. An agent can also use them to read the recording around a result, as described in Call tools on the MCP endpoint.

Some transcription models return the transcript without times. The recording is still transcribed, indexed and searchable, but its results carry no times.

List Files​

GET {base_url}/files

Headers:

  • Authorization: Bearer your-api-key

Query Parameters:

  • user_id: Filter files by user ID (optional)
  • group_id: Filter files by group ID (optional)
  • source_id: Return only the files of this data source. Use null to return only manually uploaded files (optional)
  • limit: How many files to return, from 1 to 500. Omit it to return all matching files (optional)
  • cursor: The next_cursor from the previous response (optional)
  • search: Return only the files whose name contains this text. Upper and lower case do not matter. Up to 256 characters (optional)
  • include_total: Set to true to also return the exact number of files matching user_id, group_id, source_id, and search (optional, default false)

Response:

{
"files": [
{
"id": "doc_123",
"user_id": "user_456",
"group_ids": ["group_789", "group_101"],
"metadata": {
"filename": "report.pdf",
"created_at": "2025-02-26T12:32:39.265082",
"source": "upload",
"author": "Jane Smith"
}
}
],
"next_cursor": "eyJmaWx0ZXJzIjp7fSwib3BlcmF0aW9uIjoiZmlsZXMubGlzdCJ9",
"total": 237
}

total appears only when include_total=true. It is the exact count for the active filter set, independent of limit and cursor, so you can show a full total without paging through results. Without the parameter the response is unchanged.

Files from a data source also include source_id, source_key, and source_version.

Paging: Set limit to enable paging. next_cursor is null on the last page. To get the next page, send it back as cursor and keep the same user_id, group_id, and source_id. An invalid cursor returns 400.

Delete a File​

DELETE {base_url}/files/{file_id}

Headers:

  • Authorization: Bearer your-api-key

Parameters:

  • file_id: ID of the file to delete (path parameter)

Response:

{
"success": true,
"message": "File doc_123 deleted successfully",
"files_deleted": ["doc_123"]
}

If the file came from a connected data source, deleting it is temporary. The next sync indexes it again. Remove it from the source store to keep it out of the RAG.

Content API​

Use POST /files to upload a file, or POST /files/content to send text directly. Use /files/{file_id} to manage stored content. The older /content routes still work but are deprecated:

Deprecated routeUse instead
POST /contentPOST /files/content
GET /content/{content_id}GET /files/{file_id}
PATCH /content/{content_id}PATCH /files/{file_id}
DELETE /content/{content_id}DELETE /files/{file_id}
GET /content/{content_id}/chunksGET /files/{file_id}/chunks

The new routes take the same request bodies as the old ones. DELETE /files/{file_id} also returns a message field, which the old route omits.

Upload Content​

POST {base_url}/files/content

Headers:

  • Authorization: Bearer your-api-key
  • Content-Type: application/json

Request Body:

{
"contents": [
{
"id": "doc_2024_q1",
"txt": "Q1 2024 Financial Report...",
"content_timestamp": "2024-03-31T23:59:59Z", // Optional: defaults to upload time
"metadata": {
"type": "financial_report",
"quarter": "Q1"
}
}
],
"user_id": "finance_team",
"group_id": "financial_reports"
}

Important: The content_timestamp field allows you to specify when the content was created or is effective from. This is crucial for temporal search to work correctly when documents have specific dates (e.g., dated reports, versioned documentation).

Get Content​

GET {base_url}/files/{file_id}

Headers:

  • Authorization: Bearer your-api-key

Parameters:

  • file_id: ID of the item to read (path parameter)

Response: The stored item, including id, user_id, group_ids, metadata, content_timestamp, created_at, and source_text when available.

If no item has this ID, the endpoint returns 404 with the detail Content not found.

Update Content​

PATCH {base_url}/files/{file_id}

Request Body:

{
"txt": "Updated content...", // Optional
"content_timestamp": "2024-04-01T00:00:00Z", // Optional
"metadata": {
"revised": true
}
}

Context API​

The /context endpoint provides a flexible way to retrieve relevant chunks of text from your document repository based on a query.

POST {base_url}/context

Headers:

  • Authorization: Bearer your-api-key
  • Content-Type: application/json

Request Body:

{
"query": "your search query",
"max_chunks": 4,
"filter_ids": ["doc1", "doc2"],
"group_id": "group1",
"user_id": "user123",
"metadata_filters": [
{
"field": "author",
"value": "Alice",
"operator": "eq"
},
{
"field": "rating",
"value": 4.5,
"operator": "gt"
}
],
"smart_temporal_search": true // Optional: override default temporal search setting
}

Temporal Search: When smart_temporal_search is enabled (either by the RAG's Smart temporal search by default setting or explicitly in the request), the system automatically detects temporal intent in queries like "latest policy", "what changed last month", or "original version" and retrieves the most time-relevant content.

Parameters:

ParameterTypeRequiredDefaultDescription
querystringYes-The search query to find relevant text chunks
max_chunksintegerNo4Maximum number of chunks to return
reranker_top_nintegerNo16Number of candidate chunks to retrieve for reranking
reranker_batch_sizeintegerNo16Maximum number of chunks scored in each reranker batch
filter_idsarrayNonullList of document IDs to restrict the search to
group_idstringNonullGroup ID to filter documents by (for permission-based filtering)
user_idstringNonullUser ID to filter documents by (only returns documents owned by this user)
metadata_filtersarrayNonullList of metadata filters to apply
smart_temporal_searchbooleanNoThe RAG's Smart temporal search by default settingEnable/disable intelligent temporal search

When the endpoint has Require group for retrieval turned on, a request without group_id returns no chunks. See Configure a RAG endpoint to check or change the setting.

The reranker parameters apply only when the endpoint has a reranker model selected. Without one, the endpoint skips reranking and ignores these two parameters.

Metadata Filters​

Each metadata filter object supports:

FieldTypeRequiredDescription
fieldstringYesMetadata field path (e.g., "author" or "details.category")
valuescalar or scalar arrayYesScalar to match; flat scalar arrays are supported only by contains
operatorstringNoOperator to use (default: "eq")

Supported operators:

  • eq: Exact equality (default)
  • contains: Case-insensitive substring match for strings, or element match for arrays
  • gt: Greater than
  • gte: Greater than or equal to
  • lt: Less than
  • lte: Less than or equal to

eq compares the stored JSON value exactly, including its type and string case. Send 4.5, not "4.5", to match a JSON number, and send true, not "true", to match a boolean. A lossless exact-number object returned by metadata discovery can also be copied into a numeric equality filter. Use contains rather than eq for a stored array. Only contains accepts a filter value that is an array; it matches when any requested value matches. Scalar string contains uses case-insensitive substring matching, while stored arrays use case-insensitive string membership and typed equality for non-string elements.

Range operators accept finite JSON numbers, canonical numeric strings, strict ISO dates such as 2026-08-17, local ISO times such as 09:30:15.125, and timezone-qualified ISO datetimes such as 2026-08-17T10:30:00Z. Dates represent midnight UTC when compared with datetimes. Time-only values do not accept a timezone because they have no date for a UTC conversion. If a stored field is an array, any compatible scalar element can satisfy a range filter. Incompatible stored types do not match the filter.

All metadata filters in one request use AND: every filter must match before a document is eligible for the existing semantic ranking and reranking.

Limits for one request:

LimitValue
Filters20
Field path200 characters and 5 dot-separated segments using letters, numbers, _, -, or :
Values in one contains array50
Encoded size of one filter value4 KiB
Encoded size of all metadata filters16 KiB

The Context API returns 422 Unprocessable Entity without searching documents when a metadata filter is invalid. Common causes include an unsupported operator, an invalid field path, an array value used with an operator other than contains, an unsupported range value, or a request that exceeds one of these limits. An MCP search_documents call with the same invalid input fails as a tool-call error instead of returning a Context API HTTP status.

Advanced Temporal Parameters (for testing/evaluation):

  • temporal_mode: Manual override for temporal behavior ("earliest", "latest", "specific", "range", "none")
  • temporal_weight: Balance between semantic and temporal relevance (0-1)
  • temporal_value: ISO timestamp or date range for specific/range modes
  • temporal_strictness: How strictly to enforce temporal boundaries (0-1)

Response:

{
"chunks": [
"This is a document about Python programming...",
"Machine learning frameworks in Python include..."
],
"scores": [0.92, 0.87],
"files": [
{
"id": "doc1",
"user_id": "user123",
"group_ids": ["group1", "group2"],
"created_at": 1728987000,
"metadata": {
"filename": "python_intro.pdf",
"author": "Alice",
"topic": "programming",
"tags": ["python", "beginner"],
"rating": 4.5
},
"content_timestamp": "2024-10-15T10:30:00Z",
"top_score": 0.92,
"n_chunks": 1
},
{
"id": "doc2",
"user_id": "user456",
"group_ids": ["group1"],
"created_at": 1728987500,
"metadata": {
"filename": "ml_frameworks.pdf",
"author": "Bob",
"topic": "machine learning",
"tags": ["python", "advanced"],
"rating": 4.8
},
"content_timestamp": "2024-10-15T10:30:00Z",
"top_score": 0.87,
"n_chunks": 1
}
]
}

Response fields:

FieldTypeDescription
chunksarray of stringsThe retrieved text chunks
scoresarray of numbersRelevance score per chunk, same order as chunks
filesarray of objectsSource document for each chunk, same order as chunks (empty when nothing matched)

files holds one entry per returned chunk, not per document: top_score is that chunk's score and n_chunks is always 1, so the same id repeats when several chunks come from one document.

Each files entry contains:

FieldTypeDescription
idstringDocument ID
user_idstringOwner of the document ("system" when uploaded without a user_id)
group_idsarray of stringsGroups the document belongs to
created_atintegerUpload time, Unix seconds
metadataobjectMetadata supplied at upload, plus the filename the service adds
content_timestampstringISO 8601 timestamp for when the content was created/effective
top_scorenumberRelevance score of the chunk this entry belongs to
n_chunksintegerAlways 1
audio_start_secondsintegerTime in the recording where this chunk's speech starts, in seconds. null for files that are not recordings.
audio_end_secondsintegerTime in the recording where this chunk's speech ends, in seconds. null for files that are not recordings.
page_startintegerFirst PDF page or PPTX slide of this chunk. null for files that are not PDF or PPTX files.
page_endintegerLast PDF page or PPTX slide of this chunk. null for files that are not PDF or PPTX files.
sheet_startstringName of the first XLSX sheet of this chunk. null for files that are not XLSX files.
sheet_endstringName of the last XLSX sheet of this chunk. null for files that are not XLSX files.

Both times are whole seconds. The start time is rounded down and the end time is rounded up. They are also null when the transcription model returned the transcript without times, or when the chunk's text could not be placed in the transcript.

Pages and sheets are also null for a PDF with forms or tables, whose pages the RAG cannot tell apart, and when the chunk's text could not be placed in the file. Files indexed before the RAG returned pages and sheets have none.

Answer Generation​

RAG no longer generates answers. These endpoints have been removed:

  • POST {base_url}/v1/chat/completions
  • POST {base_url}/v1/evaluation/completions
  • POST {base_url}/v1/evaluation/completions/batch

For generated answers, use the Model Gateway's /v1/responses API with RAG tools. See Use tools with the Responses API for setup and examples. For raw retrieval without answer generation, use /context.

Groups API​

Manage document groups for organization and access control.

SMB data sources store file permission SIDs in metadata.source_group_ids, separately from document groups. Syncing an SMB source leaves groups managed through this API unchanged. See Connect a RAG data source.

List Groups​

GET {base_url}/groups

Headers:

  • Authorization: Bearer your-api-key

Response:

{
"groups": ["group1", "group2", "group3"]
}

Get Group Information​

GET {base_url}/groups/{group_id}

Headers:

  • Authorization: Bearer your-api-key

Response:

{
"group_id": "group1",
"document_count": 42,
"created_at": "2023-05-15T12:00:00Z",
"last_updated": "2023-10-28T14:30:22Z"
}

Get Group Documents​

GET {base_url}/groups/{group_id}/documents

Headers:

  • Authorization: Bearer your-api-key

Response:

{
"group_id": "group1",
"document_count": 42,
"documents": [
{
"id": "doc1",
"user_id": "user123",
"group_ids": ["group1", "group2"],
"metadata": {
"filename": "example.txt",
"created_at": "2023-10-15T10:30:00Z",
"tags": ["report", "finance"]
}
}
]
}

Storage Usage API​

Monitor storage usage across your RAG system for billing and capacity planning.

These totals include documents the caller cannot read. While any source enforces permissions, all three storage usage calls require RAG admin rights. Read-only callers receive 403. Otherwise, RAG read access is sufficient.

Get Storage Summary​

GET {base_url}/storage/usage/summary

Headers:

  • Authorization: Bearer your-api-key

Response:

{
"total_size_bytes": 1523456789,
"total_size": "1.42 GB",
"document_count": 1250,
"chunk_count": 45678,
"group_count": 15,
"tables_size_bytes": 987654321,
"indexes_size_bytes": 535802468,
"tables_size_percent": 64.8,
"indexes_size_percent": 35.2,
"vector_index_size_percent": 33.1
}

Get Detailed Storage Usage​

GET {base_url}/storage/usage?include_groups=true

Query Parameters:

  • include_tables: Include table-by-table breakdown (optional)
  • include_indexes: Include index-by-index breakdown (optional)
  • include_groups: Include group-by-group usage (optional)
  • include_vector: Include vector index details (optional)

Response with groups:

{
"total_size_bytes": 1523456789,
"total_size": "1.42 GB",
"document_count": 1250,
"chunk_count": 45678,
"group_usage": [
{
"group_id": "client-a",
"size_bytes": 534567890,
"content_size_bytes": 356789012,
"estimated_index_size_bytes": 177778878,
"document_count": 456,
"chunk_count": 15678,
"human_readable_size": "509.8 MB"
}
]
}

Get Group Storage Usage​

GET {base_url}/storage/usage/{group_id}

Headers:

  • Authorization: Bearer your-api-key

Get detailed storage metrics for a specific group (useful for client billing).

Response:

{
"group_id": "client-a",
"size_bytes": 534567890,
"content_size_bytes": 356789012,
"estimated_index_size_bytes": 177778878,
"document_count": 456,
"chunk_count": 15678,
"text_content_size_bytes": 145678901,
"embeddings_size_bytes": 201234567,
"metadata_size_bytes": 9875544,
"human_readable": {
"size": "509.8 MB",
"content_size": "340.3 MB",
"index_size": "169.5 MB"
}
}

For billing purposes:

  • size_bytes: Total storage including indexes (recommended for billing)
  • content_size_bytes: Content only, excluding index overhead
  • human_readable: Pre-formatted values for display

Data Sources API​

A data source connects a RAG to an external store, such as an S3 bucket or an Azure Blob Storage container. The RAG reads the files in the store, indexes them, and reads the store again later to pick up changes. The /data-sources calls let you follow that work and start it by hand.

Data sources are configured through the portal or manager API. The source_id in these paths is the UUID assigned when the source is connected to the RAG.

These calls require RAG write access. Connection tests and file cleanup require RAG admin rights.

If the data source runtime is unavailable, these calls return 503 with the code runtime_unavailable. A source with a known configuration error remains listed with sync_status=failed and latest_error.code=invalid_configuration. Its indexed documents stay hidden and sync requests return 422 until the settings or credentials are corrected. Correcting the configuration queues a sync.

How often a source syncs​

A data source with a sync interval syncs by itself. When the time in next_sync_at passes, the RAG reads the source again and picks up files that were added, changed, or removed since the last pass. New files are searchable once their status is indexed.

A source with no interval (sync_interval_seconds is null) syncs only when you ask for it.

If a sync fails, the RAG does not try again straight away. It waits for the next scheduled time. To retry sooner, start a sync yourself.

List the status of every data source​

A disconnected source stays listed while its files are still in the RAG. After you ask for those files to be removed, it stays listed with lifecycle deleting until the RAG has removed them all. It leaves the list when the removal is done, so a source that is gone from the list has nothing left in the RAG.

GET {base_url}/data-sources/status

Headers:

  • Authorization: Bearer your-api-key

Query Parameters:

  • source_id: Return only the listed source IDs (optional, repeatable)

Response:

{
"sources": [
{
"source_id": "9f3c1a20-2a1e-4b7f-9e2d-6c1b0d4a55f1",
"lifecycle": "active",
"sync_status": "succeeded",
"sync_interval_seconds": 3600,
"next_sync_at": "2025-10-01T09:15:00Z",
"last_sync_requested_at": "2025-10-01T08:15:00Z",
"last_sync_started_at": "2025-10-01T08:15:02Z",
"last_sync_completed_at": "2025-10-01T08:17:41Z",
"current_generation": 4,
"last_completed_generation": 4,
"scan_complete": true,
"total_files": 128,
"pending_files": 0,
"indexed_files": 126,
"failed_files": 2,
"latest_error": null
}
]
}

Response fields:

FieldTypeDescription
source_idstringID of the data source
lifecyclestringactive while the source is connected to the RAG, deleted once it is disconnected and its files are still in the RAG, deleting while the RAG is removing those files
sync_statusstringnever_synced, queued, syncing, succeeded, succeeded_with_errors, failed, or cancelled
sync_interval_secondsnumberHow often the source syncs on its own. null when it syncs only when you ask
next_sync_atstringWhen the next scheduled sync is due
last_sync_requested_atstringWhen a sync was last asked for
last_sync_started_atstringWhen the last sync began
last_sync_completed_atstringWhen the last sync ended
current_generationintegerCounts the passes the RAG has made over the source
last_completed_generationintegerThe last pass that finished
scan_completebooleantrue once the source has finished at least one full pass over its files
total_filesintegerFiles the RAG has seen in the source
pending_filesintegerFiles still waiting to be indexed
indexed_filesintegerFiles indexed and searchable
failed_filesintegerFiles the RAG could not index
latest_errorobjectcode and message of the last failure, or null

Start a sync now​

POST {base_url}/data-sources/{source_id}/sync

Headers:

  • Authorization: Bearer your-api-key

Asks the RAG to read the source now. The call returns as soon as the request is accepted; it does not wait for the sync to finish. Watch the source's status to see when it is done.

Response:

{
"source_id": "9f3c1a20-2a1e-4b7f-9e2d-6c1b0d4a55f1",
"sync_status": "queued"
}

sync_status is syncing when a sync is already running. In that case the RAG finishes the current pass and then makes one more, so nothing is missed. Asking several times in a row has the same effect as asking once.

If the RAG has no active source with that ID, the answer is 404 with the code data_source_not_found.

List the files of a data source​

This is the synchronization inventory, separate from the caller-filtered GET /files list. Callers with RAG write or admin access can inspect paths and errors even when they cannot read document contents. Use status=failed for failures, or omit the filter for the full inventory. Cleanup may leave inventory entries visible until they are removed by the background worker.

GET {base_url}/data-sources/{source_id}/files

Headers:

  • Authorization: Bearer your-api-key

Query Parameters:

  • limit: How many files to return, from 1 to 200 (optional, default 100)
  • status: Return only files with this status — pending, indexed, or failed (optional)
  • cursor: The next_cursor from the previous answer, to get the next page (optional)

Response:

{
"files": [
{
"source_id": "9f3c1a20-2a1e-4b7f-9e2d-6c1b0d4a55f1",
"source_key": "reports/q3-2025.pdf",
"source_version": "d41d8cd98f00b204e9800998ecf8427e",
"source_modified_at": "2025-09-30T23:59:59Z",
"desired_operation": "upsert",
"status": "indexed",
"document_id": "doc_123",
"error": null,
"updated_at": "2025-10-01T08:17:12Z"
}
],
"next_cursor": "eyJzb3VyY2VfaWQiOiI5ZjNjMWEyMCJ9"
}

Response fields:

FieldTypeDescription
source_keystringWhere the file sits in the store, for example the object key in a bucket
source_versionstringThe version the store reports for the file, or null
source_modified_atstringWhen the store last changed the file, or null
desired_operationstringupsert when the file should be indexed, delete when it should be removed from the RAG
statusstringpending, indexed, or failed
document_idstringID of the document in the RAG, once the file is indexed. Use it with the Files API
errorobjectcode and message of the last failure for this file, or null
updated_atstringWhen this entry last changed

When the RAG cannot index a file because of its format, status is failed and error.code is unsupported_format. The file is checked again on every sync, so it is indexed if the format becomes supported later, for example when an OCR model is added for images.

Paging: next_cursor is null on the last page. To get the next page, send the value back as cursor and keep the same status — a cursor only works with the filter it was made with. A cursor the RAG cannot use gets 400 with the code invalid_cursor.

Remove indexed files from a data source​

DELETE {base_url}/data-sources/{source_id}/files

Headers:

  • Authorization: Bearer your-api-key

Starts cleanup of a source's indexed documents. It can be called before disconnecting an active source, or after disconnecting it, without waiting for a configuration rollout. When removing both a source and its files, start cleanup first so its documents are hidden before source permission settings are removed.

The request marks the source as deleting. Its documents are immediately excluded from retrieval and ordinary file lists. A background worker removes documents and inventory records, then removes the source row. An already-empty source is removed immediately. Call once and poll the source list for completion. Original files in the external store are unchanged.

The response is 204 No Content. An unknown source is also 204, so the call is idempotent.

The storage the source used is freed while its files are removed, so the figures from the storage usage calls fall step by step until the removal is done.

Test a connection​

POST {base_url}/data-sources/test

Headers:

  • Authorization: Bearer your-api-key
  • Content-Type: application/json

Checks that the RAG can reach a store with the settings you give it. Nothing is saved and nothing is indexed. You need admin rights on the RAG; write is not enough.

Request Body:

{
"type": "s3",
"config": {
"endpoint": "https://s3.example.com",
"bucket": "reports",
"prefix": "2025/",
"region": "eu-north-1"
},
"credentials": {
"access_key_id": "your-access-key-id",
"secret_access_key": "your-secret-access-key"
}
}

For an Azure Storage container, send:

{
"type": "azure",
"config": {
"account_endpoint": "https://youraccount.blob.core.windows.net",
"container": "reports",
"prefix": "2025/"
},
"credentials": {
"connection_string": "your-storage-account-connection-string"
}
}
FieldDescription
account_endpointThe blob endpoint of the storage account, for example https://youraccount.blob.core.windows.net
containerName of the container to read. Lower case, 1 to 63 characters
prefixRead only the files whose name starts with this text. Send "" to read the whole container
connection_stringConnection string of the storage account

Send every field. Leaving one out, or adding one that is not listed, gets 422 with the code invalid_configuration.

account_endpoint must be the blob endpoint of the same storage account as the connection_string. If the two do not match, the RAG does not reach the store and answers 503 with the code unavailable.

Response:

{
"object_read_verified": true
}

A 200 answer means the RAG reached the store and listed it. object_read_verified is true when the RAG also read one file. It is false when the store held no file to read — for example an empty bucket, or a prefix that matches nothing.

Errors:

StatusCodeMeaning
422invalid_configurationThe settings or the credentials cannot be used
503timeoutThe store did not answer in time
503unavailableThe RAG could not reach the store

Error answers never repeat the credentials you sent.

Temporal Search Configuration​

Setting Timestamps​

For temporal search to work effectively, ensure your documents have accurate timestamps:

# When uploading files
with open("annual_report_2023.pdf", 'rb') as file:
response = requests.post(
f"{api_base_url}/files",
headers=headers,
files={"file": file},
data={
"content_timestamp": "2023-12-31T23:59:59Z" # End of 2023
}
)

# When uploading content
response = requests.post(
f"{api_base_url}/files/content",
headers=headers,
json={
"contents": [{
"id": "policy_v3",
"txt": "Updated policy effective March 2024...",
"content_timestamp": "2024-03-01T00:00:00Z" # Effective date
}]
}
)

Note: If you don't specify content_timestamp, the system uses the upload time, which is often correct for real-time content like news updates or live documentation.

Configuring Default Behavior​

Open the RAG's page in the portal. Under Settings, turn Smart temporal search by default on or off. The RAG restarts to apply the change. Wait until it is ready before sending queries.

When enabled by default, queries like "latest updates", "current policy", or "what changed recently" will automatically use temporal-aware retrieval.

Overriding the Default​

Override the RAG's default for a single request:

# Disable temporal search for this specific request
response = requests.post(
f"{api_base_url}/context",
headers=headers,
json={
"query": "python programming basics",
"smart_temporal_search": False # Override portal default
}
)

# Enable temporal search for this specific request
response = requests.post(
f"{api_base_url}/context",
headers=headers,
json={
"query": "What's the latest company policy?",
"smart_temporal_search": True # Override portal default
}
)

Code Examples​

Upload Files​

from pathlib import Path
import requests

# Configuration from portal
api_base_url = "https://api.example.com/v1/api/your-endpoint-id"
api_key = "your-api-key-from-portal"

# Headers with authorization
headers = {
"Authorization": f"Bearer {api_key}"
}

file_path = Path("/path/to/your/document.txt")

# Upload a single file
with open(file_path, 'rb') as file:
response = requests.post(
f"{api_base_url}/files",
headers=headers,
files={"file": (file_path.name, file, "application/octet-stream")}
)

file_id = response.json()["id"]
print(f"File uploaded with ID: {file_id}")

Get Context​

import requests

# Configuration from portal
api_base_url = "https://api.example.com/v1/api/your-endpoint-id"
api_key = "your-api-key-from-portal"

# Headers with authorization
headers = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json"
}

payload = {
"query": "What are the key features of the product?",
"max_chunks": 3,
"user_id": "user123" # Filter to only documents owned by this user
}

response = requests.post(
f"{api_base_url}/context",
headers=headers,
json=payload
)
result = response.json()

for i, (chunk, score) in enumerate(zip(result["chunks"], result["scores"])):
print(f"Chunk {i + 1} (score: {score:.2f}): {chunk[:100]}...")

Using /context results​

/context returns matching passages, their scores, and source documents. You can:

  • Show the passages in a search interface
  • Process them in your own code
  • Send them to a model through the Model Gateway to generate an answer

Best Practices​

  • For optimal results, use clear and specific questions that match how information is presented in your documentation
  • Use the max_chunks parameter to control how much context is retrieved (3-5 is usually optimal)
  • Apply metadata filters to narrow down results when you have a large document repository
  • For categorized documents, use the group_id parameter to search within specific document groups
  • Use the user_id parameter when you need to restrict context retrieval to documents owned by a specific user
  • For temporal search: Ensure documents have accurate content_timestamp values when the effective date differs from upload time