RAG endpoint API
The RAG (Retrieval-Augmented Generation) Endpoint is a flexible and modular system built to enhance LLM responses with relevant context from your own documents. This system provides a standardized API for document ingestion and context retrieval, with support for temporal-aware search to automatically find the most time-relevant content.
For generated answers, use agentic RAG: give an agent or model access to the RAG as a tool so it can retrieve relevant passages and generate an answer. See Use tools with the Responses API for setup and examples.
Base URL and Authentication
All requests to the RAG endpoint must include:
-
Base URL: Your endpoint-specific base URL obtained from the portal
https://api.example.com/v1/api/{endpoint-id} -
Authorization Header: Your API key or signed-in user access token in the Authorization header
Authorization: Bearer your-api-key-from-portal
Documents protected by source permissions
An SMB source can enforce file permissions. The caller needs a valid directory SID and matching file permissions. A personal API key uses the SID its owner had when the key was created. Service-account keys have no SID.
- Search and listing.
/context,GET /files, and MCP tools return only readable documents. File totals use the same filtering, including for administrators. - Direct reads. A document or chunk the caller cannot read returns
404, just like a missing document. - Groups. Only groups with readable documents are listed. A group with no readable documents returns
404. - Lookup failures. A required directory lookup failure returns
503withSource permission lookup is unavailable. Unrelated sources are skipped when explicit source or document filters restrict the request.
API Reference
Files API
The /files endpoint provides a RESTful interface for managing files in the system.
Upload a File
POST {base_url}/files
Headers:
Authorization: Bearer your-api-keyContent-Type: multipart/form-data
Parameters:
file: File data (required)user_id: ID of the user uploading the file (optional, defaults to "system")document_id: Custom ID for the document (optional, auto-generated if not provided)group_ids: List of group IDs to associate with the file (optional)metadata: JSON string of additional metadata (optional)content_timestamp: ISO 8601 timestamp for when the content was created/effective (optional, defaults to upload time)use_ocr: Enable OCR for PDFs and images when an OCR model is configured (optional)
When the endpoint has Delete files left without a group turned on, group_ids is required.
An upload without a group answers 400 with the message
Group ID is required when auto_delete_files_without_group is enabled.
The same rule applies to group_id in Upload Content.
See Configure a RAG endpoint to check or change the setting.
Example metadata:
{
"source": "email",
"author": "John Doe",
"tags": ["important", "reference"],
"custom_field": "value"
}
Example with custom timestamp:
# Upload a document with its actual creation date
with open("q3_report_2024.pdf", 'rb') as file:
response = requests.post(
f"{api_base_url}/files",
headers={"Authorization": f"Bearer {api_key}"},
files={"file": file},
data={
"content_timestamp": "2024-09-30T23:59:59Z", # Q3 end date
"metadata": json.dumps({"quarter": "Q3", "year": 2024})
}
)
Response:
{
"id": "document_id",
"filename": "example.pdf",
"created_at": 1677409068,
"status": "processed"
}
Which files can be indexed
The RAG reads the text out of the file and indexes that text.
- Documents:
.pdf,.docx,.pptx,.xlsx,.xls,.epub, and Outlook email messages (.msg) - Text:
.txt,.text,.md,.markdown,.json,.jsonl,.csv,.html,.htm, and.ipynb - Zip archives (
.zip): unpacked, and each file inside is read by the rules above
A text file is read in the character encoding it was saved in. It does not have to be UTF-8.
An image (.jpg, .jpeg, .png) is indexed when the upload sets use_ocr and an OCR model is configured.
Video files are not indexed, and neither are the audio formats .m4a, .aac, .wma, .opus, .aiff, .aif and .amr. The upload answers 415 with a message that names the format, such as File format is not supported: .mp4. The file is not read.
A file in any other format the RAG cannot read is still read first. That upload answers 400 with the message Failed to parse file <filename>.
Audio files
An audio file in .wav, .mp3, .flac, .ogg or .oga format is transcribed, and the transcript is indexed as the text of the document.
The endpoint needs a transcription model for this. Choose one on the RAG's page in the portal, as described in Configure a RAG endpoint. Without one, the upload answers 412 with the message File <filename> requires a configured transcription model.
If the transcription model cannot decode the recording, the upload answers 415 with the message Audio format of file <filename> is not supported. This also happens when the file name ends in a supported extension but the contents are in another format or are damaged. Save the recording as .wav, .mp3, .flac, .ogg or .oga and upload it again. If the transcription fails for another reason, the upload answers 400 with the message Failed to transcribe file <filename>. If the recording holds no speech, the upload answers 400 with the message No text content found in file <filename>.
Each piece of the transcript also keeps the time in the recording where its speech starts and ends. Search results carry these times, so an answer can point back to the right moment of a call or a meeting. An agent can also use them to read the recording around a result, as described in Call tools on the MCP endpoint.
Some transcription models return the transcript without times. The recording is still transcribed, indexed and searchable, but its results carry no times.
List Files
GET {base_url}/files
Headers:
Authorization: Bearer your-api-key
Query Parameters:
user_id: Filter files by user ID (optional)group_id: Filter files by group ID (optional)source_id: Return only the files of this data source. Usenullto return only manually uploaded files (optional)limit: How many files to return, from 1 to 500. Omit it to return all matching files (optional)cursor: Thenext_cursorfrom the previous response (optional)search: Return only the files whose name contains this text. Upper and lower case do not matter. Up to 256 characters (optional)include_total: Set totrueto also return the exact number of files matchinguser_id,group_id,source_id, andsearch(optional, defaultfalse)
Response:
{
"files": [
{
"id": "doc_123",
"user_id": "user_456",
"group_ids": ["group_789", "group_101"],
"metadata": {
"filename": "report.pdf",
"created_at": "2025-02-26T12:32:39.265082",
"source": "upload",
"author": "Jane Smith"
}
}
],
"next_cursor": "eyJmaWx0ZXJzIjp7fSwib3BlcmF0aW9uIjoiZmlsZXMubGlzdCJ9",
"total": 237
}
total appears only when include_total=true. It is the exact count for the active filter set, independent of limit and cursor, so you can show a full total without paging through results. Without the parameter the response is unchanged.
Files from a data source also include source_id, source_key, and source_version.
Paging: Set limit to enable paging. next_cursor is null on the last page. To get the next page, send it back as cursor and keep the same user_id, group_id, and source_id. An invalid cursor returns 400.
Delete a File
DELETE {base_url}/files/{file_id}
Headers:
Authorization: Bearer your-api-key
Parameters:
file_id: ID of the file to delete (path parameter)
Response:
{
"success": true,
"message": "File doc_123 deleted successfully",
"files_deleted": ["doc_123"]
}
If the file came from a connected data source, deleting it is temporary. The next sync indexes it again. Remove it from the source store to keep it out of the RAG.
Content API
Use POST /files to upload a file, or POST /files/content to send text directly. Use /files/{file_id}
to manage stored content. The older /content routes still work but are deprecated:
| Deprecated route | Use instead |
|---|---|
POST /content | POST /files/content |
GET /content/{content_id} | GET /files/{file_id} |
PATCH /content/{content_id} | PATCH /files/{file_id} |
DELETE /content/{content_id} | DELETE /files/{file_id} |
GET /content/{content_id}/chunks | GET /files/{file_id}/chunks |
The new routes take the same request bodies as the old ones. DELETE /files/{file_id} also returns a
message field, which the old route omits.
Upload Content
POST {base_url}/files/content
Headers:
Authorization: Bearer your-api-keyContent-Type: application/json
Request Body:
{
"contents": [
{
"id": "doc_2024_q1",
"txt": "Q1 2024 Financial Report...",
"content_timestamp": "2024-03-31T23:59:59Z", // Optional: defaults to upload time
"metadata": {
"type": "financial_report",
"quarter": "Q1"
}
}
],
"user_id": "finance_team",
"group_id": "financial_reports"
}
Important: The content_timestamp field allows you to specify when the content was created or is effective from. This is crucial for temporal search to work correctly when documents have specific dates (e.g., dated reports, versioned documentation).
Get Content
GET {base_url}/files/{file_id}
Headers:
Authorization: Bearer your-api-key
Parameters:
file_id: ID of the item to read (path parameter)
Response: The stored item, including id, user_id, group_ids, metadata, content_timestamp,
created_at, and source_text when available.
If no item has this ID, the endpoint returns 404 with the detail Content not found.
Update Content
PATCH {base_url}/files/{file_id}
Request Body:
{
"txt": "Updated content...", // Optional
"content_timestamp": "2024-04-01T00:00:00Z", // Optional
"metadata": {
"revised": true
}
}
Context API
The /context endpoint provides a flexible way to retrieve relevant chunks of text from your document repository based on a query.
POST {base_url}/context
Headers:
Authorization: Bearer your-api-keyContent-Type: application/json
Request Body:
{
"query": "your search query",
"max_chunks": 4,
"filter_ids": ["doc1", "doc2"],
"group_id": "group1",
"user_id": "user123",
"metadata_filters": [
{
"field": "author",
"value": "Alice",
"operator": "eq"
},
{
"field": "rating",
"value": 4.5,
"operator": "gt"
}
],
"smart_temporal_search": true // Optional: override default temporal search setting
}
Temporal Search: When smart_temporal_search is enabled (either by the RAG's Smart temporal search by default setting or explicitly in the request), the system automatically detects temporal intent in queries like "latest policy", "what changed last month", or "original version" and retrieves the most time-relevant content.
Parameters:
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
query | string | Yes | - | The search query to find relevant text chunks |
max_chunks | integer | No | 4 | Maximum number of chunks to return |
reranker_top_n | integer | No | 16 | Number of candidate chunks to retrieve for reranking |
reranker_batch_size | integer | No | 16 | Maximum number of chunks scored in each reranker batch |
filter_ids | array | No | null | List of document IDs to restrict the search to |
group_id | string | No | null | Group ID to filter documents by (for permission-based filtering) |
user_id | string | No | null | User ID to filter documents by (only returns documents owned by this user) |
metadata_filters | array | No | null | List of metadata filters to apply |
smart_temporal_search | boolean | No | The RAG's Smart temporal search by default setting | Enable/disable intelligent temporal search |
When the endpoint has Require group for retrieval turned on, a request without group_id returns no chunks.
See Configure a RAG endpoint to check or change the setting.
The reranker parameters apply only when the endpoint has a reranker model selected. Without one, the endpoint skips reranking and ignores these two parameters.
Metadata Filters
Each metadata filter object supports:
| Field | Type | Required | Description |
|---|---|---|---|
field | string | Yes | Metadata field path (e.g., "author" or "details.category") |
value | scalar or scalar array | Yes | Scalar to match; flat scalar arrays are supported only by contains |
operator | string | No | Operator to use (default: "eq") |
Supported operators:
eq: Exact equality (default)contains: Case-insensitive substring match for strings, or element match for arraysgt: Greater thangte: Greater than or equal tolt: Less thanlte: Less than or equal to
eq compares the stored JSON value exactly, including its type and string case. Send 4.5, not "4.5", to match a
JSON number, and send true, not "true", to match a boolean. A lossless exact-number object returned by metadata
discovery can also be copied into a numeric equality filter. Use contains rather than eq for a stored array.
Only contains accepts a filter value that is an array; it matches when any requested value matches. Scalar string
contains uses case-insensitive substring matching, while stored arrays use case-insensitive string membership and
typed equality for non-string elements.
Range operators accept finite JSON numbers, canonical numeric strings, strict ISO dates such as 2026-08-17, local
ISO times such as 09:30:15.125, and timezone-qualified ISO datetimes such as 2026-08-17T10:30:00Z. Dates represent
midnight UTC when compared with datetimes. Time-only values do not accept a timezone because they have no date for a
UTC conversion. If a stored field is an array, any compatible scalar element can satisfy a range filter. Incompatible
stored types do not match the filter.
All metadata filters in one request use AND: every filter must match before a document is eligible for the existing
semantic ranking and reranking.
Limits for one request:
| Limit | Value |
|---|---|
| Filters | 20 |
| Field path | 200 characters and 5 dot-separated segments using letters, numbers, _, -, or : |
Values in one contains array | 50 |
| Encoded size of one filter value | 4 KiB |
| Encoded size of all metadata filters | 16 KiB |
The Context API returns 422 Unprocessable Entity without searching documents when a metadata filter is invalid.
Common causes include an unsupported operator, an invalid field path, an array value used with an operator other than
contains, an unsupported range value, or a request that exceeds one of these limits. An MCP search_documents call
with the same invalid input fails as a tool-call error instead of returning a Context API HTTP status.
Advanced Temporal Parameters (for testing/evaluation):
temporal_mode: Manual override for temporal behavior ("earliest", "latest", "specific", "range", "none")temporal_weight: Balance between semantic and temporal relevance (0-1)temporal_value: ISO timestamp or date range for specific/range modestemporal_strictness: How strictly to enforce temporal boundaries (0-1)
Response:
{
"chunks": [
"This is a document about Python programming...",
"Machine learning frameworks in Python include..."
],
"scores": [0.92, 0.87],
"files": [
{
"id": "doc1",
"user_id": "user123",
"group_ids": ["group1", "group2"],
"created_at": 1728987000,
"metadata": {
"filename": "python_intro.pdf",
"author": "Alice",
"topic": "programming",
"tags": ["python", "beginner"],
"rating": 4.5
},
"content_timestamp": "2024-10-15T10:30:00Z",
"top_score": 0.92,
"n_chunks": 1
},
{
"id": "doc2",
"user_id": "user456",
"group_ids": ["group1"],
"created_at": 1728987500,
"metadata": {
"filename": "ml_frameworks.pdf",
"author": "Bob",
"topic": "machine learning",
"tags": ["python", "advanced"],
"rating": 4.8
},
"content_timestamp": "2024-10-15T10:30:00Z",
"top_score": 0.87,
"n_chunks": 1
}
]
}
Response fields:
| Field | Type | Description |
|---|---|---|
chunks | array of strings | The retrieved text chunks |
scores | array of numbers | Relevance score per chunk, same order as chunks |
files | array of objects | Source document for each chunk, same order as chunks (empty when nothing matched) |
files holds one entry per returned chunk, not per document: top_score is that chunk's score and n_chunks is always 1, so the same id repeats when several chunks come from one document.
Each files entry contains:
| Field | Type | Description |
|---|---|---|
id | string | Document ID |
user_id | string | Owner of the document ("system" when uploaded without a user_id) |
group_ids | array of strings | Groups the document belongs to |
created_at | integer | Upload time, Unix seconds |
metadata | object | Metadata supplied at upload, plus the filename the service adds |
content_timestamp | string | ISO 8601 timestamp for when the content was created/effective |
top_score | number | Relevance score of the chunk this entry belongs to |
n_chunks | integer | Always 1 |
audio_start_seconds | integer | Time in the recording where this chunk's speech starts, in seconds. null for files that are not recordings. |
audio_end_seconds | integer | Time in the recording where this chunk's speech ends, in seconds. null for files that are not recordings. |
page_start | integer | First PDF page or PPTX slide of this chunk. null for files that are not PDF or PPTX files. |
page_end | integer | Last PDF page or PPTX slide of this chunk. null for files that are not PDF or PPTX files. |
sheet_start | string | Name of the first XLSX sheet of this chunk. null for files that are not XLSX files. |
sheet_end | string | Name of the last XLSX sheet of this chunk. null for files that are not XLSX files. |
Both times are whole seconds. The start time is rounded down and the end time is rounded up. They are also null when the transcription model returned the transcript without times, or when the chunk's text could not be placed in the transcript.
Pages and sheets are also null for a PDF with forms or tables, whose pages the RAG cannot tell apart, and when the chunk's text could not be placed in the file. Files indexed before the RAG returned pages and sheets have none.
Answer Generation
RAG no longer generates answers. These endpoints have been removed:
POST {base_url}/v1/chat/completionsPOST {base_url}/v1/evaluation/completionsPOST {base_url}/v1/evaluation/completions/batch
For generated answers, use the Model Gateway's /v1/responses API with RAG tools. See
Use tools with the Responses API for setup and examples.
For raw retrieval without answer generation, use /context.
Groups API
Manage document groups for organization and access control.
SMB data sources store file permission SIDs in metadata.source_group_ids, separately from document groups.
Syncing an SMB source leaves groups managed through this API unchanged. See
Connect a RAG data source.
List Groups
GET {base_url}/groups
Headers:
Authorization: Bearer your-api-key
Response:
{
"groups": ["group1", "group2", "group3"]
}
Get Group Information
GET {base_url}/groups/{group_id}
Headers:
Authorization: Bearer your-api-key
Response:
{
"group_id": "group1",
"document_count": 42,
"created_at": "2023-05-15T12:00:00Z",
"last_updated": "2023-10-28T14:30:22Z"
}
Get Group Documents
GET {base_url}/groups/{group_id}/documents
Headers:
Authorization: Bearer your-api-key
Response:
{
"group_id": "group1",
"document_count": 42,
"documents": [
{
"id": "doc1",
"user_id": "user123",
"group_ids": ["group1", "group2"],
"metadata": {
"filename": "example.txt",
"created_at": "2023-10-15T10:30:00Z",
"tags": ["report", "finance"]
}
}
]
}
Storage Usage API
Monitor storage usage across your RAG system for billing and capacity planning.
These totals include documents the caller cannot read. While any source enforces permissions, all three storage
usage calls require RAG admin rights. Read-only callers receive 403. Otherwise, RAG read access is sufficient.
Get Storage Summary
GET {base_url}/storage/usage/summary
Headers:
Authorization: Bearer your-api-key
Response:
{
"total_size_bytes": 1523456789,
"total_size": "1.42 GB",
"document_count": 1250,
"chunk_count": 45678,
"group_count": 15,
"tables_size_bytes": 987654321,
"indexes_size_bytes": 535802468,
"tables_size_percent": 64.8,
"indexes_size_percent": 35.2,
"vector_index_size_percent": 33.1
}
Get Detailed Storage Usage
GET {base_url}/storage/usage?include_groups=true
Query Parameters:
include_tables: Include table-by-table breakdown (optional)include_indexes: Include index-by-index breakdown (optional)include_groups: Include group-by-group usage (optional)include_vector: Include vector index details (optional)
Response with groups:
{
"total_size_bytes": 1523456789,
"total_size": "1.42 GB",
"document_count": 1250,
"chunk_count": 45678,
"group_usage": [
{
"group_id": "client-a",
"size_bytes": 534567890,
"content_size_bytes": 356789012,
"estimated_index_size_bytes": 177778878,
"document_count": 456,
"chunk_count": 15678,
"human_readable_size": "509.8 MB"
}
]
}
Get Group Storage Usage
GET {base_url}/storage/usage/{group_id}
Headers:
Authorization: Bearer your-api-key
Get detailed storage metrics for a specific group (useful for client billing).
Response:
{
"group_id": "client-a",
"size_bytes": 534567890,
"content_size_bytes": 356789012,
"estimated_index_size_bytes": 177778878,
"document_count": 456,
"chunk_count": 15678,
"text_content_size_bytes": 145678901,
"embeddings_size_bytes": 201234567,
"metadata_size_bytes": 9875544,
"human_readable": {
"size": "509.8 MB",
"content_size": "340.3 MB",
"index_size": "169.5 MB"
}
}
For billing purposes:
size_bytes: Total storage including indexes (recommended for billing)content_size_bytes: Content only, excluding index overheadhuman_readable: Pre-formatted values for display
Data Sources API
A data source connects a RAG to an external store, such as an S3 bucket or an Azure Blob Storage container. The RAG reads the files in the store, indexes them, and reads the store again later to pick up changes. The /data-sources calls let you follow that work and start it by hand.
Data sources are configured through the portal or manager API. The source_id in these paths is the UUID assigned
when the source is connected to the RAG.
These calls require RAG write access. Connection tests and file cleanup require RAG admin rights.
If the data source runtime is unavailable, these calls return 503 with the code runtime_unavailable.
A source with a known configuration error remains listed with sync_status=failed and
latest_error.code=invalid_configuration. Its indexed documents stay hidden and sync requests return 422
until the settings or credentials are corrected. Correcting the configuration queues a sync.
How often a source syncs
A data source with a sync interval syncs by itself. When the time in next_sync_at passes, the RAG reads the source again and picks up files that were added, changed, or removed since the last pass. New files are searchable once their status is indexed.
A source with no interval (sync_interval_seconds is null) syncs only when you ask for it.
If a sync fails, the RAG does not try again straight away. It waits for the next scheduled time. To retry sooner, start a sync yourself.
List the status of every data source
A disconnected source stays listed while its files are still in the RAG. After you ask for those files to be removed, it stays listed with lifecycle deleting until the RAG has removed them all. It leaves the list when the removal is done, so a source that is gone from the list has nothing left in the RAG.
GET {base_url}/data-sources/status
Headers:
Authorization: Bearer your-api-key
Query Parameters:
source_id: Return only the listed source IDs (optional, repeatable)
Response:
{
"sources": [
{
"source_id": "9f3c1a20-2a1e-4b7f-9e2d-6c1b0d4a55f1",
"lifecycle": "active",
"sync_status": "succeeded",
"sync_interval_seconds": 3600,
"next_sync_at": "2025-10-01T09:15:00Z",
"last_sync_requested_at": "2025-10-01T08:15:00Z",
"last_sync_started_at": "2025-10-01T08:15:02Z",
"last_sync_completed_at": "2025-10-01T08:17:41Z",
"current_generation": 4,
"last_completed_generation": 4,
"scan_complete": true,
"total_files": 128,
"pending_files": 0,
"indexed_files": 126,
"failed_files": 2,
"latest_error": null
}
]
}
Response fields:
| Field | Type | Description |
|---|---|---|
source_id | string | ID of the data source |
lifecycle | string | active while the source is connected to the RAG, deleted once it is disconnected and its files are still in the RAG, deleting while the RAG is removing those files |
sync_status | string | never_synced, queued, syncing, succeeded, succeeded_with_errors, failed, or cancelled |
sync_interval_seconds | number | How often the source syncs on its own. null when it syncs only when you ask |
next_sync_at | string | When the next scheduled sync is due |
last_sync_requested_at | string | When a sync was last asked for |
last_sync_started_at | string | When the last sync began |
last_sync_completed_at | string | When the last sync ended |
current_generation | integer | Counts the passes the RAG has made over the source |
last_completed_generation | integer | The last pass that finished |
scan_complete | boolean | true once the source has finished at least one full pass over its files |
total_files | integer | Files the RAG has seen in the source |
pending_files | integer | Files still waiting to be indexed |
indexed_files | integer | Files indexed and searchable |
failed_files | integer | Files the RAG could not index |
latest_error | object | code and message of the last failure, or null |
Start a sync now
POST {base_url}/data-sources/{source_id}/sync
Headers:
Authorization: Bearer your-api-key
Asks the RAG to read the source now. The call returns as soon as the request is accepted; it does not wait for the sync to finish. Watch the source's status to see when it is done.
Response:
{
"source_id": "9f3c1a20-2a1e-4b7f-9e2d-6c1b0d4a55f1",
"sync_status": "queued"
}
sync_status is syncing when a sync is already running. In that case the RAG finishes the current pass and then makes one more, so nothing is missed. Asking several times in a row has the same effect as asking once.
If the RAG has no active source with that ID, the answer is 404 with the code data_source_not_found.
List the files of a data source
This is the synchronization inventory, separate from the caller-filtered GET /files list. Callers with RAG write
or admin access can inspect paths and errors even when they cannot read document contents. Use status=failed
for failures, or omit the filter for the full inventory. Cleanup may leave inventory entries visible until they
are removed by the background worker.
GET {base_url}/data-sources/{source_id}/files
Headers:
Authorization: Bearer your-api-key
Query Parameters:
limit: How many files to return, from 1 to 200 (optional, default 100)status: Return only files with this status —pending,indexed, orfailed(optional)cursor: Thenext_cursorfrom the previous answer, to get the next page (optional)
Response:
{
"files": [
{
"source_id": "9f3c1a20-2a1e-4b7f-9e2d-6c1b0d4a55f1",
"source_key": "reports/q3-2025.pdf",
"source_version": "d41d8cd98f00b204e9800998ecf8427e",
"source_modified_at": "2025-09-30T23:59:59Z",
"desired_operation": "upsert",
"status": "indexed",
"document_id": "doc_123",
"error": null,
"updated_at": "2025-10-01T08:17:12Z"
}
],
"next_cursor": "eyJzb3VyY2VfaWQiOiI5ZjNjMWEyMCJ9"
}
Response fields:
| Field | Type | Description |
|---|---|---|
source_key | string | Where the file sits in the store, for example the object key in a bucket |
source_version | string | The version the store reports for the file, or null |
source_modified_at | string | When the store last changed the file, or null |
desired_operation | string | upsert when the file should be indexed, delete when it should be removed from the RAG |
status | string | pending, indexed, or failed |
document_id | string | ID of the document in the RAG, once the file is indexed. Use it with the Files API |
error | object | code and message of the last failure for this file, or null |
updated_at | string | When this entry last changed |
When the RAG cannot index a file because of its format, status is failed and error.code is unsupported_format. The file is checked again on every sync, so it is indexed if the format becomes supported later, for example when an OCR model is added for images.
Paging: next_cursor is null on the last page. To get the next page, send the value back as cursor and keep the same status — a cursor only works with the filter it was made with. A cursor the RAG cannot use gets 400 with the code invalid_cursor.
Remove indexed files from a data source
DELETE {base_url}/data-sources/{source_id}/files
Headers:
Authorization: Bearer your-api-key
Starts cleanup of a source's indexed documents. It can be called before disconnecting an active source, or after disconnecting it, without waiting for a configuration rollout. When removing both a source and its files, start cleanup first so its documents are hidden before source permission settings are removed.
The request marks the source as deleting. Its documents are immediately excluded from retrieval and ordinary
file lists. A background worker removes documents and inventory records, then removes the source row. An
already-empty source is removed immediately. Call once and poll the source list for completion. Original files
in the external store are unchanged.
The response is 204 No Content. An unknown source is also 204, so the call is idempotent.
The storage the source used is freed while its files are removed, so the figures from the storage usage calls fall step by step until the removal is done.
Test a connection
POST {base_url}/data-sources/test
Headers:
Authorization: Bearer your-api-keyContent-Type: application/json
Checks that the RAG can reach a store with the settings you give it. Nothing is saved and nothing is indexed. You need admin rights on the RAG; write is not enough.
Request Body:
{
"type": "s3",
"config": {
"endpoint": "https://s3.example.com",
"bucket": "reports",
"prefix": "2025/",
"region": "eu-north-1"
},
"credentials": {
"access_key_id": "your-access-key-id",
"secret_access_key": "your-secret-access-key"
}
}
For an Azure Storage container, send:
{
"type": "azure",
"config": {
"account_endpoint": "https://youraccount.blob.core.windows.net",
"container": "reports",
"prefix": "2025/"
},
"credentials": {
"connection_string": "your-storage-account-connection-string"
}
}
| Field | Description |
|---|---|
account_endpoint | The blob endpoint of the storage account, for example https://youraccount.blob.core.windows.net |
container | Name of the container to read. Lower case, 1 to 63 characters |
prefix | Read only the files whose name starts with this text. Send "" to read the whole container |
connection_string | Connection string of the storage account |
Send every field. Leaving one out, or adding one that is not listed, gets 422 with the code invalid_configuration.
account_endpoint must be the blob endpoint of the same storage account as the connection_string. If the two do not match, the RAG does not reach the store and answers 503 with the code unavailable.
Response:
{
"object_read_verified": true
}
A 200 answer means the RAG reached the store and listed it. object_read_verified is true when the RAG also read one file. It is false when the store held no file to read — for example an empty bucket, or a prefix that matches nothing.
Errors:
| Status | Code | Meaning |
|---|---|---|
422 | invalid_configuration | The settings or the credentials cannot be used |
503 | timeout | The store did not answer in time |
503 | unavailable | The RAG could not reach the store |
Error answers never repeat the credentials you sent.
Temporal Search Configuration
Setting Timestamps
For temporal search to work effectively, ensure your documents have accurate timestamps:
# When uploading files
with open("annual_report_2023.pdf", 'rb') as file:
response = requests.post(
f"{api_base_url}/files",
headers=headers,
files={"file": file},
data={
"content_timestamp": "2023-12-31T23:59:59Z" # End of 2023
}
)
# When uploading content
response = requests.post(
f"{api_base_url}/files/content",
headers=headers,
json={
"contents": [{
"id": "policy_v3",
"txt": "Updated policy effective March 2024...",
"content_timestamp": "2024-03-01T00:00:00Z" # Effective date
}]
}
)
Note: If you don't specify content_timestamp, the system uses the upload time, which is often correct for real-time content like news updates or live documentation.
Configuring Default Behavior
Open the RAG's page in the portal. Under Settings, turn Smart temporal search by default on or off. The RAG restarts to apply the change. Wait until it is ready before sending queries.
When enabled by default, queries like "latest updates", "current policy", or "what changed recently" will automatically use temporal-aware retrieval.
Overriding the Default
Override the RAG's default for a single request:
# Disable temporal search for this specific request
response = requests.post(
f"{api_base_url}/context",
headers=headers,
json={
"query": "python programming basics",
"smart_temporal_search": False # Override portal default
}
)
# Enable temporal search for this specific request
response = requests.post(
f"{api_base_url}/context",
headers=headers,
json={
"query": "What's the latest company policy?",
"smart_temporal_search": True # Override portal default
}
)
Code Examples
Upload Files
from pathlib import Path
import requests
# Configuration from portal
api_base_url = "https://api.example.com/v1/api/your-endpoint-id"
api_key = "your-api-key-from-portal"
# Headers with authorization
headers = {
"Authorization": f"Bearer {api_key}"
}
file_path = Path("/path/to/your/document.txt")
# Upload a single file
with open(file_path, 'rb') as file:
response = requests.post(
f"{api_base_url}/files",
headers=headers,
files={"file": (file_path.name, file, "application/octet-stream")}
)
file_id = response.json()["id"]
print(f"File uploaded with ID: {file_id}")
Get Context
import requests
# Configuration from portal
api_base_url = "https://api.example.com/v1/api/your-endpoint-id"
api_key = "your-api-key-from-portal"
# Headers with authorization
headers = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json"
}
payload = {
"query": "What are the key features of the product?",
"max_chunks": 3,
"user_id": "user123" # Filter to only documents owned by this user
}
response = requests.post(
f"{api_base_url}/context",
headers=headers,
json=payload
)
result = response.json()
for i, (chunk, score) in enumerate(zip(result["chunks"], result["scores"])):
print(f"Chunk {i + 1} (score: {score:.2f}): {chunk[:100]}...")
Using /context results
/context returns matching passages, their scores, and source documents. You can:
- Show the passages in a search interface
- Process them in your own code
- Send them to a model through the Model Gateway to generate an answer
Best Practices
- For optimal results, use clear and specific questions that match how information is presented in your documentation
- Use the
max_chunksparameter to control how much context is retrieved (3-5 is usually optimal) - Apply metadata filters to narrow down results when you have a large document repository
- For categorized documents, use the
group_idparameter to search within specific document groups - Use the
user_idparameter when you need to restrict context retrieval to documents owned by a specific user - For temporal search: Ensure documents have accurate
content_timestampvalues when the effective date differs from upload time