Connect a RAG data source
A data source keeps files from an S3-compatible bucket, Azure Blob container, or SMB file share synchronized with a RAG endpoint.
Fields
These fields apply to all providers:
| Field | Description |
|---|---|
| Name | Display name used in the portal |
| Prefix | Optional path inside the bucket, container, or share, such as knowledge-base/. For an SMB share, enter a folder path with no slash at the start or the end, such as docs/2026. Leave blank to sync all files |
| Sync interval | How often the RAG re-reads the source. Choose Manual, 10 minutes, 1 hour, 1 day, 7 days, or a custom interval of at least 10 minutes |
The default sync interval is 1 day. A manual source can be synchronized from its Sync action.
S3 fields
| Field | Description | Example |
|---|---|---|
| Endpoint | S3 API endpoint, including https:// | https://s3.eu-west-1.amazonaws.com |
| Region | Region containing the bucket | eu-west-1 |
| Bucket | Bucket name without a path or URL | company-documents |
| Access key ID | Access key for an identity allowed to list and read the bucket | AKIA... |
| Secret access key | Secret for the access key ID | Shown only when the key is created |
For another S3-compatible provider, use the endpoint, region, and access keys supplied by that provider.
Azure Blob fields
| Field | Description | Example |
|---|---|---|
| Container | Container name without a path or URL | company-documents |
| Connection string | Azure Storage connection string containing the account credentials | DefaultEndpointsProtocol=https;AccountName=... |
SMB file share fields
| Field | Description | Example |
|---|---|---|
| Server | Host name or IP address of the SMB server, without smb://, a share, or a port | fileserver.example.com |
| Share | Share name exported by the SMB server. For \\fileserver\documents, enter documents | documents |
| Port | Port the SMB server listens on. Standard SMB uses 445 | 445 |
| Username | Account allowed to list the share, read its files, and read their permissions | rag-reader |
| Password | Password for that account | |
| Domain | Optional Windows or Active Directory domain for the account. Leave blank for an account local to the SMB server | CORP |
| Enforce source permissions | Optional. Limits search and file reads to the documents the caller's own account or groups can read on the file server | |
| LDAP URL | Shown when enforcement is on. Address of the directory that resolves caller group membership. It must start with ldaps:// | ldaps://ad.example.com:636 |
| LDAP CA certificate | Shown when enforcement is on. PEM certificate chain that verifies the LDAP server certificate | -----BEGIN CERTIFICATE----- |
| LDAP bind username | Shown when enforcement is on. Account allowed to read group membership in the directory | [email protected] |
| LDAP bind password | Password for the LDAP bind account |
If the port is not a whole number from 1 to 65535, the portal shows Port must be an integer between 1 and 65535. and does not save the source.
Select Require SMB encryption to encrypt the connection to the file server. Leave it clear if the file server does not support SMB encryption. The setting is stored with the source, and the source details show Required or Not required.
Select Enforce source permissions to limit each caller to the files their own account can read on the file server. The source details then show Enforced under Source permissions, together with the LDAP URL. See Enforce file permissions.
Which files are indexed
Files are read the same way as file uploads. See Which files can be indexed for the formats.
Audio and video files are never downloaded. They are listed as failed with an error that names the format, such as Data source file format is not supported: .mp4. A file of one of these formats that was indexed before is listed as failed after the next sync.
An image file (.jpg, .jpeg, .png, .gif, .bmp, .tiff, .webp) is indexed only when the RAG has an OCR model. Without one, the image is listed as failed with an error such as Data source image requires a configured vision model: .png. Select an OCR model on the RAG's page in the portal, then use the source's Sync action to index the images.
Any other file the RAG cannot read is also listed as failed. The other files are still indexed.
After the platform is updated, files that failed before are tried again on the next sync. To index them right away, use the source's Sync action.
Before you start
You need:
- Admin access to the RAG endpoint in the portal
- Permission to create or manage the source storage and its credentials
- A source endpoint reachable from the RAG deployment
For S3 and Azure Blob, the source endpoint must resolve to a public address. Keep the bucket or container private; public object access is not required. For an SMB file share, you also need an account on the file server that is allowed to list the share, read its files, and read the permissions of those files, and the port the file server listens on. The file server only needs to be reachable from the RAG deployment, so it can sit on a private network.
Set up Amazon S3
Already have a bucket? Start at step 2. Already have the policy? Start at step 3. Already have an access key with the required policy? Start at step 4.
1. Create a bucket
- Open the Amazon S3 console and select Create bucket.
- Select General purpose as the bucket type.
- Enter a unique bucket name.
- Select the AWS region closest to the RAG deployment. Record its region code, such as
eu-west-1. - Keep Block all public access enabled.
- Select Create bucket.
- Open the bucket and upload the files to synchronize.
See Create a bucket for all AWS bucket options.
2. Create a read-only IAM policy
The connector uses these permissions:
s3:ListBucketto find objectss3:GetObjectto inspect and download objects
Create a policy for only the source bucket:
- Open the IAM console.
- Select Policies → Create policy.
- Select JSON and enter the policy below.
- Replace both instances of
company-documentswith the bucket name. - Select Next, name the policy, and select Create policy.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::company-documents"
},
{
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::company-documents/*"
}
]
}
If the objects use an AWS KMS customer-managed key, also allow kms:Decrypt on that key.
See Create an IAM policy for the full AWS procedure.
3. Create the connector identity and key
- In the IAM console, select Users → Create user.
- Enter a name such as
confidentialmind-rag-reader. Console access is not needed. - On the permissions step, attach the policy created above.
- Create the user, open it, then select Security credentials.
- Under Access keys, select Create access key.
- Select Other → Next → Create access key.
- Copy the Access key ID and Secret access key, or download the CSV. The secret cannot be retrieved later.
The AWS administrator performing these steps needs s3:CreateBucket, iam:CreatePolicy, iam:CreateUser,
iam:AttachUserPolicy, and iam:CreateAccessKey, plus the read and list actions used by the AWS console. The
credentials given to the portal need only the S3 permissions in the policy above.
See Create an IAM user and Create an access key for the corresponding AWS instructions.
4. Add the source in the portal
- Open RAG and select the endpoint.
- Open the Files tab and select Add source.
- Select S3.
- Enter a name.
- Set Endpoint to the regional S3 endpoint. For example, use
https://s3.eu-west-1.amazonaws.comforeu-west-1. See the S3 endpoint list. - Enter the region code and bucket name.
- Enter a prefix to limit synchronization to that path, or leave it blank.
- Enter the access key ID and secret access key.
- Select a sync interval.
- Wait for Connection succeeded, then select Create.
If the test connects but cannot read an object, check the bucket name, prefix, IAM policy, and KMS permissions.
Set up Azure Blob Storage
Already have a storage account? Start at step 3. Already have a container? Start at step 4. Already have the container name and connection string? Start at step 5.
1. Check your Azure permissions
To create a storage account, you need a role such as Contributor on the target resource group. To view its
connection strings, your role must include Microsoft.Storage/storageAccounts/listkeys/action. Owner,
Contributor, and Storage Account Key Operator Service Role include this action.
See Manage storage account access keys for the Azure permission details.
2. Create a storage account
- Open the Azure portal and select Storage accounts → Create.
- Select the subscription and resource group.
- Enter a unique storage account name and select a region.
- Select Standard performance unless the workload requires Premium storage.
- Select the required redundancy option.
- Under Advanced, keep Allow storage account key access enabled.
- Under Networking, enable public network access. Choose one of these scopes:
- Enable from all networks.
- Enable from selected virtual networks and IP addresses, then allow the RAG deployment's outbound IP.
- Select Review + create → Create.
- When deployment finishes, select Go to resource.
Private endpoints are not currently supported by this connector. Public network access does not make blobs public; access still requires the connection string.
See Create a storage account and Azure Storage network security for all account and network options.
3. Create a container
- In the storage account, select Data storage → Containers.
- Select + Container.
- Enter a lowercase container name.
- Set Anonymous access level to Private (no anonymous access).
- Select Create.
- Open the container and upload the files to synchronize.
See Manage blob containers for all container options.
4. Copy the connection string
- In the storage account, select Security + networking → Access keys.
- Select Show keys and copy a Connection string from either key.
An account-key connection string grants full access to the storage account data, although the connector only lists and downloads blobs. Use a dedicated storage account if the credential should not cover unrelated data. Store the connection string as a secret and rotate it if it is exposed.
5. Add the source in the portal
- Open RAG and select the endpoint.
- Open the Files tab and select Add source.
- Select Azure Blob Storage.
- Enter a name.
- Enter the container name.
- Enter a prefix to limit synchronization to that path, or leave it blank.
- Paste the connection string.
- Select a sync interval.
- Wait for Connection succeeded, then select Create.
If the test connects but cannot read a blob, check the container, prefix, network rules, and connection string.
Set up an SMB file share
The RAG deployment connects to the Server directly, so use an address it can reach:
| File server location | Server example |
|---|---|
| Kubernetes service in the same namespace | samba |
| Kubernetes service in another namespace | samba.storage.svc.cluster.local |
| Docker Desktop host | host.docker.internal |
| Private network or VPN | fileserver.corp.local or 10.20.30.40 |
| External server | fileserver.example.com |
Note: In Kubernetes, localhost refers to the RAG pod itself. Use another address unless the SMB server runs in that pod.
Before creating the source:
- Confirm that the cluster can resolve the server address and reach its SMB port.
- Enter only the host name or IP address in Server. Do not include
smb://, the share, or the port. - Enter the share and port in their own fields.
- Open RAG and select the endpoint.
- Open the Files tab and select Add source.
- Select SMB file share.
- Enter a name.
- Enter the server and the name of the share.
- Keep the port at
445unless the file server listens on another port. - Select Require SMB encryption if the connection to the file server must be encrypted.
- Select Enforce source permissions if each caller must only reach the files their own account can read on the file server. Enter the LDAP URL and paste the LDAP CA certificate.
- Enter a prefix to limit synchronization to a folder inside the share, such as
docs/2026, with no slash at the start or the end. Leave it blank to synchronize the whole share. - Enter the username and password of the account that can read the share. Enter the domain if the account belongs to a Windows domain, or leave it blank.
- If you selected Enforce source permissions, enter the LDAP bind username and LDAP bind password.
- Select a sync interval.
- Wait for Connection succeeded, then select Create.
If the test connects but cannot read a file, check the share name, the prefix, and the read permissions of the account.
Use the account of another source
When the RAG endpoint already has an SMB source that uses the same account, the form shows a Credentials list. Select Copy from that source, and the username, password, domain and LDAP bind fields disappear. The new source then uses the stored account of the source you picked.
A source appears in the list only when all of these are true:
- It is an active SMB source on this RAG endpoint and its credentials are stored.
- Its Server and Port are the same as the ones you entered.
- Its LDAP URL and LDAP CA certificate are the same as the ones you entered.
- It enforces source permissions, if the new source enforces them.
The share and the folder may differ. If you change the server, port, LDAP URL or CA certificate after picking a source, the choice is cleared and the credential fields come back.
Each source keeps its own copy of the credentials. Changing the credentials of one source later does not change the other.
The connection is not tested when you copy credentials, so Connection succeeded does not appear. Select Create once the other fields are filled in.
When indexing succeeds
The connector reads the permissions of each new or changed file on the file server before downloading it. It indexes a file only when it can work out a plain list of the accounts and groups allowed to read it.
Indexing fails when:
- Its permission list is missing, which would let everyone read it.
- No entry in the list grants read access.
- An entry denies read access to anyone.
- The list contains an entry the connector does not support.
- The permissions cannot be read at all.
A failed file shows as failed in the sync, so the sync finishes with errors. New files are not indexed.
Previously indexed documents remain when permissions cannot be read because of a connection failure.
When enforcement is enabled, an unsupported permission list removes the indexed document and reports
unsupported_acl. A later successful sync restores it. Without enforcement, the existing document remains.
Successfully read permissions replace the stored permissions before downloading content. If downloading or indexing then fails, the existing content remains with the updated permissions.
File permissions in metadata
Each indexed file stores the Windows security identifiers (SIDs) allowed to read it on the file server in
metadata.source_group_ids. For example, a file readable by one account has
"source_group_ids": ["S-1-5-21-1-2-3-1201"] in its metadata.
These SIDs record the last successfully read file permissions. They limit access when source permission
enforcement is enabled. Each successful permission read replaces metadata.source_group_ids, so revoked SIDs
do not accumulate. The connector does
not write these SIDs to group_ids or change manual document groups. Manage those groups through the Groups API.
S3 and Azure Blob sources are not affected. Their documents keep the groups they have today.
Enforce file permissions
SMB source permission enforcement is off by default. When enabled, documents are returned only when the caller's own directory SID or one of their directory group SIDs matches the file's stored permissions. This applies to search, file lists, document and chunk reads, groups, and MCP tools, including calls by RAG administrators. S3 and Azure Blob sources are unaffected.
Configure enforcement through the manager's cmind.ragmanager.v1.RagDataSourceService API. On creation, use
config.smb.permission_settings. On update, use permission_settings and include it in update_mask.paths.
In the portal, enable Enforce source permissions when creating or editing an SMB source. Enter the LDAP URL
and CA certificate. Supply LDAP bind credentials with the source credentials when you create the source or rotate
its credentials. You can also copy the SMB and LDAP credentials from another active source. Each source keeps its own copy, so later credential changes are
independent.
When you edit an SMB source, Rotate credentials is selected and cannot be cleared if you change the LDAP URL or the LDAP CA certificate, even when enforcement is off. Enter the username, password and domain again, or copy them from another source. When enforcement is on, also enter the LDAP bind username and LDAP bind password.
Turning Enforce source permissions on keeps the stored credentials, including the LDAP bind account. If the source has no stored LDAP bind account, the save fails. Select Rotate credentials and enter the source credentials together with the LDAP bind username and LDAP bind password. Turning enforcement off on its own keeps the LDAP URL and the CA certificate on the source and does not ask for new credentials.
| Setting | Purpose |
|---|---|
enforce_source_permissions | Enable permission checks |
ldap_url | Directory address beginning with ldaps://, reachable from RAG |
ldap_ca_certificate | PEM CA certificate chain used to verify the directory server |
credentials.smb.ldap_username and ldap_password | Account permitted to read directory group membership |
Enforcement requires all these settings. Credential updates use the credentials update-mask path.
To disable enforcement and remove LDAP credentials together, include both paths in the same update.
The platform administrator must configure the identity provider to supply the signed-in user's directory SID
through the Keycloak onprem_sid claim. LDAP federation can supply base64 binary SIDs, while Entra brokering can
supply the S-1-... form. Users without a valid SID, personal API keys without a stored SID and service-account API
keys cannot read protected documents. Their access to other documents is unchanged. A personal API key uses the SID
its owner had when the key was created. Changing or removing the owner's SID does not update existing keys, so revoke
them when that happens.
- Directory membership. Group lookups are cached for up to five minutes. A user can be in any organizational unit in the configured directory. There is no base DN restriction.
- File-server groups. Built-in Windows groups are excluded because membership can differ between the directory and file server. Files accessible only through groups local to the file server remain inaccessible.
- Missing permissions. Documents without stored SIDs stay hidden while enforcement is on. Sync the source to read and store their permissions.
- Directory outages. Requests that need a failed directory lookup return
503. They never fall back to unfiltered documents. Explicit source or document filters avoid lookups for unrelated sources. - Configuration errors. Invalid source settings or missing credentials are reported as
latest_error.code=invalid_configuration. The source remains visible, but syncing stops and its documents stay hidden until the configuration is corrected. In the portal the source shows Configuration error in place of its sync status, with the message from the platform and a note that syncing is paused and the indexed documents are hidden from search and file reads. Sync stays unavailable. Edit the source to correct its settings or credentials; the state clears on its own once the platform accepts them. Only administrators of the RAG endpoint see this state. If documents from a source stop appearing in search, ask an administrator to check the source.
What the portal shows
The File access column shows Restricted for a source that enforces source permissions and Open for one that does not. It appears on the Data sources table of the RAG's Files tab and on the RAG Endpoints list. An endpoint shows Restricted when at least one of its data sources enforces source permissions. — means the source is being deleted, or the portal cannot read the endpoint's data sources.
Uploaded files notes that restricted sources show only files you can access. An empty list reads No files you can access. The View uploaded files button of a restricted source shows no count, because the count includes every file in the source.
Disconnecting a source and removing indexed files
The source deletion dialog selects Delete indexed files by default. Cleanup hides documents immediately and removes them in the background. Source removal proceeds once cleanup has been accepted. If cleanup cannot start, the source is not deleted.
If cleanup starts but the source itself cannot be removed, the portal reports File deletion has started, but source settings could not be deleted. Retry deleting the source. The files keep being deleted in the background and the source stays in the list. Select Delete on it again to finish removing it.
Clear the checkbox to stop synchronization and remove credentials while retaining indexed files. The dialog warns that retained files lose source permission enforcement and remain accessible to callers with RAG read access.
To remove indexed files, use the source file cleanup endpoint. Once cleanup is accepted, the source's documents disappear from search, file lists, document reads, and MCP tools while deletion runs in the background. Cleanup does not delete original files from the file server.
Permission changes and sync time
A change to a file's permissions counts as a change to the file. The next sync indexes it again, even when the content, the size and the modification time stay the same.
The connector detects permission changes from the file's change time, which the file server updates when the permissions change. Only new and changed files are opened to read their permissions. Failed files are checked again on every sync.
The first sync of an SMB source after the platform is updated indexes every file in the share again. Large shares take longer than usual on that one sync.