Skip to main content
Version: 3.6.1-rc.2

Connect a RAG data source

A data source keeps files from an S3-compatible bucket, Azure Blob container, or SMB file share synchronized with a RAG endpoint.

Fields​

These fields apply to all providers:

FieldDescription
NameDisplay name used in the portal
PrefixOptional path inside the bucket, container, or share, such as knowledge-base/. For an SMB share, enter a folder path with no slash at the start or the end, such as docs/2026. Leave blank to sync all files
Sync intervalHow often the RAG re-reads the source. Choose Manual, 10 minutes, 1 hour, 1 day, 7 days, or a custom interval of at least 10 minutes

The default sync interval is 1 day. A manual source can be synchronized from its Sync action.

S3 fields​

FieldDescriptionExample
EndpointS3 API endpoint, including https://https://s3.eu-west-1.amazonaws.com
RegionRegion containing the bucketeu-west-1
BucketBucket name without a path or URLcompany-documents
Access key IDAccess key for an identity allowed to list and read the bucketAKIA...
Secret access keySecret for the access key IDShown only when the key is created

For another S3-compatible provider, use the endpoint, region, and access keys supplied by that provider.

Azure Blob fields​

FieldDescriptionExample
ContainerContainer name without a path or URLcompany-documents
Connection stringAzure Storage connection string containing the account credentialsDefaultEndpointsProtocol=https;AccountName=...

SMB file share fields​

FieldDescriptionExample
ServerHost name or IP address of the SMB server, without smb://, a share, or a portfileserver.example.com
ShareShare name exported by the SMB server. For \\fileserver\documents, enter documentsdocuments
PortPort the SMB server listens on. Standard SMB uses 445445
UsernameAccount allowed to list the share, read its files, and read their permissionsrag-reader
PasswordPassword for that account
DomainOptional Windows or Active Directory domain for the account. Leave blank for an account local to the SMB serverCORP
Enforce source permissionsOptional. Limits search and file reads to the documents the caller's own account or groups can read on the file server
LDAP URLShown when enforcement is on. Address of the directory that resolves caller group membership. It must start with ldaps://ldaps://ad.example.com:636
LDAP CA certificateShown when enforcement is on. PEM certificate chain that verifies the LDAP server certificate-----BEGIN CERTIFICATE-----
LDAP bind usernameShown when enforcement is on. Account allowed to read group membership in the directory[email protected]
LDAP bind passwordPassword for the LDAP bind account

If the port is not a whole number from 1 to 65535, the portal shows Port must be an integer between 1 and 65535. and does not save the source.

Select Require SMB encryption to encrypt the connection to the file server. Leave it clear if the file server does not support SMB encryption. The setting is stored with the source, and the source details show Required or Not required.

Select Enforce source permissions to limit each caller to the files their own account can read on the file server. The source details then show Enforced under Source permissions, together with the LDAP URL. See Enforce file permissions.

Which files are indexed​

Files are read the same way as file uploads. See Which files can be indexed for the formats.

Audio and video files are never downloaded. They are listed as failed with an error that names the format, such as Data source file format is not supported: .mp4. A file of one of these formats that was indexed before is listed as failed after the next sync.

An image file (.jpg, .jpeg, .png, .gif, .bmp, .tiff, .webp) is indexed only when the RAG has an OCR model. Without one, the image is listed as failed with an error such as Data source image requires a configured vision model: .png. Select an OCR model on the RAG's page in the portal, then use the source's Sync action to index the images.

Any other file the RAG cannot read is also listed as failed. The other files are still indexed.

After the platform is updated, files that failed before are tried again on the next sync. To index them right away, use the source's Sync action.

Before you start​

You need:

  • Admin access to the RAG endpoint in the portal
  • Permission to create or manage the source storage and its credentials
  • A source endpoint reachable from the RAG deployment

For S3 and Azure Blob, the source endpoint must resolve to a public address. Keep the bucket or container private; public object access is not required. For an SMB file share, you also need an account on the file server that is allowed to list the share, read its files, and read the permissions of those files, and the port the file server listens on. The file server only needs to be reachable from the RAG deployment, so it can sit on a private network.

Set up Amazon S3​

Already have a bucket? Start at step 2. Already have the policy? Start at step 3. Already have an access key with the required policy? Start at step 4.

1. Create a bucket​

  1. Open the Amazon S3 console and select Create bucket.
  2. Select General purpose as the bucket type.
  3. Enter a unique bucket name.
  4. Select the AWS region closest to the RAG deployment. Record its region code, such as eu-west-1.
  5. Keep Block all public access enabled.
  6. Select Create bucket.
  7. Open the bucket and upload the files to synchronize.

See Create a bucket for all AWS bucket options.

2. Create a read-only IAM policy​

The connector uses these permissions:

  • s3:ListBucket to find objects
  • s3:GetObject to inspect and download objects

Create a policy for only the source bucket:

  1. Open the IAM console.
  2. Select Policies → Create policy.
  3. Select JSON and enter the policy below.
  4. Replace both instances of company-documents with the bucket name.
  5. Select Next, name the policy, and select Create policy.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::company-documents"
},
{
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::company-documents/*"
}
]
}

If the objects use an AWS KMS customer-managed key, also allow kms:Decrypt on that key.

See Create an IAM policy for the full AWS procedure.

3. Create the connector identity and key​

  1. In the IAM console, select Users → Create user.
  2. Enter a name such as confidentialmind-rag-reader. Console access is not needed.
  3. On the permissions step, attach the policy created above.
  4. Create the user, open it, then select Security credentials.
  5. Under Access keys, select Create access key.
  6. Select Other → Next → Create access key.
  7. Copy the Access key ID and Secret access key, or download the CSV. The secret cannot be retrieved later.

The AWS administrator performing these steps needs s3:CreateBucket, iam:CreatePolicy, iam:CreateUser, iam:AttachUserPolicy, and iam:CreateAccessKey, plus the read and list actions used by the AWS console. The credentials given to the portal need only the S3 permissions in the policy above.

See Create an IAM user and Create an access key for the corresponding AWS instructions.

4. Add the source in the portal​

  1. Open RAG and select the endpoint.
  2. Open the Files tab and select Add source.
  3. Select S3.
  4. Enter a name.
  5. Set Endpoint to the regional S3 endpoint. For example, use https://s3.eu-west-1.amazonaws.com for eu-west-1. See the S3 endpoint list.
  6. Enter the region code and bucket name.
  7. Enter a prefix to limit synchronization to that path, or leave it blank.
  8. Enter the access key ID and secret access key.
  9. Select a sync interval.
  10. Wait for Connection succeeded, then select Create.

If the test connects but cannot read an object, check the bucket name, prefix, IAM policy, and KMS permissions.

Set up Azure Blob Storage​

Already have a storage account? Start at step 3. Already have a container? Start at step 4. Already have the container name and connection string? Start at step 5.

1. Check your Azure permissions​

To create a storage account, you need a role such as Contributor on the target resource group. To view its connection strings, your role must include Microsoft.Storage/storageAccounts/listkeys/action. Owner, Contributor, and Storage Account Key Operator Service Role include this action.

See Manage storage account access keys for the Azure permission details.

2. Create a storage account​

  1. Open the Azure portal and select Storage accounts → Create.
  2. Select the subscription and resource group.
  3. Enter a unique storage account name and select a region.
  4. Select Standard performance unless the workload requires Premium storage.
  5. Select the required redundancy option.
  6. Under Advanced, keep Allow storage account key access enabled.
  7. Under Networking, enable public network access. Choose one of these scopes:
    • Enable from all networks.
    • Enable from selected virtual networks and IP addresses, then allow the RAG deployment's outbound IP.
  8. Select Review + create → Create.
  9. When deployment finishes, select Go to resource.

Private endpoints are not currently supported by this connector. Public network access does not make blobs public; access still requires the connection string.

See Create a storage account and Azure Storage network security for all account and network options.

3. Create a container​

  1. In the storage account, select Data storage → Containers.
  2. Select + Container.
  3. Enter a lowercase container name.
  4. Set Anonymous access level to Private (no anonymous access).
  5. Select Create.
  6. Open the container and upload the files to synchronize.

See Manage blob containers for all container options.

4. Copy the connection string​

  1. In the storage account, select Security + networking → Access keys.
  2. Select Show keys and copy a Connection string from either key.

An account-key connection string grants full access to the storage account data, although the connector only lists and downloads blobs. Use a dedicated storage account if the credential should not cover unrelated data. Store the connection string as a secret and rotate it if it is exposed.

5. Add the source in the portal​

  1. Open RAG and select the endpoint.
  2. Open the Files tab and select Add source.
  3. Select Azure Blob Storage.
  4. Enter a name.
  5. Enter the container name.
  6. Enter a prefix to limit synchronization to that path, or leave it blank.
  7. Paste the connection string.
  8. Select a sync interval.
  9. Wait for Connection succeeded, then select Create.

If the test connects but cannot read a blob, check the container, prefix, network rules, and connection string.

Set up an SMB file share​

The RAG deployment connects to the Server directly, so use an address it can reach:

File server locationServer example
Kubernetes service in the same namespacesamba
Kubernetes service in another namespacesamba.storage.svc.cluster.local
Docker Desktop hosthost.docker.internal
Private network or VPNfileserver.corp.local or 10.20.30.40
External serverfileserver.example.com

Note: In Kubernetes, localhost refers to the RAG pod itself. Use another address unless the SMB server runs in that pod.

Before creating the source:

  • Confirm that the cluster can resolve the server address and reach its SMB port.
  • Enter only the host name or IP address in Server. Do not include smb://, the share, or the port.
  • Enter the share and port in their own fields.
  1. Open RAG and select the endpoint.
  2. Open the Files tab and select Add source.
  3. Select SMB file share.
  4. Enter a name.
  5. Enter the server and the name of the share.
  6. Keep the port at 445 unless the file server listens on another port.
  7. Select Require SMB encryption if the connection to the file server must be encrypted.
  8. Select Enforce source permissions if each caller must only reach the files their own account can read on the file server. Enter the LDAP URL and paste the LDAP CA certificate.
  9. Enter a prefix to limit synchronization to a folder inside the share, such as docs/2026, with no slash at the start or the end. Leave it blank to synchronize the whole share.
  10. Enter the username and password of the account that can read the share. Enter the domain if the account belongs to a Windows domain, or leave it blank.
  11. If you selected Enforce source permissions, enter the LDAP bind username and LDAP bind password.
  12. Select a sync interval.
  13. Wait for Connection succeeded, then select Create.

If the test connects but cannot read a file, check the share name, the prefix, and the read permissions of the account.

Use the account of another source​

When the RAG endpoint already has an SMB source that uses the same account, the form shows a Credentials list. Select Copy from that source, and the username, password, domain and LDAP bind fields disappear. The new source then uses the stored account of the source you picked.

A source appears in the list only when all of these are true:

  • It is an active SMB source on this RAG endpoint and its credentials are stored.
  • Its Server and Port are the same as the ones you entered.
  • Its LDAP URL and LDAP CA certificate are the same as the ones you entered.
  • It enforces source permissions, if the new source enforces them.

The share and the folder may differ. If you change the server, port, LDAP URL or CA certificate after picking a source, the choice is cleared and the credential fields come back.

Each source keeps its own copy of the credentials. Changing the credentials of one source later does not change the other.

The connection is not tested when you copy credentials, so Connection succeeded does not appear. Select Create once the other fields are filled in.

When indexing succeeds​

The connector reads the permissions of each new or changed file on the file server before downloading it. It indexes a file only when it can work out a plain list of the accounts and groups allowed to read it.

Indexing fails when:

  • Its permission list is missing, which would let everyone read it.
  • No entry in the list grants read access.
  • An entry denies read access to anyone.
  • The list contains an entry the connector does not support.
  • The permissions cannot be read at all.

A failed file shows as failed in the sync, so the sync finishes with errors. New files are not indexed. Previously indexed documents remain when permissions cannot be read because of a connection failure. When enforcement is enabled, an unsupported permission list removes the indexed document and reports unsupported_acl. A later successful sync restores it. Without enforcement, the existing document remains.

Successfully read permissions replace the stored permissions before downloading content. If downloading or indexing then fails, the existing content remains with the updated permissions.

File permissions in metadata​

Each indexed file stores the Windows security identifiers (SIDs) allowed to read it on the file server in metadata.source_group_ids. For example, a file readable by one account has "source_group_ids": ["S-1-5-21-1-2-3-1201"] in its metadata.

These SIDs record the last successfully read file permissions. They limit access when source permission enforcement is enabled. Each successful permission read replaces metadata.source_group_ids, so revoked SIDs do not accumulate. The connector does not write these SIDs to group_ids or change manual document groups. Manage those groups through the Groups API.

S3 and Azure Blob sources are not affected. Their documents keep the groups they have today.

Enforce file permissions​

SMB source permission enforcement is off by default. When enabled, documents are returned only when the caller's own directory SID or one of their directory group SIDs matches the file's stored permissions. This applies to search, file lists, document and chunk reads, groups, and MCP tools, including calls by RAG administrators. S3 and Azure Blob sources are unaffected.

Configure enforcement through the manager's cmind.ragmanager.v1.RagDataSourceService API. On creation, use config.smb.permission_settings. On update, use permission_settings and include it in update_mask.paths. In the portal, enable Enforce source permissions when creating or editing an SMB source. Enter the LDAP URL and CA certificate. Supply LDAP bind credentials with the source credentials when you create the source or rotate its credentials. You can also copy the SMB and LDAP credentials from another active source. Each source keeps its own copy, so later credential changes are independent.

When you edit an SMB source, Rotate credentials is selected and cannot be cleared if you change the LDAP URL or the LDAP CA certificate, even when enforcement is off. Enter the username, password and domain again, or copy them from another source. When enforcement is on, also enter the LDAP bind username and LDAP bind password.

Turning Enforce source permissions on keeps the stored credentials, including the LDAP bind account. If the source has no stored LDAP bind account, the save fails. Select Rotate credentials and enter the source credentials together with the LDAP bind username and LDAP bind password. Turning enforcement off on its own keeps the LDAP URL and the CA certificate on the source and does not ask for new credentials.

SettingPurpose
enforce_source_permissionsEnable permission checks
ldap_urlDirectory address beginning with ldaps://, reachable from RAG
ldap_ca_certificatePEM CA certificate chain used to verify the directory server
credentials.smb.ldap_username and ldap_passwordAccount permitted to read directory group membership

Enforcement requires all these settings. Credential updates use the credentials update-mask path. To disable enforcement and remove LDAP credentials together, include both paths in the same update.

The platform administrator must configure the identity provider to supply the signed-in user's directory SID through the Keycloak onprem_sid claim. LDAP federation can supply base64 binary SIDs, while Entra brokering can supply the S-1-... form. Users without a valid SID, personal API keys without a stored SID and service-account API keys cannot read protected documents. Their access to other documents is unchanged. A personal API key uses the SID its owner had when the key was created. Changing or removing the owner's SID does not update existing keys, so revoke them when that happens.

  • Directory membership. Group lookups are cached for up to five minutes. A user can be in any organizational unit in the configured directory. There is no base DN restriction.
  • File-server groups. Built-in Windows groups are excluded because membership can differ between the directory and file server. Files accessible only through groups local to the file server remain inaccessible.
  • Missing permissions. Documents without stored SIDs stay hidden while enforcement is on. Sync the source to read and store their permissions.
  • Directory outages. Requests that need a failed directory lookup return 503. They never fall back to unfiltered documents. Explicit source or document filters avoid lookups for unrelated sources.
  • Configuration errors. Invalid source settings or missing credentials are reported as latest_error.code=invalid_configuration. The source remains visible, but syncing stops and its documents stay hidden until the configuration is corrected. In the portal the source shows Configuration error in place of its sync status, with the message from the platform and a note that syncing is paused and the indexed documents are hidden from search and file reads. Sync stays unavailable. Edit the source to correct its settings or credentials; the state clears on its own once the platform accepts them. Only administrators of the RAG endpoint see this state. If documents from a source stop appearing in search, ask an administrator to check the source.

What the portal shows​

The File access column shows Restricted for a source that enforces source permissions and Open for one that does not. It appears on the Data sources table of the RAG's Files tab and on the RAG Endpoints list. An endpoint shows Restricted when at least one of its data sources enforces source permissions. — means the source is being deleted, or the portal cannot read the endpoint's data sources.

Uploaded files notes that restricted sources show only files you can access. An empty list reads No files you can access. The View uploaded files button of a restricted source shows no count, because the count includes every file in the source.

Disconnecting a source and removing indexed files​

The source deletion dialog selects Delete indexed files by default. Cleanup hides documents immediately and removes them in the background. Source removal proceeds once cleanup has been accepted. If cleanup cannot start, the source is not deleted.

If cleanup starts but the source itself cannot be removed, the portal reports File deletion has started, but source settings could not be deleted. Retry deleting the source. The files keep being deleted in the background and the source stays in the list. Select Delete on it again to finish removing it.

Clear the checkbox to stop synchronization and remove credentials while retaining indexed files. The dialog warns that retained files lose source permission enforcement and remain accessible to callers with RAG read access.

To remove indexed files, use the source file cleanup endpoint. Once cleanup is accepted, the source's documents disappear from search, file lists, document reads, and MCP tools while deletion runs in the background. Cleanup does not delete original files from the file server.

Permission changes and sync time​

A change to a file's permissions counts as a change to the file. The next sync indexes it again, even when the content, the size and the modification time stay the same.

The connector detects permission changes from the file's change time, which the file server updates when the permissions change. Only new and changed files are opened to read their permissions. Failed files are checked again on every sync.

The first sync of an SMB source after the platform is updated indexes every file in the share again. Large shares take longer than usual on that one sync.