Connect a RAG data source

A data source keeps files from an S3-compatible bucket, Azure Blob container, or SMB file share synchronized with a RAG endpoint.

Fields

These fields apply to all providers:

Field	Description
Name	Display name used in the portal
Prefix	Optional path inside the bucket, container, or share, such as knowledge-base/. For an SMB share, enter a folder path with no slash at the start or the end, such as docs/2026. Leave blank to sync all files
Sync interval	How often the RAG re-reads the source. Choose Manual, 10 minutes, 1 hour, 1 day, 7 days, or a custom interval of at least 10 minutes

The default sync interval is 1 day. A manual source can be synchronized from its Sync action.

S3 fields

Field	Description	Example
Endpoint	S3 API endpoint, including https://	https://s3.eu-west-1.amazonaws.com
Region	Region containing the bucket	eu-west-1
Bucket	Bucket name without a path or URL	company-documents
Access key ID	Access key for an identity allowed to list and read the bucket	AKIA...
Secret access key	Secret for the access key ID	Shown only when the key is created

For another S3-compatible provider, use the endpoint, region, and access keys supplied by that provider.

Azure Blob fields

Field	Description	Example
Container	Container name without a path or URL	company-documents
Connection string	Azure Storage connection string containing the account credentials	DefaultEndpointsProtocol=https;AccountName=...

SMB file share fields

Field	Description	Example
Server	Host name or IP address of the SMB server, without smb://, a share, or a port	fileserver.example.com
Share	Share name exported by the SMB server. For \\fileserver\documents, enter documents	documents
Port	Port the SMB server listens on. Standard SMB uses 445	445
Username	Account allowed to list the share, read its files, and read their permissions	rag-reader
Password	Password for that account
Domain	Optional Windows or Active Directory domain for the account. Leave blank for an account local to the SMB server	CORP

If the port is not a whole number from 1 to 65535, the portal shows Port must be an integer between 1 and 65535. and does not save the source.

Select Require SMB encryption to encrypt the connection to the file server. Leave it clear if the file server does not support SMB encryption. The setting is stored with the source, and the source details show Required or Not required.

Which files are indexed

Files are read the same way as file uploads. See Which files can be indexed for the formats.

A file in a format the RAG cannot read, such as a video file, is listed as failed. The other files are still indexed.

After the platform is updated, files that failed before are tried again on the next sync. To index them right away, use the source's Sync action.

Before you start

You need:

• Admin access to the RAG endpoint in the portal
• Permission to create or manage the source storage and its credentials
• A source endpoint reachable from the RAG deployment

For S3 and Azure Blob, the source endpoint must resolve to a public address. Keep the bucket or container private; public object access is not required. For an SMB file share, you also need an account on the file server that is allowed to list the share, read its files, and read the permissions of those files, and the port the file server listens on. The file server only needs to be reachable from the RAG deployment, so it can sit on a private network.

Set up Amazon S3

Already have a bucket? Start at step 2. Already have the policy? Start at step 3. Already have an access key with the required policy? Start at step 4.

1. Create a bucket

1. Open the Amazon S3 console and select Create bucket.
2. Select General purpose as the bucket type.
3. Enter a unique bucket name.
4. Select the AWS region closest to the RAG deployment. Record its region code, such as eu-west-1.
5. Keep Block all public access enabled.
6. Select Create bucket.
7. Open the bucket and upload the files to synchronize.

See Create a bucket for all AWS bucket options.

2. Create a read-only IAM policy

The connector uses these permissions:

• s3:ListBucket to find objects
• s3:GetObject to inspect and download objects

Create a policy for only the source bucket:

1. Open the IAM console.
2. Select Policies → Create policy.
3. Select JSON and enter the policy below.
4. Replace both instances of company-documents with the bucket name.
5. Select Next, name the policy, and select Create policy.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::company-documents"
    },
    {
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::company-documents/*"
    }
  ]
}

If the objects use an AWS KMS customer-managed key, also allow kms:Decrypt on that key.

See Create an IAM policy for the full AWS procedure.

3. Create the connector identity and key

1. In the IAM console, select Users → Create user.
2. Enter a name such as confidentialmind-rag-reader. Console access is not needed.
3. On the permissions step, attach the policy created above.
4. Create the user, open it, then select Security credentials.
5. Under Access keys, select Create access key.
6. Select Other → Next → Create access key.
7. Copy the Access key ID and Secret access key, or download the CSV. The secret cannot be retrieved later.

The AWS administrator performing these steps needs s3:CreateBucket, iam:CreatePolicy, iam:CreateUser, iam:AttachUserPolicy, and iam:CreateAccessKey, plus the read and list actions used by the AWS console. The credentials given to the portal need only the S3 permissions in the policy above.

See Create an IAM user and Create an access key for the corresponding AWS instructions.

4. Add the source in the portal

1. Open RAG and select the endpoint.
2. Open the Files tab and select Add source.
3. Select S3.
4. Enter a name.
5. Set Endpoint to the regional S3 endpoint. For example, use https://s3.eu-west-1.amazonaws.com for eu-west-1. See the S3 endpoint list.
6. Enter the region code and bucket name.
7. Enter a prefix to limit synchronization to that path, or leave it blank.
8. Enter the access key ID and secret access key.
9. Select a sync interval.
10. Wait for Connection succeeded, then select Create.

If the test connects but cannot read an object, check the bucket name, prefix, IAM policy, and KMS permissions.

Set up Azure Blob Storage

Already have a storage account? Start at step 3. Already have a container? Start at step 4. Already have the container name and connection string? Start at step 5.

1. Check your Azure permissions

To create a storage account, you need a role such as Contributor on the target resource group. To view its connection strings, your role must include Microsoft.Storage/storageAccounts/listkeys/action. Owner, Contributor, and Storage Account Key Operator Service Role include this action.

See Manage storage account access keys for the Azure permission details.

2. Create a storage account

1. Open the Azure portal and select Storage accounts → Create.
2. Select the subscription and resource group.
3. Enter a unique storage account name and select a region.
4. Select Standard performance unless the workload requires Premium storage.
5. Select the required redundancy option.
6. Under Advanced, keep Allow storage account key access enabled.
7. Under Networking, enable public network access. Choose one of these scopes:
   • Enable from all networks.
   • Enable from selected virtual networks and IP addresses, then allow the RAG deployment's outbound IP.
8. Select Review + create → Create.
9. When deployment finishes, select Go to resource.

Private endpoints are not currently supported by this connector. Public network access does not make blobs public; access still requires the connection string.

See Create a storage account and Azure Storage network security for all account and network options.

3. Create a container

1. In the storage account, select Data storage → Containers.
2. Select + Container.
3. Enter a lowercase container name.
4. Set Anonymous access level to Private (no anonymous access).
5. Select Create.
6. Open the container and upload the files to synchronize.

See Manage blob containers for all container options.

4. Copy the connection string

1. In the storage account, select Security + networking → Access keys.
2. Select Show keys and copy a Connection string from either key.

An account-key connection string grants full access to the storage account data, although the connector only lists and downloads blobs. Use a dedicated storage account if the credential should not cover unrelated data. Store the connection string as a secret and rotate it if it is exposed.

5. Add the source in the portal

1. Open RAG and select the endpoint.
2. Open the Files tab and select Add source.
3. Select Azure Blob Storage.
4. Enter a name.
5. Enter the container name.
6. Enter a prefix to limit synchronization to that path, or leave it blank.
7. Paste the connection string.
8. Select a sync interval.
9. Wait for Connection succeeded, then select Create.

If the test connects but cannot read a blob, check the container, prefix, network rules, and connection string.

Set up an SMB file share

The RAG deployment connects to the Server directly, so use an address it can reach:

File server location	Server example
Kubernetes service in the same namespace	samba
Kubernetes service in another namespace	samba.storage.svc.cluster.local
Docker Desktop host	host.docker.internal
Private network or VPN	fileserver.corp.local or 10.20.30.40
External server	fileserver.example.com

Note: In Kubernetes, localhost refers to the RAG pod itself. Use another address unless the SMB server runs in that pod.

Before creating the source:

• Confirm that the cluster can resolve the server address and reach its SMB port.
• Enter only the host name or IP address in Server. Do not include smb://, the share, or the port.
• Enter the share and port in their own fields.

1. Open RAG and select the endpoint.
2. Open the Files tab and select Add source.
3. Select SMB file share.
4. Enter a name.
5. Enter the server and the name of the share.
6. Keep the port at 445 unless the file server listens on another port.
7. Select Require SMB encryption if the connection to the file server must be encrypted.
8. Enter a prefix to limit synchronization to a folder inside the share, such as docs/2026, with no slash at the start or the end. Leave it blank to synchronize the whole share.
9. Enter the username and password of the account that can read the share. Enter the domain if the account belongs to a Windows domain, or leave it blank.
10. Select a sync interval.
11. Wait for Connection succeeded, then select Create.

If the test connects but cannot read a file, check the share name, the prefix, and the read permissions of the account.

When indexing succeeds

The connector reads the permissions of each new or changed file on the file server before downloading it. It indexes a file only when it can work out a plain list of the accounts and groups allowed to read it.

Indexing fails when:

• Its permission list is missing, which would let everyone read it.
• No entry in the list grants read access.
• An entry denies read access to anyone.
• The list contains an entry the connector does not support.
• The permissions cannot be read at all.

A failed file shows as failed in the sync, so the sync finishes with errors. New files are not indexed. For previously indexed files, the existing document, manual groups, and last successfully read permissions remain until a later sync succeeds. Permission-read failures do not delete documents.

File permissions in metadata

Each indexed file stores the Windows security identifiers (SIDs) allowed to read it on the file server in metadata.source_group_ids. For example, a file readable by one account has "source_group_ids": ["S-1-5-21-1-2-3-1201"] in its metadata.

These SIDs record the file permissions from the last successful ingest. The RAG does not match callers to their SIDs, so this metadata does not enforce access control or limit search results.

Each successful ingest replaces metadata.source_group_ids, so revoked SIDs do not accumulate. The connector does not write these SIDs to group_ids or change manual document groups. Manage those groups through the Groups API.

S3 and Azure Blob sources are not affected. Their documents keep the groups they have today.

Permission changes and sync time

A change to a file's permissions counts as a change to the file. The next sync indexes it again, even when the content, the size and the modification time stay the same.

The connector detects permission changes from the file's change time, which the file server updates when the permissions change. Only new and changed files are opened to read their permissions. Failed files are checked again on every sync.

The first sync of an SMB source after the platform is updated indexes every file in the share again. Large shares take longer than usual on that one sync.