Connect a RAG data source

A data source keeps files from an S3-compatible bucket or Azure Blob container synchronized with a RAG endpoint.

Fields

These fields apply to both providers:

Field	Description
Name	Display name used in the portal
Prefix	Optional path inside the bucket or container, such as knowledge-base/. Leave blank to sync all files
Sync interval	How often the RAG re-reads the source. Choose Manual, 10 minutes, 1 hour, 1 day, 7 days, or a custom interval of at least 60 seconds

The default sync interval is 1 day. A manual source can be synchronized from its Sync action.

S3 fields

Field	Description	Example
Endpoint	S3 API endpoint, including https://	https://s3.eu-west-1.amazonaws.com
Region	Region containing the bucket	eu-west-1
Bucket	Bucket name without a path or URL	company-documents
Access key ID	Access key for an identity allowed to list and read the bucket	AKIA...
Secret access key	Secret for the access key ID	Shown only when the key is created

For another S3-compatible provider, use the endpoint, region, and access keys supplied by that provider.

Azure Blob fields

Field	Description	Example
Container	Container name without a path or URL	company-documents
Connection string	Azure Storage connection string containing the account credentials	DefaultEndpointsProtocol=https;AccountName=...

Before you start

You need:

• Admin access to the RAG endpoint in the portal
• Permission to create or manage the source storage and its credentials
• A source endpoint reachable from the RAG deployment

The source endpoint must resolve to a public address. Keep the bucket or container private; public object access is not required.

Set up Amazon S3

Already have a bucket? Start at step 2. Already have the policy? Start at step 3. Already have an access key with the required policy? Start at step 4.

1. Create a bucket

1. Open the Amazon S3 console and select Create bucket.
2. Select General purpose as the bucket type.
3. Enter a unique bucket name.
4. Select the AWS region closest to the RAG deployment. Record its region code, such as eu-west-1.
5. Keep Block all public access enabled.
6. Select Create bucket.
7. Open the bucket and upload the files to synchronize.

See Create a bucket for all AWS bucket options.

2. Create a read-only IAM policy

The connector uses these permissions:

• s3:ListBucket to find objects
• s3:GetObject to inspect and download objects

Create a policy for only the source bucket:

1. Open the IAM console.
2. Select Policies → Create policy.
3. Select JSON and enter the policy below.
4. Replace both instances of company-documents with the bucket name.
5. Select Next, name the policy, and select Create policy.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::company-documents"
    },
    {
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::company-documents/*"
    }
  ]
}

If the objects use an AWS KMS customer-managed key, also allow kms:Decrypt on that key.

See Create an IAM policy for the full AWS procedure.

3. Create the connector identity and key

1. In the IAM console, select Users → Create user.
2. Enter a name such as confidentialmind-rag-reader. Console access is not needed.
3. On the permissions step, attach the policy created above.
4. Create the user, open it, then select Security credentials.
5. Under Access keys, select Create access key.
6. Select Other → Next → Create access key.
7. Copy the Access key ID and Secret access key, or download the CSV. The secret cannot be retrieved later.

The AWS administrator performing these steps needs s3:CreateBucket, iam:CreatePolicy, iam:CreateUser, iam:AttachUserPolicy, and iam:CreateAccessKey, plus the read and list actions used by the AWS console. The credentials given to the portal need only the S3 permissions in the policy above.

See Create an IAM user and Create an access key for the corresponding AWS instructions.

4. Add the source in the portal

1. Open RAG endpoints and select the endpoint.
2. Open the Files tab and select Add source.
3. Select S3.
4. Enter a name.
5. Set Endpoint to the regional S3 endpoint. For example, use https://s3.eu-west-1.amazonaws.com for eu-west-1. See the S3 endpoint list.
6. Enter the region code and bucket name.
7. Enter a prefix to limit synchronization to that path, or leave it blank.
8. Enter the access key ID and secret access key.
9. Select a sync interval.
10. Wait for Connection succeeded, then select Create.

If the test connects but cannot read an object, check the bucket name, prefix, IAM policy, and KMS permissions.

Set up Azure Blob Storage

Already have a storage account? Start at step 3. Already have a container? Start at step 4. Already have the container name and connection string? Start at step 5.

1. Check your Azure permissions

To create a storage account, you need a role such as Contributor on the target resource group. To view its connection strings, your role must include Microsoft.Storage/storageAccounts/listkeys/action. Owner, Contributor, and Storage Account Key Operator Service Role include this action.

See Manage storage account access keys for the Azure permission details.

2. Create a storage account

1. Open the Azure portal and select Storage accounts → Create.
2. Select the subscription and resource group.
3. Enter a unique storage account name and select a region.
4. Select Standard performance unless the workload requires Premium storage.
5. Select the required redundancy option.
6. Under Advanced, keep Allow storage account key access enabled.
7. Under Networking, enable public network access. Choose one of these scopes:
   • Enable from all networks.
   • Enable from selected virtual networks and IP addresses, then allow the RAG deployment's outbound IP.
8. Select Review + create → Create.
9. When deployment finishes, select Go to resource.

Private endpoints are not currently supported by this connector. Public network access does not make blobs public; access still requires the connection string.

See Create a storage account and Azure Storage network security for all account and network options.

3. Create a container

1. In the storage account, select Data storage → Containers.
2. Select + Container.
3. Enter a lowercase container name.
4. Set Anonymous access level to Private (no anonymous access).
5. Select Create.
6. Open the container and upload the files to synchronize.

See Manage blob containers for all container options.

4. Copy the connection string

1. In the storage account, select Security + networking → Access keys.
2. Select Show keys and copy a Connection string from either key.

An account-key connection string grants full access to the storage account data, although the connector only lists and downloads blobs. Use a dedicated storage account if the credential should not cover unrelated data. Store the connection string as a secret and rotate it if it is exposed.

5. Add the source in the portal

1. Open RAG endpoints and select the endpoint.
2. Open the Files tab and select Add source.
3. Select Azure Blob Storage.
4. Enter a name.
5. Enter the container name.
6. Enter a prefix to limit synchronization to that path, or leave it blank.
7. Paste the connection string.
8. Select a sync interval.
9. Wait for Connection succeeded, then select Create.

If the test connects but cannot read a blob, check the container, prefix, network rules, and connection string.