Skip to main content

Log Export

Send platform logs to Splunk, so audit and operational data land in the same place as the rest of your estate.

Streams

Two streams are available and they never overlap, so you can export one without the other.

StreamContents
auditAuthenticated actions and their outcome: who acted, what they acted on, whether it was allowed or denied, and the tenant it happened in.
platformContainer logs from platform workloads, plus per-request logs from the service mesh. Operational detail, at far higher volume than audit. Logs from the monitoring, kube-system and argocd namespaces are not collected, so the observability stack's own logs and the cluster control plane are outside this stream.

Audit events reach Splunk only through the audit stream. Enabling platform does not duplicate them.

The HEC token

There is no values field for the token: values end up in the platform's own configuration objects, so a token there would sit in plain text where anyone with cluster read access could find it. Instead you place the token in the secret store, and the platform fetches it into the monitoring namespace on your behalf.

Where you put it depends on which store the platform is configured with. In both cases the name and key must match what the platform looks for, or the fetch finds nothing.

Using your own secret store

With secrets.useSecretsProvider: true — OpenBao, Vault or another supported provider — add the token to that store, at the path the platform reads:

Keycm-platform-log-export
Propertysplunk-hec-token

Both are configurable under splunkHec.token if the token already lives elsewhere in your store, which saves moving it.

Using the in-platform store

With secrets.useSecretsProvider: false the store is the cm-platform-secrets namespace in the cluster itself. Create the Secret there, named for the key above with the property as its data key:

kubectl -n cm-platform-secrets create secret generic cm-platform-log-export \
--from-literal=splunk-hec-token=<HEC-token>

Checking it arrived

The platform reconciles the token into a splunk-hec-token Secret in monitoring once a Splunk stream is enabled. If the collector will not start, this is the first thing to look at — a name that doesn't match leaves it unfulfilled:

kubectl -n monitoring get externalsecret splunk-hec-token

SecretSynced means the token was found.

When you rotate the token, update it in the store and then restart the collector. The platform picks the new value up within a few minutes, but the collector reads it at startup, so a running one keeps sending the old token until it restarts:

kubectl -n monitoring rollout restart deploy otel-collector-gateway-collector

TLS

Splunk Cloud endpoints present a publicly-trusted certificate and need nothing further. A self-hosted Splunk usually presents one from your own CA, or the self-signed certificate it ships with; either way, add that CA as a ConfigMap in the monitoring namespace:

kubectl -n monitoring create configmap splunk-ca --from-file=ca.crt=/path/to/ca.crt

If you don't have the certificate to hand, read it off the endpoint:

openssl s_client -connect splunk.example.com:8088 -showcerts </dev/null 2>/dev/null \
| openssl x509 > ca.crt

Configuration

monitoring:
logExport:
splunkHec:
enabled: true
endpoint: https://splunk.example.com:8088/services/collector
# Only needed for a private CA.
tls:
ca:
configMap:
name: splunk-ca
streams:
audit:
enabled: true
index: cm_audit
platform:
enabled: false

An index left empty uses whatever default index the HEC token is configured with. Each stream also takes a sourcetype, defaulting to cm:audit and cm:logs.

Retaining audit events in the platform

Audit events can additionally be kept in the platform's own store:

monitoring:
logExport:
internal:
audit:
clickhouse: true

This is independent of Splunk — enable both to keep a local copy, or leave it off and treat Splunk as the system of record.

Delivery

Events are buffered in memory and retried when Splunk is unavailable, which covers restarts and brief network faults. Events still buffered when a collector pod is replaced are lost, so if you need a stricter guarantee, keep the local copy as well.

Other destinations

Destinations without a section of their own can be configured directly as OpenTelemetry Collector exporters, per stream:

monitoring:
logExport:
exporters:
audit:
otlphttp/audit:
endpoint: https://collector.example.com:4318

Only the exporters built into the platform's collector are available: otlp and otlphttp for anything that speaks OpenTelemetry, including a collector of your own that forwards onward, and debug for checking what a stream contains. Naming anything else stops the whole collector from starting, so a destination that needs its own protocol needs platform support — contact us. The same applies to options that need a credential, a private CA, or a delivery guarantee, since those reach parts of the collector this passthrough cannot.

An exporter ID has to be unique across both streams. The same ID under audit and platform collides in the generated configuration.