It is required to configure rabbitmq persistence volume in the following cases:
1. Memory settings for rabbitmq significantly changed (param rabbitmq.rabbitmqMemoryHighWatermark > 768MiB)
2. You need for more persistance data in rabbitmq. Please note, restart of rabbitmq can take a lot of time if persistance data size more then 4Gi
Recommended persistance volume size is 3Gi, it means rabbitmq will store 2Gi data (rabbitmq.persistentVolume.size-rabbitmq.rabbitmqMemoryHighWatermark) and data from memory (rabbitmq.rabbitmqMemoryHighWatermark)
Steps to reconfigure rabbitmq memory:
1. Disable fluentbit or fluentd by deleting of its daemonset in kublr namespace
2. Wait until rabbitmq queue emptied (use Grafana or check the receipt of log records has stopped for cluster)
3. Scale kublr-logging-rabbitmq StatefulSet to 0
4. Delete Persistent Volume Claims data-kublr-logging-rabbitmq-0
5. Edit cluster specification:
rabbitmq.persistentVolume.size = 3Gi + rabbitmq.rabbitmqMemoryHighWatermark (or more)
rabbitmq.rabbitmqDiskFreeLimit = rabbitmq.rabbitmqMemoryHighWatermark + 512MiB
rabbitmq.resources.limits.memory = rabbitmq.rabbitmqMemoryHighWatermark * 2
rabbitmq.resources.requests.memory = rabbitmq.resources.limits.memory
Logging feature values example for watermark 1.5Gb:
rabbitmq: persistentVolume: size: 5Gi resources: requests: memory: 3Gi limits: memory: 3Gi rabbitmqMemoryHighWatermark: 1536MiB rabbitmqDiskFreeLimit: 2048MiB
6. Apply cluster specification. Kublr will recreate rabbitmq StatefulSet and it's PVC, also fluentd/fluentbit.