Persistence in production logs is expected, but it's also a feature. A feature we've postponed long enough.
Okctl 0.0.94 configures Loki to use S3 as storage for logs and DynamoDB for index tables.
To upgrade, download the latest okctl and run:
okctl upgrade
What does it mean for your existing cluster?
Not much. Loki allows switching where it sends logs on a specific date. 24 hours after running the Loki upgrade, Loki will send logs to S3, instead of Loki's filesystem.
The upgrade is designed to work perfectly. However, trust is good - control is better. You might want to back up your logs before initiating the upgrade. This can be done by running the following commands:
# Prepare a directory for the backup
mkdir BACKUP
# Download relevant data from the Loki instance
kubectl -n monitoring cp loki-0:/data/loki BACKUP/loki
To restore logs in the event of an error, run:
# Upload the backup to the relevant folder in the Loki instance
kubectl -n monitoring cp BACKUP/loki loki-0:/data/loki
Happy logging!