Persistence in production logs is expected, but it's also a feature. A feature we've postponed long enough.

Okctl 0.0.94 configures Loki to use S3 as storage for logs and DynamoDB for index tables.

To upgrade, download the latest okctl and run‌‌:

okctl upgrade

What does it mean for your existing cluster?

Not much. Loki allows switching where it sends logs on a specific date. 24 hours after running the Loki upgrade, Loki will send logs to S3, instead of Loki's filesystem.

The upgrade is designed to work perfectly. However, trust is good - control is better. You might want to back up your logs before initiating the upgrade. This can be done by running the following commands:

# Prepare a directory for the backup
mkdir BACKUP

# Download relevant data from the Loki instance
kubectl -n monitoring cp loki-0:/data/loki BACKUP/loki

To restore logs in the event of an error, run:‌‌

# Upload the backup to the relevant folder in the Loki instance
kubectl -n monitoring cp BACKUP/loki loki-0:/data/loki

Happy logging!