Some more Prometheus content now, this time I will share how I do my alerting. I want to to receive alerts from my homelab, but I do not care about them too much. If something goes wrong at 2am, I really could not careless. In fact, there is no reason why I would want to be disturbed by them under any circumstances.

To that end, the whole point of my homelab is that I can use the same techniques I would use in production. To that end, I do want to set up alerts, I just want to keep them as non-intrusive as possible. A good solution for my alerts then is Telegram, as I can have them sent to a channel that I mute. Now I can see them, but I am never actually disturbed by them.

I run Kubernetes in my homelab and use the (now classic) Prometheus/Grafana stack for monitoring everything. In my experience, the thing that Prometheus does not do well is long-term storage. Prometheus’ own database is fine for a few days storage, but performance and reliability quickly degrade. There are quite a few solutions to this problem, but I choose to use Thanos, partly because Kube Prometheus Stack has very good support for it.