I run Kubernetes in my homelab and use the (now classic) Prometheus/Grafana stack for monitoring everything. In my experience, the thing that Prometheus does not do well is long-term storage. Prometheus’ own database is fine for a few days storage, but performance and reliability quickly degrade. There are quite a few solutions to this problem, but I choose to use Thanos, partly because Kube Prometheus Stack has very good support for it.