I’m looking for experiences and opinions on kubernetes storage.

I want to create a highly available homelab that spans 3 locations where the pods have a preferred locations but can move if necessary.

I’ve looked at linstore or seaweedfs/garage with juicefs but I’m not sure how well the performance of those options is across the internet and how well they last in long term operation. Is anyone else hosting k3s across the internet in their homelab?

Edit: fixed wording

  • ChaosMonkey@lemmy.dbzer0.comEnglish
    6·
    17 hours ago

    Longhorn is pretty easy to use. Garage works well too. Ceph is harder to use but provides both block and object storage (s3).

    • InnerScientist@lemmy.worldOPEnglish
      1·
      16 hours ago

      Ceph (and longhorn) want “10 Gbps network bandwidth between nodes” while I’ll have around 1gbit between nodes, or even lower.

      What’s your experience with Garage?

      • ChaosMonkey@lemmy.dbzer0.comEnglish
        2·
        15 hours ago

        I guess the network will be a bottleneck on Garage too. If you want high performance you might need a hybrid solution, like clustering of stateful apps on local storage as well as periodic full backups on a distributed storage.

        • InnerScientist@lemmy.worldOPEnglish
          2·
          15 hours ago

          It’s fine if the bottleneck is upload/download speed, there’s no easy way around that.
          The other problems like high latency or using more bandwith than is required are more my fear. Maybe local read cache or stuff like that can be a solution too but that’s why I’m asking for what is in use and what works vs what is better reserved for dedicated networks.