Rocket Surgeon

  • 10 Posts
  • 411 Comments
Joined 8 months ago
cake
Cake day: March 10th, 2025

help-circle


  • SSH key management in PVE is handled in a set of secondary files, while the original debian files are replaced with symlinks. Well, that’s still debian. And in some circumstances the symlinks get b0rked or replaced with the original SSH files, the keys get out of sync, and one machine in the cluster can’t talk to another. The really irritating thing about this is that the tools meant to fix it (pvecm updatecerts) don’t work. I’ve got an elaborate set of procedures to gather the certs from the hosts and fix the files when it breaks, but it sux bad enough that I’ve got two clusters I’m putting off fixing.

    Corosync is the cluster. It’s a shared file system that immediately replicates any changes to all members. That’s essentially anything under /etc/pve/. Corosync is very sensitive. I believe they ask for 10ms lag or less between hosts, so it can’t work over a WAN connection. Shit like VM restores or vmotion between hosts can flood it out. Looks fukin awful when it goes down. Your whole cluster goes kaput.

    All corosync does is push around this set of config files, so a dedicated NIC is overkill, but in busy environments, you might wind up resorting to that. You can put cororsync on its own network, but you obviously need a network for that. And you can establish throttles on various types of host file transfer activities, but that’s a balancing act that I’ve only gotten right in our colos where we only have 1gb networks. I have my systems provisioned on a dedicated corosync vlan and also use a secondary IP on a different physical interface, but corosync is too dumb to fall back to the secondary if the primary is still “up”, regardless of whether its actually communicating, so I get calls on my day off about “the cluster is down!!!1” when people restore backups.



  • I use PVE professionally. I could spent some time bitching about how it handles ssh keys and the fragile corosync cluster management. I could complain about the sloppy release cycle and the way they move fast and break shit. Or all the janky shit they’ve slapped together in PBS. I could go on.

    But I actually pay for a license for my homelab. And ya, it is THE thing at work now.

    I’ve often heard it said that Proxmox isn’t a great option. But its the best one.
    If you do try it, don’t bother asking questions here.
    Go to the source. https://forum.proxmox.com/












  • Perhaps that is broadly true. Mostly my oven gets used for baking biscuits. Sometimes I’ll make a lasagna or very rarely a casserole. The only actual dietary sin that takes place there is the occasional pizza. I’m not perfect, I do eat pizza.
    There are no taquitos or chicken tenders or wtf-ever frozen garbage in my home or my oven.