ronindesign

CEPH HA Setup

Note this should only be done once you are sure you have reliable TB mesh network.

this is because proxmox UI seems fragile wrt to changing underlying network after configuration of ceph.

All installation done via command line due to gui not understanding the mesh network

This setup doesn't attempt to seperate the ceph public network and ceph cluster network (not same as proxmox clutser network), The goal is to get an easy working setup.

**2025.04.24 NOTE: some folks had to switch to IPv6 for ceph due to IPv4 unreliability issues, we think as of pve 8.4.1 and all the input the community has give to update this set of gsists - that IPv4 is now reliable even on MS-01. As such i advising everyone to use IPv4 for ceph as if you have IPv6 you will have issues with SDN at this time (if you don't use SDN this is not an issue).

ronindesign

Proxmox cluster Setup

this gist is part of this series

Network Design

Put simply I am not sure what the design should be. I have the thunderbolt mesh network and the 2.5gbe NIC on each node. The ideal design guidelies cause my brain to have a race conditions because:

1. ceph shold have a dedicated network
2. proxmox should not have migration traffic and cluster communications network
3. one wants cluster communicationsnetwork reddundant

ronindesign

Enable Dual Stack (IPv4 and IPv6) OpenFabric Routing

!!WORK IN PROGRESS!!

Version 2.1 (2025.04.24)

this gist is part of this series

This assumes you are running Proxmox 8.4 and that the line source /etc/network/interfaces.d/* is at the end of the interfaces file (this is automatically added to both new and upgraded installations of Proxmox 8.2).

This changes the previous file design thanks to @NRGNet and @tisayama to make the system much more reliable in general, more maintainable esp for folks using IPv4 on the private cluster network (i still recommend the use of the IPv6 FC00 network you will see in these docs)

ronindesign

ProxMox Cluster - Soup-to-Nutz

aka what i did to get from nothing to done.

note: these are designed to be primarily a re-install guide for myself (writing things down helps me memorize the knowledge), as such don't take any of this on blind faith - some areas are well tested and the docs are very robust, some items, less so). YMMV

Purpose of Proxmox cluster project

Required Outomces of cluster project

ronindesign

<!-- put this to IDEA keymaps config folder. For v13 it is <userdir>\.IntelliJIdea13\config\keymaps\ -->
<?xml version="1.0" encoding="UTF-8"?>
<keymap version="1" name="Mac OS X 10.5+ Windows Ctrl" parent="Mac OS X 10.5+">
  <action id="$Copy">
    <keyboard-shortcut first-keystroke="meta C" />
    <keyboard-shortcut first-keystroke="meta INSERT" />
    <keyboard-shortcut first-keystroke="control C" />
    <keyboard-shortcut first-keystroke="control INSERT" />
  </action>
  <action id="$Cut">

ronindesign

object User "pagerduty" {
  pager = "SERVICE_KEY_HERE"
  groups = [ "icingaadmins" ]
  display_name = "PagerDuty Notification User"
  states = [ OK, Warning, Critical, Unknown ]
  types = [ Problem, Recovery ]
}

object NotificationCommand "notify-service-by-pagerduty" {
  import "plugin-notification-command"