-
Collect information about your environment.
❯ oc get nodes NAME STATUS ROLES AGE VERSION cp0 Ready control-plane,master,worker 66d v1.29.8+f10c92d cp1 Ready control-plane,master,worker 66d v1.29.8+f10c92d cp2 Ready control-plane,master,worker 66d v1.29.8+f10c92d
-
Have a look at each of the nodes associated physical disks.
NODE_NAME=node/cp0 cat <<EOF | oc debug $NODE_NAME chroot /host lsblk -o NAME,ROTA,SIZE,TYPE EOF
The output will look similar to the following:
sh-5.1# lsblk -o NAME,ROTA,SIZE,TYPE sh-5.1# exit NAME ROTA SIZE TYPE sda 1 1.8T disk sdb 1 5.5T disk sdc 1 7.3T disk sdd 1 3.6T disk sde 1 931G disk |-sde1 1 1M part |-sde2 1 127M part |-sde3 1 384M part `-sde4 1 930.5G part sr0 1 1024M rom Removing debug pod ...
-
Now I want you to run the following command, which will list out all of the mappings between disk assignments (i.e.
/dev/sdc) and disk by-path (i.e./dev/disk/by-path/).NODE_NAME=node/cp0 cat <<EOF | oc debug $NODE_NAME chroot /host ls -aslc /dev/disk/by-path/ EOF
The output will look like the following:
sh-5.1# ls -aslc /dev/disk/by-path/ sh-5.1# exit total 0 0 drwxr-xr-x. 2 root root 260 Sep 7 00:30 . 0 drwxr-xr-x. 9 root root 180 Sep 7 00:30 .. 0 lrwxrwxrwx. 1 root root 9 Sep 7 00:30 pci-0000:00:17.0-ata-8 -> ../../sr0 0 lrwxrwxrwx. 1 root root 9 Sep 7 00:30 pci-0000:00:17.0-ata-8.0 -> ../../sr0 0 lrwxrwxrwx. 1 root root 9 Sep 7 00:30 pci-0000:1a:00.0-scsi-0:2:0:0 -> ../../sda 0 lrwxrwxrwx. 1 root root 9 Sep 7 00:30 pci-0000:1a:00.0-scsi-0:2:1:0 -> ../../sdc 0 lrwxrwxrwx. 1 root root 9 Sep 7 00:30 pci-0000:1a:00.0-scsi-0:2:2:0 -> ../../sdb 0 lrwxrwxrwx. 1 root root 9 Sep 7 00:30 pci-0000:1a:00.0-scsi-0:2:3:0 -> ../../sdd 0 lrwxrwxrwx. 1 root root 9 Sep 7 00:30 pci-0000:1a:00.0-scsi-0:2:4:0 -> ../../sde 0 lrwxrwxrwx. 1 root root 10 Sep 7 00:30 pci-0000:1a:00.0-scsi-0:2:4:0-part1 -> ../../sde1 0 lrwxrwxrwx. 1 root root 10 Sep 7 00:30 pci-0000:1a:00.0-scsi-0:2:4:0-part2 -> ../../sde2 0 lrwxrwxrwx. 1 root root 10 Sep 7 00:30 pci-0000:1a:00.0-scsi-0:2:4:0-part3 -> ../../sde3 0 lrwxrwxrwx. 1 root root 10 Sep 7 00:30 pci-0000:1a:00.0-scsi-0:2:4:0-part4 -> ../../sde4 Removing debug pod ...
Source Documentation: HERE
-
The first thing I would like you to do is download a Red Hat utility called
butane. I'm going to be installing the ARM-based MacOS binary, but be sure to change theARCHvariable to something more useful if required for your system.ARCH="darwin-amd64" mkdir -p ~/.local/bin/ curl https://mirror.openshift.com/pub/openshift-v4/clients/butane/latest/butane-$ARCH --output ~/.local/bin/butane chmod +x ~/.local/bin/butane butane --version
-
Now, have a look at the docs for Butane, however I'm going to show you everything you need to do right here. We're going to use the
butaneutility to create a customMachineConfigthat we can apply to each of our nodes. So let's create a file called99-fake-nonrotational-mc.bu.cat << 'EOF' > 99-fake-nonrotational-mc.bu variant: openshift version: 4.17.0 metadata: name: 99-fake-nonrotational labels: machineconfiguration.openshift.io/role: master storage: files: - path: /etc/fake-nonrotational.sh mode: 0755 contents: inline: | #!/bin/bash # Find disks with a size of 120G: target_disks=$(lsblk -dn -o NAME,SIZE | awk '$2 == "120G" {print $1}') # Mark these disks as NonRotational: for disk in $target_disks; do echo "Changing disk: /dev/$disk to non-rotational." echo 0 > /sys/block/$disk/queue/rotational done systemd: units: - name: fake-nonrotational.service enabled: true contents: | [Unit] Description=Force specific-size disks to be nonrotational After=local-fs.target Wants=local-fs.target [Service] Type=oneshot ExecStart=/etc/fake-nonrotational.sh Restart=on-failure RestartSec=5s RemainAfterExit=true User=root [Install] WantedBy=multi-user.target EOF
What you'll notice is that we have the script embedded in our
butane.bufile (see the example below). If you want to make changes to the script for your use case; make these changes now.#!/bin/bash # Find disks with a size of 120G: target_disks=$(lsblk -dn -o NAME,SIZE | awk '$2 == "120G" {print $1}') # Mark that disk as NonRotational: for disk in $target_disks; do echo "Changing disk: /dev/$disk to non-rotational." echo 0 > /sys/block/$disk/queue/rotational done
My script example above is doing the following:
- Look for a disk with 120G (exactly)
- Change the disk from a
Rotationaldisk to aNonRotationaldisk by editing the value in/sys/block/$disk/queue/rotational(but only for the disk that matches 120G
-
Now you can run
butaneto create theMachineConfigby using the following command.butane 99-fake-nonrotational-mc.bu -o 99-fake-nonrotational-mc.yaml
-
Below was the final result of my
butaneresults. I am simply going to add anannotations.descriptionto myMachiineConfig, but after I've done that I'm going to apply the manifest to my cluster like so.cat <<EOF | oc apply -f - --- apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: master annotations: description: "MC sets disks with size 120G to non-rotational. Documentation found at: https://tinyurl.com/mc-nonrotational-hack" name: 99-fake-nonrotational spec: config: ignition: version: 3.4.0 storage: files: - contents: compression: gzip source: data:;base64,H4sIAAAAAAAC/0zNvU7DQBDE8X6fYnBOCkiYTVKCjIRQQBTQ0NGgc+5ir2ztgu9CxNe7I44i9P/fzOyIW1FufeqJZrgRDQiShoS95B4eST4ibIvlanFL2U9dzM8laNzxmNpxQB0UteHh6n59+nj3tMYX/H7A3K3QNKh+YYXPl0k0wy2/5ydEW5vKC0Th/o9eIBgBQNz0huq699qJdiU+B4f4xq7AbFDTerLss5j68aw6uAUuwek9cTvaZvgT/LqLu8gHQME00k8AAAD//0gklW0AAQAA mode: 493 path: /etc/fake-nonrotational.sh systemd: units: - contents: | [Unit] Description=Force specific-size disks to be nonrotational After=local-fs.target Wants=local-fs.target [Service] Type=oneshot ExecStart=/etc/fake-nonrotational.sh Restart=on-failure RestartSec=5s RemainAfterExit=true User=root [Install] WantedBy=multi-user.target enabled: true name: fake-nonrotational.service EOF
-
To verify that things are working, use the following command:
NODE_NAME=node/cp0 cat <<EOF | oc debug $NODE_NAME cat /sys/block/sda/queue/rotational cat /sys/block/sdb/queue/rotational EOF
-
Sometimes you may see an error that looks like this:
Conditions: Last Transition Time: 2024-12-18T17:05:42Z Message: Reason: Status: False Type: Updated Last Transition Time: 2024-12-18T17:14:54Z Message: Reason: Status: True Type: Degraded Last Transition Time: 2024-12-18T17:05:42Z Message: All nodes are updating to MachineConfig rendered-master-957dc299504d99fbee5caf23b9bd8177 Reason: Status: True Type: Updating Last Transition Time: 2024-12-18T19:53:45Z Message: Reason: Status: False Type: RenderDegraded Last Transition Time: 2024-12-18T17:14:54Z Message: Node ztp-node-00 is reporting: "unexpected on-disk state validating against rendered-master-4fa0363ea5ab86be4f8dca739f6a75e5: could not stat file \"/etc/fake-nonrotational.sh\": lstat /etc/fake-nonrotational.sh: no such file or directory" Reason: 1 nodes are reporting degraded status on sync Status: True Type: NodeDegraded -
When this happens, you can force the change on the remote host with the following command (keep in mind that the host will reboot as a result of this forced application):
touch /run/machine-config-daemon-force
-
Now if you accidentally delete a machine-config that was applied to a node, you will need to follow the Red Hat KCP article HERE