Bala FA balamurugana

Challenges

Get drives from each node and identify drives for usability. DirectPV uses sysfs (/sys), udev (/run/udev/data) and procfs (/proc) on each node to collect all necessary information for this and they are stored in directpvnodes CRD.
Uniquely identify each drive not by drive name as it may change on node reboots/drive swaps/removal. DirectPV computes a hash using values collected for each drive to make sure it is unique across all nodes. This hash is called ID which is also stored in directpvnodes CRD.
Add/Remove of nodes and drives. DirectPV does not monitor addition or deletion of nodes and drives instead it collects information on demand to update directpvnodes CRD.

Run command $ kubectl edit directpvdrives <NEW-DRIVE-ID> and add directpv.min.io.volume/pvc-12345678 in the finalizer
Run command $ kubectl directpv cordon <NEW-DRIVE-ID>
Run command $ kubectl edit directpvdrives <NEW-DRIVE-ID> and remove the directpv.min.io.volume/pvc-12345678 finalizer
Run command $ ./replace.sh <OLD-DRIVE-ID> <NEW-DRIVE-ID>

Drive states

DriveStatusUnavailable

Drive CRD object is marked as Unavailable, if drive met one of the following condition at startup or added at runtime or dynamic discovery sync.

device size is less than 16 MiB.
device is in readonly state.
device contains partitions.
device is used as swap space in linux.
device is used in LVM, linux RAID or LUKS device.
device is mounted somewhere else i.e. outside of /var/lib/directcsi.

As MinIO needs the CN to have the policy name and original mkcert does not support this, download updated mkcert from https://github.com/kanagarajkm/mkcert/releases/download/v1.4.3-1/mkcert

Server certs

Generate certificate files

mkcert localhost

Copy generated localhost-key.pem and localhost.pem to certs directory of minio server.

cp -avi localhost-key.pem ~/.minio/certs/private.key

Object Lock Support

Objects can be set to immutable using object lock. Once an object is uploaded, this feature is manageable by below APIs.

A bucket can be set to have default object lock configuration using below API.

PUT Bucket object lock configuration

DataStore

A lock-free storage which supports to upload, download and delete data using Get, Put and Delete respectively. Every Put uses tmp directory as interim storage and every Delete is staged and actual removal is done once all Get are finished.

DataStore
|-- data/

Setup Information:

8 servers of t1.small.x86
minio version

Version: 2018-08-22T12:25:38Z
Release-Tag: DEVELOPMENT.2018-08-22T12-25-38Z
Commit-ID: 8601f29d95f2ad2055613bedcc917e692ffab5fd

mc version

Servers

Server1: minio server --address localhost:9001 https://localhost:9001/tmp/minio01/disk0{1..4} https://localhost:9002/tmp/minio02/disk0{1..4} https://localhost:9003/tmp/minio03/disk0{1..4} https://localhost:9004/tmp/minio04/disk0{1..2} https://localhost:9005/tmp/minio05/disk0{1..2}
Server2: minio server --address localhost:9002 https://localhost:9001/tmp/minio01/disk0{1..4} https://localhost:9002/tmp/minio02/disk0{1..4} https://localhost:9003/tmp/minio03/disk0{1..4} https://localhost:9004/tmp/minio04/disk0{1..2} https://localhost:9005/tmp/minio05/disk0{1..2}
Server3: minio server --address localhost:9003 https://localhost:9001/tmp/minio01/disk0{1..4} https://localhost:9002/tmp/minio02/disk0{1..4} https://localhost:9003/tmp/minio03/disk0{1..4} https://localhost:9004/tmp/minio04/disk0{1..2} https://localhost:9005/tmp/minio05/disk0{1..2}
Server4: minio server --address localhost:9004 https://localhost:9001/tmp/minio01/disk0{1..4} https://localhost:9002/tmp/minio02/disk0{1..4} https://localhost:9003

Setup information

minio version

+ minio version
Version: 2018-08-10T10:26:45Z
Release-Tag: DEVELOPMENT.2018-08-10T10-26-45Z
Commit-ID: 8fa104f4287549dc2df5aa50c0fd0316e7186886

	#!/usr/bin/python

	import json
	import os
	import subprocess

	def _read_file(filename):
	try:
	with open(filename) as file:
	return file.read().strip()