Kubernetes on YugabyteDB: Final Instruction

k3s_yugabytedb_final.md

Start YugabyteDB Instance

rm -r ~/yb_docker_data
mkdir ~/yb_docker_data

docker network create custom-network

docker run -d --name yugabytedb_node1 --net custom-network \
  -p 15433:15433 -p 7001:7000 -p 9000:9000 -p 5433:5433 \
  -v ~/yb_docker_data/node1:/home/yugabyte/yb_data --restart unless-stopped \
  yugabytedb/yugabyte:latest \
  bin/yugabyted start --tserver_flags="ysql_sequence_cache_minval=1" \
  --base_dir=/home/yugabyte/yb_data --daemon=false

docker run -d --name yugabytedb_node2 --net custom-network \
  -p 15434:15433 -p 7002:7000 -p 9002:9000 -p 5434:5433 \
  -v ~/yb_docker_data/node2:/home/yugabyte/yb_data --restart unless-stopped \
  yugabytedb/yugabyte:latest \
  bin/yugabyted start --join=yugabytedb_node1 --tserver_flags="ysql_sequence_cache_minval=1" \
  --base_dir=/home/yugabyte/yb_data --daemon=false
      
docker run -d --name yugabytedb_node3 --net custom-network \
  -p 15435:15433 -p 7003:7000 -p 9003:9000 -p 5435:5433 \
  -v ~/yb_docker_data/node3:/home/yugabyte/yb_data --restart unless-stopped \
  yugabytedb/yugabyte:latest \
  bin/yugabyted start --join=yugabytedb_node1 --tserver_flags="ysql_sequence_cache_minval=1" \
  --base_dir=/home/yugabyte/yb_data --daemon=false

With auto-explain:

docker run -d --name yugabytedb_node1 --net custom-network \
  -p 15433:15433 -p 7001:7000 -p 9000:9000 -p 5433:5433 \
  -v ~/yb_docker_data/node1:/home/yugabyte/yb_data --restart unless-stopped \
  yugabytedb/yugabyte:latest \
  bin/yugabyted start --tserver_flags="flagfile=/home/yugabyte/yb_data/tserver_flags" \
  --base_dir=/home/yugabyte/yb_data --daemon=false

Start Default Kine Instance

1. Clone the Kine repo:

   git clone https://github.com/k3s-io/kine

2. Start a Kine instance connecting to the YugabyteDB cluster:

   go run . --endpoint "postgres://yugabyte:[email protected]:5433/yugabyte"

3. Connect to YugabyteDB with psql:

   psql -h 127.0.0.1 -p 5433 -U yugabyte

4. Check that the Kine schema is ready:

   yugabyte=# \d
                List of relations
    Schema |    Name     |   Type   |  Owner
   --------+-------------+----------+----------
    public | kine        | table    | yugabyte
    public | kine_id_seq | sequence | yugabyte
   (2 rows)

5. Stop the Kine instance and drop the schema on YugabyteDB end:

   drop table kine cascade;

Start Kine Instance With YugabyteDB Support

Now, experiment with the Kine version that supports YugabyteDB backend. The YugabyteDB backend is an optimized version of the original Kine's backend for Postgres.

1. Clone the Kine repo:

   git clone https://github.com/dmagda/kine-yugabytedb.git

2. Start a Kine instance connecting to the YugabyteDB cluster:

   go run . --endpoint "yugabytedb://yugabyte:[email protected]:5433/yugabyte"

3. Connect to YugabyteDB with psql:

   psql -h 127.0.0.1 -p 5433 -U yugabyte

4. Check that the Kine schema is ready:

   yugabyte=# \d
               List of relations
    Schema |   Name   |   Type   |  Owner
   --------+----------+----------+----------
    public | kine     | table    | yugabyte
    public | kine_seq | sequence | yugabyte
   (2 rows)

5. Make sure the batched nested loops are set to 1024:

    yugabyte=# show yb_bnl_batch_size;
    yb_bnl_batch_size
   -------------------
    1024
   (1 row)

6. Stop the Kine instance and drop the schema on YugabyteDB end:

   drop table kine cascade;
   drop sequence kine_seq;

Build k3s With YugabyteDB

Now, you need to start a k3s instance using the Kine version support the YugabyteDB backend. To do that, you need to be k3s from sources: https://github.com/k3s-io/k3s/blob/master/BUILDING.md

1. Clone k3s:

   git clone --depth 1 https://github.com/k3s-io/k3s.git

2. Remove artifacts from a previous release if any:

   rm -r build
   rm -r dist

3. Open the go.mod file and add the following line to the end of the replace (..) section:

   github.com/k3s-io/kine => github.com/dmagda/kine-yugabytedb v0.2.0

4. Add the private repo:

   go env -w GOPRIVATE=github.com/dmagda/kine-yugabytedb

5. Make sure the changes take effect:

   go mod tidy

6. Prepare for a k3s full release:

   mkdir -p build/data && make download && make generate

7. Build a full release:

   SKIP_VALIDATE=true make

Run k3s With YugabyteDB

1. Navigate to the directory with the k3s build artifacts:

cd dist/artifacts/

2. Start a k3s server connecting it to a YugabyteDB backend:

  sudo ./k3s server \
  --token=sample_secret_token \
  --datastore-endpoint="yugabytedb://yugabyte:[email protected]:5433/yugabyte"

3. Make sure the server node is ready:

sudo ./k3s kubectl get nodes
NAME                   STATUS   ROLES                  AGE   VERSION
market-orders-app-vm   Ready    control-plane,master   11m   v1.27.3+k3s-be442433

Deploy Application

Let's deploy a sample application to confirm the cluster is usable. You can consider this one: https://github.com/digitalocean/kubernetes-sample-apps/tree/master/emojivoto-example

Deploy:

sudo ./k3s kubectl apply -k ~/kubernetes-sample-apps/emojivoto-example/kustomize

Confirm:

sudo ./k3s kubectl get all -n emojivoto

Uninstall scrupt for k3s:

First this command:

sudo kill $(ps aux | grep 'k3s' | awk '{print $2}')

and then this script:

#!/bin/sh
set -x
systemctl stop k3s
systemctl disable k3s
systemctl daemon-reload

rm -f /etc/systemd/system/k3s.service
rm -f /usr/local/bin/k3s
if [ -L /usr/local/bin/kubectl ]; then
    rm -f /usr/local/bin/kubectl
fi
if [ -L /usr/local/bin/crictl ]; then
    rm -f /usr/local/bin/crictl
fi
if [ -e /sys/fs/cgroup/systemd/system.slice/k3s.service/cgroup.procs ]; then
    kill -9 `cat /sys/fs/cgroup/systemd/system.slice/k3s.service/cgroup.procs`
fi
umount `cat /proc/self/mounts | awk '{print $2}' | grep '^/run/k3s'`
umount `cat /proc/self/mounts | awk '{print $2}' | grep '^/var/lib/rancher/k3s'`

rm -rf /var/lib/rancher/k3s
rm -rf /etc/rancher/k3s
rm -rf /etc/rancher

rm -f /usr/local/bin/k3s-uninstall.sh

Creating a new Kine YugabyteDB release:

1. Clone the repo:

git clone https://github.com/dmagda/kine-yugabytedb.git

2. Create and push a new tag:

git tag vX.Y.Z
git push --tags

3. Proceed to the Github release page to publish a new release from the tag:
   https://github.com/dmagda/kine-yugabytedb/releases

INFO[0015] Slow SQL (started: 2023-07-20 19:14:12.762674737 +0000 UTC m=+1.291026213) (total time: 14.334123498s): SELECT * FROM ( SELECT ( SELECT MAX(rkv.id) AS id FROM kine AS rkv), ( SELECT MAX(crkv.prev_revision) AS prev_revision FROM kine AS crkv WHERE crkv.name = 'compact_rev_key'), kv.id AS theid, kv.name, kv.created, kv.deleted, kv.create_revision, kv.prev_revision, kv.lease, kv.value, kv.old_value FROM kine AS kv JOIN ( SELECT MAX(mkv.id) AS id FROM kine AS mkv WHERE mkv.name LIKE $1 GROUP BY mkv.name) AS maxkv ON maxkv.id = kv.id WHERE kv.deleted = 0 OR $2 ) AS lkv ORDER BY lkv.theid ASC LIMIT 10001

Start with postgres:

sudo ./k3s server \
  --token=sample_secret_token \
  --datastore-endpoint="postgres://postgres:[email protected]:5432/postgres"