kubeadm dual stack

gistfile1.txt

Install

I0224 18:07:50.738072   97872 initconfiguration.go:104] detected and using CRI socket: /var/run/dockershim.sock
I0224 18:07:50.739159   97872 interface.go:400] Looking for default routes with IPv4 addresses
I0224 18:07:50.739187   97872 interface.go:405] Default route transits interface "ens160"
I0224 18:07:50.742019   97872 interface.go:208] Interface ens160 is up
I0224 18:07:50.742251   97872 interface.go:256] Interface "ens160" has 2 addresses :[10.6.177.40/16 fe80::c46f:119c:6e9c:cf7c/64].
I0224 18:07:50.742318   97872 interface.go:223] Checking addr  10.6.177.40/16.
I0224 18:07:50.742344   97872 interface.go:230] IP found 10.6.177.40
I0224 18:07:50.742367   97872 interface.go:262] Found valid IPv4 address 10.6.177.40 for interface "ens160".
I0224 18:07:50.742417   97872 interface.go:411] Found active IP 10.6.177.40
I0224 18:07:50.905117   97872 version.go:185] fetching Kubernetes version from URL: https://dl.k8s.io/release/stable-1.txt
[init] Using Kubernetes version: v1.20.4
[preflight] Running pre-flight checks
I0224 18:07:51.981545   97872 checks.go:577] validating Kubernetes and kubeadm version
I0224 18:07:51.981636   97872 checks.go:166] validating if the firewall is enabled and active
I0224 18:07:52.019138   97872 checks.go:201] validating availability of port 6443
I0224 18:07:52.019805   97872 checks.go:201] validating availability of port 10259
I0224 18:07:52.019909   97872 checks.go:201] validating availability of port 10257
I0224 18:07:52.019987   97872 checks.go:286] validating the existence of file /etc/kubernetes/manifests/kube-apiserver.yaml
I0224 18:07:52.020046   97872 checks.go:286] validating the existence of file /etc/kubernetes/manifests/kube-controller-manager.yaml
I0224 18:07:52.020072   97872 checks.go:286] validating the existence of file /etc/kubernetes/manifests/kube-scheduler.yaml
I0224 18:07:52.020091   97872 checks.go:286] validating the existence of file /etc/kubernetes/manifests/etcd.yaml
I0224 18:07:52.020116   97872 checks.go:432] validating if the connectivity type is via proxy or direct
I0224 18:07:52.020153   97872 checks.go:471] validating http connectivity to first IP address in the CIDR
I0224 18:07:52.020194   97872 checks.go:471] validating http connectivity to first IP address in the CIDR
I0224 18:07:52.020226   97872 checks.go:471] validating http connectivity to first IP address in the CIDR
I0224 18:07:52.020253   97872 checks.go:471] validating http connectivity to first IP address in the CIDR
I0224 18:07:52.020277   97872 checks.go:102] validating the container runtime
I0224 18:07:52.131338   97872 checks.go:128] validating if the "docker" service is enabled and active
I0224 18:07:52.285665   97872 checks.go:335] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables
I0224 18:07:52.285877   97872 checks.go:335] validating the contents of file /proc/sys/net/ipv4/ip_forward
I0224 18:07:52.286007   97872 checks.go:649] validating whether swap is enabled or not
I0224 18:07:52.286478   97872 checks.go:376] validating the presence of executable conntrack
I0224 18:07:52.286890   97872 checks.go:376] validating the presence of executable ip
I0224 18:07:52.287191   97872 checks.go:376] validating the presence of executable iptables
I0224 18:07:52.287289   97872 checks.go:376] validating the presence of executable mount
I0224 18:07:52.287618   97872 checks.go:376] validating the presence of executable nsenter
I0224 18:07:52.287701   97872 checks.go:376] validating the presence of executable ebtables
I0224 18:07:52.287762   97872 checks.go:376] validating the presence of executable ethtool
I0224 18:07:52.287874   97872 checks.go:376] validating the presence of executable socat
I0224 18:07:52.287926   97872 checks.go:376] validating the presence of executable tc
	[WARNING FileExisting-tc]: tc not found in system path
I0224 18:07:52.288066   97872 checks.go:376] validating the presence of executable touch
I0224 18:07:52.288128   97872 checks.go:520] running all checks
	[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.2. Latest validated version: 19.03
I0224 18:07:52.414067   97872 checks.go:406] checking whether the given node name is reachable using net.LookupHost
I0224 18:07:52.414092   97872 checks.go:618] validating kubelet version
I0224 18:07:52.499361   97872 checks.go:128] validating if the "kubelet" service is enabled and active
I0224 18:07:52.522104   97872 checks.go:201] validating availability of port 10250
I0224 18:07:52.522401   97872 checks.go:201] validating availability of port 2379
I0224 18:07:52.522608   97872 checks.go:201] validating availability of port 2380
I0224 18:07:52.522699   97872 checks.go:249] validating the existence and emptiness of directory /var/lib/etcd
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0224 18:07:52.573544   97872 checks.go:839] image exists: k8s.gcr.io/kube-apiserver:v1.20.4
I0224 18:07:52.627950   97872 checks.go:839] image exists: k8s.gcr.io/kube-controller-manager:v1.20.4
I0224 18:07:52.677646   97872 checks.go:839] image exists: k8s.gcr.io/kube-scheduler:v1.20.4
I0224 18:07:52.729249   97872 checks.go:839] image exists: k8s.gcr.io/kube-proxy:v1.20.4
I0224 18:07:52.776103   97872 checks.go:839] image exists: k8s.gcr.io/pause:3.2
I0224 18:07:52.827973   97872 checks.go:839] image exists: k8s.gcr.io/etcd:3.4.13-0
I0224 18:07:52.886191   97872 checks.go:839] image exists: k8s.gcr.io/coredns:1.7.0
[certs] Using certificateDir folder "/etc/kubernetes/pki"
I0224 18:07:52.886336   97872 certs.go:110] creating a new certificate authority for ca
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [daocloud kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [172.31.0.1 10.6.177.40]
[certs] Generating "apiserver-kubelet-client" certificate and key
I0224 18:07:53.791393   97872 certs.go:110] creating a new certificate authority for front-proxy-ca
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
I0224 18:07:54.162602   97872 certs.go:110] creating a new certificate authority for etcd-ca
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [daocloud localhost] and IPs [10.6.177.40 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [daocloud localhost] and IPs [10.6.177.40 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
I0224 18:07:55.167061   97872 certs.go:76] creating new public/private key files for signing service account users
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0224 18:07:55.309676   97872 kubeconfig.go:101] creating kubeconfig file for admin.conf
[kubeconfig] Writing "admin.conf" kubeconfig file
I0224 18:07:55.728865   97872 kubeconfig.go:101] creating kubeconfig file for kubelet.conf
[kubeconfig] Writing "kubelet.conf" kubeconfig file
I0224 18:07:55.867842   97872 kubeconfig.go:101] creating kubeconfig file for controller-manager.conf
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0224 18:07:56.193183   97872 kubeconfig.go:101] creating kubeconfig file for scheduler.conf
[kubeconfig] Writing "scheduler.conf" kubeconfig file
I0224 18:07:56.472539   97872 kubelet.go:63] Stopping the kubelet
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
I0224 18:07:56.694670   97872 manifests.go:96] [control-plane] getting StaticPodSpecs
I0224 18:07:56.695787   97872 manifests.go:109] [control-plane] adding volume "ca-certs" for component "kube-apiserver"
I0224 18:07:56.695827   97872 manifests.go:109] [control-plane] adding volume "etc-pki" for component "kube-apiserver"
I0224 18:07:56.695846   97872 manifests.go:109] [control-plane] adding volume "k8s-certs" for component "kube-apiserver"
I0224 18:07:56.706277   97872 manifests.go:126] [control-plane] wrote static Pod manifest for component "kube-apiserver" to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
I0224 18:07:56.706317   97872 manifests.go:96] [control-plane] getting StaticPodSpecs
I0224 18:07:56.706703   97872 manifests.go:109] [control-plane] adding volume "ca-certs" for component "kube-controller-manager"
I0224 18:07:56.706722   97872 manifests.go:109] [control-plane] adding volume "etc-pki" for component "kube-controller-manager"
I0224 18:07:56.706731   97872 manifests.go:109] [control-plane] adding volume "flexvolume-dir" for component "kube-controller-manager"
I0224 18:07:56.706737   97872 manifests.go:109] [control-plane] adding volume "k8s-certs" for component "kube-controller-manager"
I0224 18:07:56.706750   97872 manifests.go:109] [control-plane] adding volume "kubeconfig" for component "kube-controller-manager"
I0224 18:07:56.707839   97872 manifests.go:126] [control-plane] wrote static Pod manifest for component "kube-controller-manager" to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[control-plane] Creating static Pod manifest for "kube-scheduler"
I0224 18:07:56.707872   97872 manifests.go:96] [control-plane] getting StaticPodSpecs
I0224 18:07:56.708226   97872 manifests.go:109] [control-plane] adding volume "kubeconfig" for component "kube-scheduler"
I0224 18:07:56.708914   97872 manifests.go:126] [control-plane] wrote static Pod manifest for component "kube-scheduler" to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0224 18:07:56.709944   97872 local.go:74] [etcd] wrote Static Pod manifest for a local etcd member to "/etc/kubernetes/manifests/etcd.yaml"
I0224 18:07:56.709970   97872 waitcontrolplane.go:87] [wait-control-plane] Waiting for the API server to be healthy
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
I0224 18:09:07.051656   97872 request.go:943] Got a Retry-After 1s response for attempt 1 to https://10.6.177.40:6443/healthz?timeout=10s
[apiclient] All control plane components are healthy after 105.504600 seconds
I0224 18:09:42.216442   97872 uploadconfig.go:108] [upload-config] Uploading the kubeadm ClusterConfiguration to a ConfigMap
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
I0224 18:09:42.237247   97872 uploadconfig.go:122] [upload-config] Uploading the kubelet component config to a ConfigMap
[kubelet] Creating a ConfigMap "kubelet-config-1.20" in namespace kube-system with the configuration for the kubelets in the cluster
I0224 18:09:42.248965   97872 uploadconfig.go:127] [upload-config] Preserving the CRISocket information for the control-plane node
I0224 18:09:42.248996   97872 patchnode.go:30] [patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "daocloud" as an annotation
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node daocloud as control-plane by adding the labels "node-role.kubernetes.io/master=''" and "node-role.kubernetes.io/control-plane='' (deprecated)"
[mark-control-plane] Marking the node daocloud as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 0xdgpd.bs4g0c75dg74n3bs
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
I0224 18:09:43.322877   97872 clusterinfo.go:45] [bootstrap-token] loading admin kubeconfig
I0224 18:09:43.323490   97872 clusterinfo.go:53] [bootstrap-token] copying the cluster from admin.conf to the bootstrap kubeconfig
I0224 18:09:43.323737   97872 clusterinfo.go:65] [bootstrap-token] creating/updating ConfigMap in kube-public namespace
I0224 18:09:43.326109   97872 clusterinfo.go:79] creating the RBAC rules for exposing the cluster-info ConfigMap in the kube-public namespace
I0224 18:09:43.330185   97872 kubeletfinalize.go:88] [kubelet-finalize] Assuming that kubelet client certificate rotation is enabled: found "/var/lib/kubelet/pki/kubelet-client-current.pem"
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
I0224 18:09:43.331384   97872 kubeletfinalize.go:132] [kubelet-finalize] Restarting the kubelet to enable client certificate rotation
[addons] Applied essential addon: CoreDNS
I0224 18:09:43.674182   97872 request.go:591] Throttling request took 51.134984ms, request: POST:https://10.6.177.40:6443/api/v1/namespaces/kube-system/configmaps?timeout=10s
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.6.177.40:6443 --token 0xdgpd.xxx \
    --discovery-token-ca-cert-hash sha256:xxx



Join

[root@dce-kubeadm-2 ~]# kubeadm join 10.6.177.40:6443 --token 0xdgpd.xxx     --discovery-token-ca-cert-hash sha256:xxxx --v=4
I0224 18:44:35.667340    2925 join.go:395] [preflight] found NodeName empty; using OS hostname as NodeName
I0224 18:44:35.667472    2925 initconfiguration.go:104] detected and using CRI socket: /run/containerd/containerd.sock
[preflight] Running pre-flight checks
I0224 18:44:35.667580    2925 preflight.go:90] [preflight] Running general checks
I0224 18:44:35.667634    2925 checks.go:249] validating the existence and emptiness of directory /etc/kubernetes/manifests
I0224 18:44:35.667651    2925 checks.go:286] validating the existence of file /etc/kubernetes/kubelet.conf
I0224 18:44:35.667669    2925 checks.go:286] validating the existence of file /etc/kubernetes/bootstrap-kubelet.conf
I0224 18:44:35.667679    2925 checks.go:102] validating the container runtime
I0224 18:44:35.679526    2925 checks.go:376] validating the presence of executable crictl
I0224 18:44:35.679582    2925 checks.go:335] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables
I0224 18:44:35.679653    2925 checks.go:335] validating the contents of file /proc/sys/net/ipv4/ip_forward
I0224 18:44:35.679683    2925 checks.go:649] validating whether swap is enabled or not
I0224 18:44:35.679723    2925 checks.go:376] validating the presence of executable conntrack
I0224 18:44:35.679744    2925 checks.go:376] validating the presence of executable ip
I0224 18:44:35.679762    2925 checks.go:376] validating the presence of executable iptables
I0224 18:44:35.679781    2925 checks.go:376] validating the presence of executable mount
I0224 18:44:35.679838    2925 checks.go:376] validating the presence of executable nsenter
I0224 18:44:35.679859    2925 checks.go:376] validating the presence of executable ebtables
I0224 18:44:35.679877    2925 checks.go:376] validating the presence of executable ethtool
I0224 18:44:35.679897    2925 checks.go:376] validating the presence of executable socat
I0224 18:44:35.679916    2925 checks.go:376] validating the presence of executable tc
	[WARNING FileExisting-tc]: tc not found in system path
I0224 18:44:35.679970    2925 checks.go:376] validating the presence of executable touch
I0224 18:44:35.679997    2925 checks.go:520] running all checks
I0224 18:44:35.694647    2925 checks.go:406] checking whether the given node name is reachable using net.LookupHost
I0224 18:44:35.694789    2925 checks.go:618] validating kubelet version
I0224 18:44:35.771737    2925 checks.go:128] validating if the "kubelet" service is enabled and active
I0224 18:44:35.787093    2925 checks.go:201] validating availability of port 10250
I0224 18:44:35.787264    2925 checks.go:286] validating the existence of file /etc/kubernetes/pki/ca.crt
I0224 18:44:35.787302    2925 checks.go:432] validating if the connectivity type is via proxy or direct
I0224 18:44:35.787349    2925 join.go:465] [preflight] Discovering cluster-info
I0224 18:44:35.787397    2925 token.go:78] [discovery] Created cluster-info discovery client, requesting info from "10.6.177.40:6443"
I0224 18:44:35.805992    2925 token.go:116] [discovery] Requesting info from "10.6.177.40:6443" again to validate TLS against the pinned public key
I0224 18:44:35.818685    2925 token.go:133] [discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "10.6.177.40:6443"
I0224 18:44:35.818717    2925 discovery.go:51] [discovery] Using provided TLSBootstrapToken as authentication credentials for the join process
I0224 18:44:35.818730    2925 join.go:479] [preflight] Fetching init configuration
I0224 18:44:35.818735    2925 join.go:517] [preflight] Retrieving KubeConfig objects
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
I0224 18:44:35.829741    2925 interface.go:400] Looking for default routes with IPv4 addresses
I0224 18:44:35.829763    2925 interface.go:405] Default route transits interface "ens160"
I0224 18:44:35.830025    2925 interface.go:208] Interface ens160 is up
I0224 18:44:35.830112    2925 interface.go:256] Interface "ens160" has 5 addresses :[10.6.177.91/16 fa01:150::191/64 fe80::c46f:119c:6e9c:cf7c/64 fe80::ab8a:58aa:4abb:6a59/64 fe80::d066:4846:3028:f632/64].
I0224 18:44:35.830140    2925 interface.go:223] Checking addr  10.6.177.91/16.
I0224 18:44:35.830149    2925 interface.go:230] IP found 10.6.177.91
I0224 18:44:35.830198    2925 interface.go:262] Found valid IPv4 address 10.6.177.91 for interface "ens160".
I0224 18:44:35.830208    2925 interface.go:411] Found active IP 10.6.177.91
I0224 18:44:36.557581    2925 preflight.go:101] [preflight] Running configuration dependant checks
I0224 18:44:36.557621    2925 controlplaneprepare.go:211] [download-certs] Skipping certs download
I0224 18:44:36.557649    2925 kubelet.go:110] [kubelet-start] writing bootstrap kubelet config file at /etc/kubernetes/bootstrap-kubelet.conf
I0224 18:44:36.559060    2925 kubelet.go:118] [kubelet-start] writing CA certificate at /etc/kubernetes/pki/ca.crt
I0224 18:44:36.560188    2925 kubelet.go:139] [kubelet-start] Checking for an existing Node in the cluster with name "dce-kubeadm-2" and status "Ready"
I0224 18:44:36.567741    2925 kubelet.go:153] [kubelet-start] Stopping the kubelet
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
I0224 18:44:41.835901    2925 cert_rotation.go:137] Starting client certificate rotation controller
I0224 18:44:41.838196    2925 kubelet.go:188] [kubelet-start] preserving the crisocket information for the node
I0224 18:44:41.838228    2925 patchnode.go:30] [patchnode] Uploading the CRI Socket information "/run/containerd/containerd.sock" to the Node API object "dce-kubeadm-2" as an annotation

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.