Kubernetes - troubleshooting and commands

Posted on Sat 29 October 2016 by Pavlo Khmel

This post is related to the previous one: Kubernetes – simple install on CentOS 7 with Web UI

  • How to reset kubernetes configuration and setup
  • Change service IP subnet
  • Useful commands
  • Logs

0. How to reset kubernetes configuration and setup

When you start experimenting with kubernetes it is useful to reset all and start from scratch. During "reset" etcd and flanneld should be running and configured correctly because docker is in dependency.

Stop kubernetes on all servers, delete related settings, restore kubelet, delete containers and images:

systemctl stop kube-apiserver kube-controller-manager kube-scheduler kube-proxy kubelet
rm -rf /var/lib/kubelet
rm -rf /var/run/kubernetes
systemctl restart flanneld docker
# wait for message: Recovery completed 
# and interrupt with Crtl+C
docker rm $(docker ps -aq)
docker rmi $(docker images -q)

On master clean etcd:

systemctl start etcd
etcdctl ls
etcdctl ls | xargs -I {} etcdctl rm {} --recursive

Disable all services on all nodes:

systemctl disable kube-apiserver kube-controller-manager kube-scheduler kube-proxy kubelet flanneld docker etc

Now you can reboot all nodes and configure kubernetes from scratch.

1. Change service subnet

If you changed service subnet you'll need to recreate service kubernetes by deleting:

[root@kube-master ~]# kubectl get services -a --all-namespaces
NAMESPACE     NAME                    CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
default       kubernetes         <none>        443/TCP   51m

[root@kube-master ~]# kubectl delete service kubernetes

Service kubernetes will be automatically recreated.

2. Useful commands

It is useful to use additional options with kubectl get commands:

 -a               # When printing, show all resources (default hide terminated pods.)
 -o wide          # Output format
 --all-namespaces # If present, list the requested object(s) across all namespaces.

[root@kube-master ~]# kubectl get pods -a -o wide --all-namespaces
NAMESPACE     NAME                                     READY     STATUS             RESTARTS   AGE       NODE
default       busybox                                  0/1       Completed          0          13h       kube-minion-1
default       busybox1                                 0/1       Completed          0          12h       kube-minion-3
default       busybox2                                 0/1       Error              0          12h       kube-minion-2
default       nginx                                    1/1       Running            0          14h       kube-minion-3
default       webserver-controller-hm353               0/1       ImagePullBackOff   0          14h       kube-minion-2
kube-system   kubernetes-dashboard-2091788654-yucft    0/1       CrashLoopBackOff   171        14h       kube-minion-3

[root@kube-master ~]# kubectl get deployments -a -o wide --all-namespaces
kube-system   kubernetes-dashboard    1         1         1            0           14h

[root@kube-master ~]# kubectl get services -a -o wide --all-namespaces
default       kubernetes          <none>         443/TCP   14h       <none>
kube-system   kubernetes-dashboard2   80/TCP    1h        app=kubernetes-dashboard2

Status check:

[root@kube-master ~]# kubectl get componentstatuses
NAME                 STATUS    MESSAGE              ERROR
scheduler            Healthy   ok                   
controller-manager   Healthy   ok                   
etcd-0               Healthy   {"health": "true"}

[root@kube-master ~]# kubectl get namespaces
NAME          STATUS    AGE
default       Active    15d
kube-system   Active    14h

# kubectl cluster-info
Kubernetes master is running at http://localhost:8080

Deleting. You need to specify namespace when you delete items if you are not deleting from default namespace:

kubectl delete pod kubernetes-dashboard-29963172-ncipu --namespace=kube-system
kubectl delete deployment kubernetes-dashboard --namespace=kube-system
kubectl delete service kubernetes-dashboard --namespace=kube-system

3. Logs

All services are writing logs to /var/log/messages

To get logs from pods, get pod name:

[root@kube-master ~]# kubectl get pods -a -o wide --all-namespaces
NAMESPACE     NAME                                  READY     STATUS    RESTARTS   AGE       NODE
default       nginx                                 1/1       Running   0          16h       kube-minion-3
kube-system   kubernetes-dashboard-29963172-hsjrd   1/1       Running   0          2d        kube-minion-2

Show logs:

[root@kube-master ~]# kubectl logs kubernetes-dashboard-29963172-hsjrd --namespace=kube-system
Starting HTTP server on port 9090
Creating API server client for
Successful initial request to the apiserver, version: v1.2.0
Creating in-cluster Heapster client
Getting application global configuration
Application configuration {"serverTime":1477567294910}
[2016-10-27T11:21:36Z] Incoming HTTP/1.1 GET /api/v1/workload/default?itemsPerPage=10&page=1 request from
Getting lists of all workloads
Getting pod metrics
[2016-10-27T11:21:36Z] Outcoming response to with 201 status code