离线安装k8s 1.9.0

说明
本文参考 https://segmentfault.com/a/1190000012755243。在前 文基础上整理、增加说明，避坑。

踩过的坑： 安装k8s 1.9.0 实践：问题集锦

环境说明

环境信息（采用一个master节点+两个node节点)
192.168.1.137 tensorflow0 node
192.168.1.138 tensorflow1 master
192.168.1.139 tensorflow2 node

操作系统版本：

[root@tensorflow1 ~]# cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)

内核版本：

[root@tensorflow1 ~]# cat /proc/version
Linux version 3.10.0-693.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Tue Aug 22 21:09:27 UTC 2017

软件版本：

kubernetes v1.9.0
docker：17.03.2-ce
kubeadm:v1.9.0
kube-apiserver:v1.9.0
kube-controller-manager:v1.9.0
kube-scheduler:v1.9.0
k8s-dns-sidecar:1.14.7
k8s-dns-kube-dns:1.14.7
k8s-dns-dnsmasq-nanny:1.14.7
kube-proxy:v1.9.0
etcd:3.1.10
pause :3.0
flannel:v0.9.1
kubernetes-dashboard:v1.8.1

采用kubeadm安装
kubeadm为kubernetes官方推荐的自动化部署工具，他将kubernetes的组件以pod的形式部署在master和node节点上，并自动完成证书认证等操作。
因为kubeadm默认要从google的镜像仓库下载镜像，但目前国内无法访问google镜像仓库，所以这里我提交将镜像下好了，只需要将离线包的镜像导入到节点中就可以了。
开始安装
下载
链接: https://pan.baidu.com/s/1c2O1gIW 密码: 9s92

比对md5解压离线包

MD5 (k8s_images.tar.bz2) = b60ad6a638eda472b8ddcfa9006315ee

解压下载下来的离线包

tar -xjvf k8s_images.tar.bz2

所有节点操作

环境设置

绑定hosts
将节点ip和host写入hosts文件

[root@tensorflow1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.1.137 tensorflow0
192.168.1.138 tensorflow1
192.168.1.139 tensorflow2

关闭防火墙

systemctl stop firewalld && systemctl disable firewalld

关闭selinux

修改vi /etc/selinux/config 文件，将SELINUX改为disabled
SELINUX=disabledsetenforce 0关闭swapswapoff -a
设置永久关闭swap

修改/etc/fstab中内容，将swap那一行用#注释掉。

配置系统路由参数,防止kubeadm报路由警告

echo "
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
" >> /etc/sysctl.conf
sysctl -p

安装docker
安装docker-ce17.03 (kubeadmv1.9最大支持docker-ce17.03)

rpm -ihv docker-ce-selinux-17.03.2.ce-1.el7.centos.noarch.rpm
rpm -ivh docker-ce-17.03.2.ce-1.el7.centos.x86_64.rpm

启动docker-ce

systemctl start docker && systemctl enable docker

检查docker服务

systemctl status docker
active (running)则正常

安装k8s
导入镜像

docker load <etcd-amd64_v3.1.10.tar
docker load <flannel\:v0.9.1-amd64.tar
docker load <k8s-dns-dnsmasq-nanny-amd64_v1.14.7.tar
docker load <k8s-dns-kube-dns-amd64_1.14.7.tar
docker load <k8s-dns-sidecar-amd64_1.14.7.tar
docker load <kube-apiserver-amd64_v1.9.0.tar
docker load <kube-controller-manager-amd64_v1.9.0.tar
docker load <kube-scheduler-amd64_v1.9.0.tar
docker load <kube-proxy-amd64_v1.9.0.tar
docker load <pause-amd64_3.0.tar
docker load <kubernetes-dashboard_v1.8.1.tar

注意kubernetes-dashboard_v1.8.1.tar与其他包不在同一个目录下，在上一级目录中

安装kubelet kubeadm kubectl包

rpm -ivh socat-1.7.3.2-2.el7.x86_64.rpm
rpm -ivh kubernetes-cni-0.6.0-0.x86_64.rpm kubelet-1.9.9-9.x86_64.rpm kubectl-1.9.0-0.x86_64.rpm
rpm -ivh kubectl-1.9.0-0.x86_64.rpm
rpm -ivh kubeadm-1.9.0-0.x86_64.rpm

修改kublet配置文件

查看docker cgroup driver:

docker info|grep Cgroup

有systemd和cgroupfs两种，把kubelet service配置改成与docker一致

vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

修改 Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"。修改 systemd为cgroupfs

启动kubelet

systemctl enable kubelet && sudo systemctl start kubelet

检查kubelet服务

systemctl status kubelet

kubelet启动后 ，不停重启、ca文件不存在是正常现象，在后续步骤 kubeadm init执行后会生成ca文件，就会正常运行。

The kubelet is now restarting every few seconds, as it waits in a crashloop for kubeadm to tell it what to do. This crashloop is expected and normal, please proceed with the next step and the kubelet will start running normally.

master节点操作
开始初始化master
kubeadm init --kubernetes-version=v1.9.0 --pod-network-cidr=10.244.0.0/16
kubernetes默认支持多重网络插件如flannel、weave、calico，这里使用flanne，就必须要设置--pod-network-cidr参数，10.244.0.0/16是kube-flannel.yml里面配置的默认网段，如果需要修改的话，需要把kubeadm init的--pod-network-cidr参数和后面的kube-flannel.yml里面修改成一样的网段就可以了。

将kubeadm join xxx保存下来，等下node节点加入集群需要使用

eg:kubeadm join --token 5ce44e.47b6dc4e4b66980f 192.168.1.138:6443 --discovery-token-ca-cert-hash sha256:9d7eac82d66744405c783de5403e1f2bb7191b4c1b350d721b7b8570c62ff83a

如果忘记了，可以在master上通过kubeadmin token list得到

kubeadmin token list

默认token 24小时就会过期，后续的机器要加入集群需要重新生成token

kubeadm token create
sha256获取方式 master节点执行:
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

按照上面提示，此时root用户还不能使用kubelet控制集群需要，配置下环境变量

对于非root用户

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
对于root用户

可以直接放到~/.bash_profile

echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
source一下环境变量source ~/.bash_profilekubectl version测试kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T21:07:38Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T20:55:30Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

安装网络，可以使用flannel、calico、weave、macvlan这里我们用flannel。直接使用离线包里面的。

若要修改网段，需要kubeadm --pod-network-cidr=和这里同步

vi kube-flannel.yml修改network项"Network": "10.244.0.0/16",

执行

kubectl create -f kube-flannel.yml

node节点操作
使用刚刚kubeadm后的kubeadm join

kubeadm join --token 5ce44e.47b6dc4e4b66980f 192.168.1.138:6443 --discovery-token-ca-cert-hash sha256:9d7eac82d66744405c783de5403e1f2bb7191b4c1b350d721b7b8570c62ff83a

在master节点上确认一下

[root@tensorflow1 hadoop]# kubectl get nodes
NAME          STATUS    ROLES     AGE       VERSION
tensorflow0   Ready     <none>    1d        v1.9.0
tensorflow1   Ready     master    1d        v1.9.0
tensorflow2   Ready     <none>    1d        v1.9.0

kubernetes会在每个node节点创建flannel和kube-proxy的pod

[root@tensorflow1 hadoop]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                   READY     STATUS    RESTARTS   AGE
kube-system   etcd-tensorflow1                       1/1       Running   0          1d
kube-system   kube-apiserver-tensorflow1             1/1       Running   0          1d
kube-system   kube-controller-manager-tensorflow1    1/1       Running   0          1d
kube-system   kube-dns-6f4fd4bdf-59ttf               3/3       Running   0          1d
kube-system   kube-flannel-ds-fb75p                  1/1       Running   0          1d
kube-system   kube-flannel-ds-ppm2t                  1/1       Running   0          1d
kube-system   kube-flannel-ds-w54wh                  1/1       Running   0          1d
kube-system   kube-proxy-4lftj                       1/1       Running   0          1d
kube-system   kube-proxy-cj4st                       1/1       Running   0          1d
kube-system   kube-proxy-kd7vb                       1/1       Running   0          1d
kube-system   kube-scheduler-tensorflow1             1/1       Running   0          1d

至此kubernetes基本集群安装完成。

--后续补充部署dashboard内容--

安装nvidia-gpu组件
这里是为了给容器使用gpu，需要安装组件，否则不用安装。

编辑/etc/docker/daemon.json

cat /etc/docker/daemon.json
{"runtimes": {"nvidia": {"path": "/usr/bin/nvidia-container-runtime","runtimeArgs": []}},"default-runtime": "nvidia"
}

重启docker

systemctl restart docker

编辑kubelet配置文件

资源则需要增加一行 Environment="KUBELET_EXTRA_ARGS=--feature-gates=DevicePlugins=true"  注意要加在 ExecStart= 之前

重启kubelet

systemctl daemon-reload && systemctl restart kubelet

根据gpu型号下载相应的gpu插件镜像，挂载镜像

docker load < 
# docker images
REPOSITORY                                               TAG                 IMAGE ID            CREATED             SIZE
nvidia/k8s-device-plugin                                 1.9                 3325c3b04513        2 weeks ago         63 MB

通过yaml文件启动

kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v1.9/nvidia-device-plugin.yml
文件内容如下：
[root@tensorflow1 tf_gpu]# cat nvidia-device-plugin.yml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:name: nvidia-device-plugin-daemonsetnamespace: kube-system
spec:template:metadata:# Mark this pod as a critical add-on; when enabled, the critical add-on scheduler# reserves resources for critical add-on pods so that they can be rescheduled after# a failure.  This annotation works in tandem with the toleration below.annotations:scheduler.alpha.kubernetes.io/critical-pod: ""labels:name: nvidia-device-plugin-dsspec:tolerations:# Allow this pod to be rescheduled while the node is in "critical add-ons only" mode.# This, along with the annotation above marks this pod as a critical add-on.- key: CriticalAddonsOnlyoperator: Existscontainers:- image: nvidia/k8s-device-plugin:1.9name: nvidia-device-plugin-ctrsecurityContext:allowPrivilegeEscalation: falsecapabilities:drop: ["ALL"]volumeMounts:- name: device-pluginmountPath: /var/lib/kubelet/device-pluginsvolumes:- name: device-pluginhostPath:path: /var/lib/kubelet/device-pluginssystemctl daemon-reload && systemctl restart kubelet

本文转自CSDN-离线安装k8s 1.9.0