概要
kubernetes 部署连接:Installation Guide - NGINX Ingress Controller
本案例使用yaml的方式部署,分为3个步骤:
- 部署Ingress Controller
- 创建 Ingress 策略
- 客户端通过 Ingress Controller 访问后端 webapp 服务
1. 部署 Ingress Controller
ingress.yaml 内容如下:
---
apiVersion: v1
kind: Namespace
metadata:
name: nginx-ingress
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: nginx-ingress
namespace: nginx-ingress
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nginx-ingress
rules:
- apiGroups:
- ""
resources:
- services
- endpoints
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- list
- watch
- update
- create
- apiGroups:
- ""
resources:
- pods
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- list
- apiGroups:
- extensions
resources:
- ingresses
verbs:
- list
- watch
- get
- apiGroups:
- "extensions"
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- k8s.nginx.org
resources:
- virtualservers
- virtualserverroutes
- globalconfigurations
- transportservers
- policies
verbs:
- list
- watch
- get
- apiGroups:
- k8s.nginx.org
resources:
- virtualservers/status
- virtualserverroutes/status
verbs:
- update
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nginx-ingress
subjects:
- kind: ServiceAccount
name: nginx-ingress
namespace: nginx-ingress
roleRef:
kind: ClusterRole
name: nginx-ingress
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: Secret
metadata:
name: default-server-secret
namespace: nginx-ingress
type: Opaque
data:
tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN2akNDQWFZQ0NRREFPRjl0THNhWFhEQU5CZ2txaGtpRzl3MEJBUXNGQURBaE1SOHdIUVlEVlFRRERCWk8KUjBsT1dFbHVaM0psYzNORGIyNTBjbTlzYkdWeU1CNFhEVEU0TURreE1qRTRNRE16TlZvWERUSXpNRGt4TVRFNApNRE16TlZvd0lURWZNQjBHQTFVRUF3d1dUa2RKVGxoSmJtZHlaWE56UTI5dWRISnZiR3hsY2pDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUwvN2hIUEtFWGRMdjNyaUM3QlBrMTNpWkt5eTlyQ08KR2xZUXYyK2EzUDF0azIrS3YwVGF5aGRCbDRrcnNUcTZzZm8vWUk1Y2Vhbkw4WGM3U1pyQkVRYm9EN2REbWs1Qgo4eDZLS2xHWU5IWlg0Rm5UZ0VPaStlM2ptTFFxRlBSY1kzVnNPazFFeUZBL0JnWlJVbkNHZUtGeERSN0tQdGhyCmtqSXVuektURXUyaDU4Tlp0S21ScUJHdDEwcTNRYzhZT3ExM2FnbmovUWRjc0ZYYTJnMjB1K1lYZDdoZ3krZksKWk4vVUkxQUQ0YzZyM1lma1ZWUmVHd1lxQVp1WXN2V0RKbW1GNWRwdEMzN011cDBPRUxVTExSakZJOTZXNXIwSAo1TmdPc25NWFJNV1hYVlpiNWRxT3R0SmRtS3FhZ25TZ1JQQVpQN2MwQjFQU2FqYzZjNGZRVXpNQ0F3RUFBVEFOCkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQWpLb2tRdGRPcEsrTzhibWVPc3lySmdJSXJycVFVY2ZOUitjb0hZVUoKdGhrYnhITFMzR3VBTWI5dm15VExPY2xxeC9aYzJPblEwMEJCLzlTb0swcitFZ1U2UlVrRWtWcitTTFA3NTdUWgozZWI4dmdPdEduMS9ienM3bzNBaS9kclkrcUI5Q2k1S3lPc3FHTG1US2xFaUtOYkcyR1ZyTWxjS0ZYQU80YTY3Cklnc1hzYktNbTQwV1U3cG9mcGltU1ZmaXFSdkV5YmN3N0NYODF6cFErUyt1eHRYK2VBZ3V0NHh3VlI5d2IyVXYKelhuZk9HbWhWNThDd1dIQnNKa0kxNXhaa2VUWXdSN0diaEFMSkZUUkk3dkhvQXprTWIzbjAxQjQyWjNrN3RXNQpJUDFmTlpIOFUvOWxiUHNoT21FRFZkdjF5ZytVRVJxbStGSis2R0oxeFJGcGZnPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBdi91RWM4b1JkMHUvZXVJTHNFK1RYZUprckxMMnNJNGFWaEMvYjVyYy9XMlRiNHEvClJOcktGMEdYaVN1eE9ycXgrajlnamx4NXFjdnhkenRKbXNFUkJ1Z1B0ME9hVGtIekhvb3FVWmcwZGxmZ1dkT0EKUTZMNTdlT1l0Q29VOUZ4amRXdzZUVVRJVUQ4R0JsRlNjSVo0b1hFTkhzbysyR3VTTWk2Zk1wTVM3YUhudzFtMApxWkdvRWEzWFNyZEJ6eGc2clhkcUNlUDlCMXl3VmRyYURiUzc1aGQzdUdETDU4cGszOVFqVUFQaHpxdmRoK1JWClZGNGJCaW9CbTVpeTlZTW1hWVhsMm0wTGZzeTZuUTRRdFFzdEdNVWozcGJtdlFmazJBNnljeGRFeFpkZFZsdmwKMm82MjBsMllxcHFDZEtCRThCay90elFIVTlKcU56cHpoOUJUTXdJREFRQUJBb0lCQVFDZklHbXowOHhRVmorNwpLZnZJUXQwQ0YzR2MxNld6eDhVNml4MHg4Mm15d1kxUUNlL3BzWE9LZlRxT1h1SENyUlp5TnUvZ2IvUUQ4bUFOCmxOMjRZTWl0TWRJODg5TEZoTkp3QU5OODJDeTczckM5bzVvUDlkazAvYzRIbjAzSkVYNzZ5QjgzQm9rR1FvYksKMjhMNk0rdHUzUmFqNjd6Vmc2d2szaEhrU0pXSzBwV1YrSjdrUkRWYmhDYUZhNk5nMUZNRWxhTlozVDhhUUtyQgpDUDNDeEFTdjYxWTk5TEI4KzNXWVFIK3NYaTVGM01pYVNBZ1BkQUk3WEh1dXFET1lvMU5PL0JoSGt1aVg2QnRtCnorNTZud2pZMy8yUytSRmNBc3JMTnIwMDJZZi9oY0IraVlDNzVWYmcydVd6WTY3TWdOTGQ5VW9RU3BDRkYrVm4KM0cyUnhybnhBb0dCQU40U3M0ZVlPU2huMVpQQjdhTUZsY0k2RHR2S2ErTGZTTXFyY2pOZjJlSEpZNnhubmxKdgpGenpGL2RiVWVTbWxSekR0WkdlcXZXaHFISy9iTjIyeWJhOU1WMDlRQ0JFTk5jNmtWajJTVHpUWkJVbEx4QzYrCk93Z0wyZHhKendWelU0VC84ajdHalRUN05BZVpFS2FvRHFyRG5BYWkyaW5oZU1JVWZHRXFGKzJyQW9HQkFOMVAKK0tZL0lsS3RWRzRKSklQNzBjUis3RmpyeXJpY05iWCtQVzUvOXFHaWxnY2grZ3l4b25BWlBpd2NpeDN3QVpGdwpaZC96ZFB2aTBkWEppc1BSZjRMazg5b2pCUmpiRmRmc2l5UmJYbyt3TFU4NUhRU2NGMnN5aUFPaTVBRHdVU0FkCm45YWFweUNweEFkREtERHdObit3ZFhtaTZ0OHRpSFRkK3RoVDhkaVpBb0dCQUt6Wis1bG9OOTBtYlF4VVh5YUwKMjFSUm9tMGJjcndsTmVCaWNFSmlzaEhYa2xpSVVxZ3hSZklNM2hhUVRUcklKZENFaHFsV01aV0xPb2I2NTNyZgo3aFlMSXM1ZUtka3o0aFRVdnpldm9TMHVXcm9CV2xOVHlGanIrSWhKZnZUc0hpOGdsU3FkbXgySkJhZUFVWUNXCndNdlQ4NmNLclNyNkQrZG8wS05FZzFsL0FvR0FlMkFVdHVFbFNqLzBmRzgrV3hHc1RFV1JqclRNUzRSUjhRWXQKeXdjdFA4aDZxTGxKUTRCWGxQU05rMXZLTmtOUkxIb2pZT2pCQTViYjhibXNVU1BlV09NNENoaFJ4QnlHbmR2eAphYkJDRkFwY0IvbEg4d1R0alVZYlN5T294ZGt5OEp0ek90ajJhS0FiZHd6NlArWDZDODhjZmxYVFo5MWpYL3RMCjF3TmRKS2tDZ1lCbyt0UzB5TzJ2SWFmK2UwSkN5TGhzVDQ5cTN3Zis2QWVqWGx2WDJ1VnRYejN5QTZnbXo5aCsKcDNlK2JMRUxwb3B0WFhNdUFRR0xhUkcrYlNNcjR5dERYbE5ZSndUeThXczNKY3dlSTdqZVp2b0ZpbmNvVlVIMwphdmxoTUVCRGYxSjltSDB5cDBwWUNaS2ROdHNvZEZtQktzVEtQMjJhTmtsVVhCS3gyZzR6cFE9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
---
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-config
namespace: nginx-ingress
data:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-ingress
namespace: nginx-ingress
spec:
replicas: 1
selector:
matchLabels:
app: nginx-ingress
template:
metadata:
labels:
app: nginx-ingress
spec:
nodeSelector:
role: ingress-nginx-controller
serviceAccountName: nginx-ingress
containers:
- image: nginx/nginx-ingress:1.7.2
imagePullPolicy: IfNotPresent
name: nginx-ingress
ports:
- name: http
containerPort: 80
hostPort: 80
- name: https
containerPort: 443
hostPort: 443
securityContext:
allowPrivilegeEscalation: true
runAsUser: 101 #nginx
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
args:
- -nginx-configmaps=$(POD_NAMESPACE)/nginx-config
- -default-server-tls-secret=$(POD_NAMESPACE)/default-server-secret
前提条件:需要在部署的 node 上打标签
[root@master1 ~]# kubectl label node node2 role=ingress-nginx-controller
node/node2 labeled
不然部署的时候,会报如下事件:
[root@master1 ~]# kubectl describe pods --namespace=nginx-ingress nginx-ingress-75c88594dc-j687p
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 11m default-scheduler 0/4 nodes are available: 4 node(s) didn't match node selector.
Warning FailedScheduling 11m default-scheduler 0/4 nodes are available: 4 node(s) didn't match node selector.`
通过 kubectl create 命令创建 nginx-ingres-controller:
[root@master1 ~]# kubectl apply -f ingress.yaml
namespace/nginx-ingress created
serviceaccount/nginx-ingress created
clusterrole.rbac.authorization.k8s.io/nginx-ingress created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress created
secret/default-server-secret created
configmap/nginx-config created
deployment.apps/nginx-ingress created
ingress.networking.k8s.io/mywebsite-ingress created
查看 nginx-ingress-controller 容器,确认其正常运行:
[root@master1 ~]# kubectl get pods -n nginx-ingress -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-ingress-75c88594dc-42256 1/1 Running 0 2m48s 10.0.104.42 node2 <none> <none>
用 curl 访问 Nginx Ingress Controller 所在的宿主机的 80 端口,验证其服务是否正常,在没有配置后端服务时 Nginx 会返回 404 应答:
[root@master1 ~]# curl http://node2
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.19.0</center>
</body>
</html>
2. 创建 Ingress 策略
本例对域名 mywebsite.com 的访问设置 Ingress 策略,定义对其 /demo 路径的访问转发到后端 myweb Service 的规则:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: mywebsite-ingress
spec:
rules:
- host: mywebsite.com
http:
paths:
- path: /demo
pathType: ImplementationSpecific
backend:
service:
name: myweb
port:
number: 8080
通过该 Ingress 定义设置的效果:客户端对目标地址 http://mywebsite.com/demo 的访问将被转发到集群内的服务“myweb”上,完整的URL为:“http://myweb:8080/demo”。
在 Ingress 策略生效之前,需要先确保 webapp 服务正确运行。同时注意 Ingress 中对路径的定义需要与后端 myweb 服务提供的访问路径一直,否则将被转发到一个不存在的路径上引发错误。
创建两个测试yaml文件,分别是 mysql.yaml 和 myweb.yaml
mysql.yaml 内容如下:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: mysql
name: mysql
spec:
replicas: 1
selector:
matchLabels:
app: mysql
template:
metadata:
labels:
app: mysql
spec:
hostname: mysql
subdomain: database
containers:
- image: mysql:5.7
name: mysql
ports:
- containerPort: 3306
env:
- name: MYSQL_ROOT_PASSWORD
value: "123456"
---
apiVersion: v1
kind: Service
metadata:
name: database
spec:
ports:
- port: 3306
selector:
app: mysql
myweb.yaml 内容如下:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: myweb
name: myweb
spec:
replicas: 2
selector:
matchLabels:
app: myweb
template:
metadata:
labels:
app: myweb
spec:
containers:
- image: kubeguide/tomcat-app:v1
name: myweb
ports:
- containerPort: 8080
env:
- name: MYSQL_SERVICE_HOST
value: mysql.database.default.svc.cluster.local
---
apiVersion: v1
kind: Service
metadata:
name: myweb
spec:
type: NodePort
ports:
- port: 8080
nodePort: 8080
selector:
app: myweb
创建两个资源,并查看mysql的IP是 10.0.104.5,两个web的 IP,是 10.0.166.184 和 10.0.104.2。
[root@master1 ~]# kubectl apply -f mysql.yaml
deployment.apps/mysql created
service/database created
[root@master1 ~]# kubectl apply -f myweb.yaml
deployment.apps/myweb created
service/myweb created
[root@master1 ~]# kubectl get pods -l app=mysql -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
mysql-6b479bcd9d-qtr8w 1/1 Running 0 3m17s 10.0.104.5 node2 <none> <none>
[root@master1 ~]#
[root@master1 ~]# kubectl get pods -l app=myweb -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
myweb-554c69c9bd-pzn4n 1/1 Running 0 3m22s 10.0.166.184 node1 <none> <none>
myweb-554c69c9bd-q8vrg 1/1 Running 0 3m21s 10.0.104.2 node2 <none> <none>
创建ingress对象:
[root@master1 ~]# kubectl apply -f mywebsite-ingress.yaml
ingress.networking.k8s.io/mywebsite-ingress created
[root@master1 ~]#
[root@master1 ~]# kubectl get ingress
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
mywebsite-ingress <none> mywebsite.com 80 5s
一旦 Ingress 资源成功创建,Ingress Controller 就会监控到其配置的路由策略,并更新到 Nginx 的配置文件中生效。以本例中的 Nginx Controller 为例,它将更新其配置文件的内容为在 Ingress 中设定的路由策略。
登录一个 nginx-ingress-controller Pod,在/etc/nginx/conf.d 目录下可以看到 Nginx IngressController 自动生成的配置文件default-mywebsite-ingress.conf,查看其内容,可以看到对 mywebsite.com/demo 的转发规则的正确配置:
[root@master1 ~]# kubectl exec -it --namespace nginx-ingress nginx-ingress-75c88594dc-42256 -- bash
nginx@nginx-ingress-75c88594dc-42256:/$ ls /etc/nginx/conf.d/
default-mywebsite-ingress.conf
nginx@nginx-ingress-75c88594dc-42256:/$ cat /etc/nginx/conf.d/default-mywebsite-ingress.conf
# configuration for default/mywebsite-ingress
upstream default-mywebsite-ingress-mywebsite.com-myweb-8080 {
zone default-mywebsite-ingress-mywebsite.com-myweb-8080 256k;
random two least_conn;
server 10.0.104.2:8080 max_fails=1 fail_timeout=10s max_conns=0;
server 10.0.166.184:8080 max_fails=1 fail_timeout=10s max_conns=0;
}
server {
listen 80;
server_tokens on;
server_name mywebsite.com;
location /demo {
proxy_http_version 1.1;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 1m;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering on;
proxy_pass http://default-mywebsite-ingress-mywebsite.com-myweb-8080;
}
}
发现 nginx 监听 mywebsite.com 的 80 端口,并转发到 upstream 里的 10.0.166.184:8080 和 10.0.104.2:8080。
3. 客户端通过 Ingress Controller 访问后端 webapp 服务
由于 Ingress Controller 容器通过 hostPort 将服务端口号 80 映射到了宿主机上,所以客户端可以通过 Ingress Controller 所在的 Node 访问 Mywebsite.com 提供的服务。
需要注意的是,客户端只能通过域名 mywebsite.com 访问服务,这时要求客户端或者 DNS 将 mywebsite.com 域名解析到 Node 的正式 IP 地址上。刚刚上面执行kubectl get pods -n nginx-ingress -o wide
的时候看到,nginx-ingress是在node2上,node2的正式IP是:192.168.100.202。
[root@master1 4.6.1]# curl --resolve mywebsite.com:80:192.168.100.202 http://mywebsite.com/demo/
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>HPE University Docker&Kubernetes Learning</title>
</head>
<body align="center">
<h2>Congratulations!!</h2>
<br></br>
<input type="button" value="Add..." onclick="location.href='input.html'" >
<br></br>
<TABLE align="center" border="1" width="600px">
<TR>
<TD>Name</TD>
<TD>Level(Score)</TD>
</TR>
<TR>
<TD>google</TD>
<TD>100</TD>
</TR>
...
或在windows修改hosts文件后使用浏览器访问
参考《Kubernetes权威指南(第五版)》