鲜花的主人
文章
224
分类
23
评论
25

k8s部署Rabbitmq集群

| Kubernetes • RabbitMQ •
字数总计 11359 | 阅读时长 28分钟 | 阅读量 881

rabbitmq-peer-discovery-k8s是RabbitMQ官方基于第三方开源项目rabbitmq-autocluster开发，对3.7.X版本提供的Kubernetes下的对等发现插件，可实现rabbitmq集群在k8s中的自动化部署；低于3.7.X版本请使用rabbitmq-autocluster

创建RBAC授权

[root@k8s01 rabbitmq]# cat rbac.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: rabbitmq 
  namespace: test-env 
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: endpoint-reader
  namespace: test-env
rules:
- apiGroups: [""]
  resources: ["endpoints"]
  verbs: ["get"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: endpoint-reader
  namespace: test-env
subjects:
- kind: ServiceAccount
  name: rabbitmq
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: endpoint-reader
  
[root@k8s01 rabbitmq]# kubectl apply -f rbac.yaml
serviceaccount/rabbitmq created
role.rbac.authorization.k8s.io/endpoint-reader created
rolebinding.rbac.authorization.k8s.io/endpoint-reader created

创建configmap

[root@k8s01 rabbitmq]# cat configmap.yaml 
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: rabbitmq-config
  namespace: test-env
data:
  enabled_plugins: |
      [rabbitmq_management,rabbitmq_peer_discovery_k8s].
  rabbitmq.conf: |
      ## Cluster formation. See https://blog.rabbitmq.com/cluster-formation.html to learn more.
      cluster_formation.peer_discovery_backend  = rabbit_peer_discovery_k8s
      cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
      ## Should RabbitMQ node name be computed from the pod's hostname or IP address?
      ## IP addresses are not stable, so using [stable] hostnames is recommended when possible.
      ## Set to "hostname" to use pod hostnames.
      ## When this value is changed, so should the variable used to set the RABBITMQ_NODENAME
      ## environment variable.
      cluster_formation.k8s.address_type = hostname
      ## How often should node cleanup checks run?
      cluster_formation.node_cleanup.interval = 30
      ## Set to false if automatic removal of unknown/absent nodes
      ## is desired. This can be dangerous, see
      ##  * https://blog.rabbitmq.com/cluster-formation.html#node-health-checks-and-cleanup
      ##  * https://groups.google.com/forum/#!msg/rabbitmq-users/wuOfzEywHXo/k8z_HWIkBgAJ
      cluster_formation.node_cleanup.only_log_warning = true
      cluster_partition_handling = autoheal
      ## See https://blog.rabbitmq.com/ha.html#master-migration-data-locality
      queue_master_locator=min-masters
      ## See https://blog.rabbitmq.com/access-control.html#loopback-users
      loopback_users.guest = false

      cluster_formation.randomized_startup_delay_range.min = 0
      cluster_formation.randomized_startup_delay_range.max = 2
      # 必须设置service_name，否则Pod无法正常启动，这里设置后可以不设置statefulset下env中的K8S_SERVICE_NAME变量
      cluster_formation.k8s.service_name = rabbitmq
      # 必须设置hostname_suffix，否则节点不能成为集群，同时保证namespace正确
      cluster_formation.k8s.hostname_suffix = .rabbitmq.test-env.svc.cluster.local
      # 内存上限
      vm_memory_high_watermark.absolute = 1GB
      # 硬盘上限
      disk_free_limit.absolute = 2GB

[root@k8s01 rabbitmq]# kubectl apply -f configmap.yaml
configmap/rabbitmq-config created

rabbitmq_peer_discovery_k8s参数解析：

  • cluster_formation.peer_discovery_backend：对待发现后台插件
  • cluster_formation.k8s.host：k8s服务主机(默认即可)
  • cluster_formation.k8s.address_type：基于IP/Hostname
  • cluster_formation.node_cleanup.interval：清理节点检查时间间隔

注意：官方demo示例中，cluster_formation.k8s.address_type = ip，使用的是IP模式；也就是说RabbitMQ Node的命名和访问地址是以IP地址作为区分，如rabbit@172.0.5.1，但这样的配置会产生比较大的问题，如果我们使用pv和pvc去做数据的持久化，那么每个节点的配置和数据存储都会放在rabbit@172.0.5.1这样的文件夹下，而Kubernetes集群中，Pod的IP都是不稳定的，当有RabbitMQ Node的Pod挂掉后，重新创建的Pod IP可能会变，这就会使得节点的配置和数据全部丢失。所以我们更希望RabbitMQ Node的命名是以一定规则编写的相对稳定的名称，如rabbit@rabbit-0，这就需要修改 cluster_formation.k8s.address_type = hostname，以启用hostname模式，这里headless service就派上用场了

创建headless service

[root@k8s01 rabbitmq]# cat service.yaml
---
kind: Service
apiVersion: v1
metadata:
  namespace: test-env
  name: rabbitmq
  labels:
    app: rabbitmq
spec:
  clusterIP: None
  # 由于使用DNS访问Pod需Pod和Headless service启动之后才能访问，publishNotReadyAddresses设置成true，防止readinessProbe在服务没启动时找不到DNS
  publishNotReadyAddresses: true 
  ports:
   - name: http
     protocol: TCP
     port: 15672
     targetPort: 15672
   - name: amqp
     protocol: TCP
     port: 5672
     targetPort: 5672
  selector:
    app: rabbitmq
    
[root@k8s01 rabbitmq]# kubectl apply -f service.yaml
service/rabbitmq created

创建statefulset部署

[root@k8s01 rabbitmq]# cat statefulset.yaml
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: rabbitmq
  namespace: test-env
spec:
  serviceName: rabbitmq
  replicas: 3
  template:
    metadata:
      labels:
        app: rabbitmq
    spec:
      serviceAccountName: rabbitmq
      terminationGracePeriodSeconds: 10
      containers:        
      - name: rabbitmq
        image: rabbitmq:3.7-management
        volumeMounts:
          - name: config-volume
            mountPath: /etc/rabbitmq
          - name: rabbitmq-data
            mountPath: /var/lib/rabbitmq
        ports:
          - name: http
            protocol: TCP
            containerPort: 15672
          - name: amqp
            protocol: TCP
            containerPort: 5672
        livenessProbe:
          exec:
            command: ["rabbitmqctl", "status"]
          initialDelaySeconds: 60
          # See https://blog.rabbitmq.com/monitoring.html for monitoring frequency recommendations.
          periodSeconds: 60
          timeoutSeconds: 15
        readinessProbe:
          exec:
            command: ["rabbitmqctl", "status"]
          initialDelaySeconds: 20
          periodSeconds: 60
          timeoutSeconds: 10
        imagePullPolicy: Always
        env:
          - name: HOSTNAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: RABBITMQ_USE_LONGNAME
            value: "true"
          # See a note on cluster_formation.k8s.address_type in the config file section
          - name: RABBITMQ_NODENAME
            value: "rabbit@$(HOSTNAME).rabbitmq-cluster.$(NAMESPACE).svc.cluster.local"
        #   若在ConfigMap中设置了service_name，则此处无需再次设置
        #   - name: K8S_SERVICE_NAME
        #     value: "rabbitmq-cluster"
          - name: RABBITMQ_ERLANG_COOKIE
            value: "mycookie" 
      volumes:
        - name: config-volume
          configMap:
            name: rabbitmq-config
            items:
            - key: rabbitmq.conf
              path: rabbitmq.conf
            - key: enabled_plugins
              path: enabled_plugins
  volumeClaimTemplates:
  - metadata:
      name: rabbitmq-data
      labels:
        name: rabbitmq-data
    spec:
      storageClassName: managed-nfs-storage
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 100Mi
          
[root@k8s01 rabbitmq]# kubectl apply -f statefulset.yaml
statefulset.apps/rabbitmq created

查看已部署的pod，等待所有Pod全是Running状态后，登陆到pod内部查询集群信息

[root@k8s01 ~]# kubectl get -n test-env po -owide|grep rabbit
rabbitmq-0                    1/1     Running   0          9m54s   10.244.1.98   k8s02   <none>           <none>
rabbitmq-1                    1/1     Running   0          8m53s   10.244.0.60   k8s01   <none>           <none>
rabbitmq-2                    1/1     Running   0          3m58s   10.244.2.58   k8s03   <none>           <none>
[root@k8s01 ~]# kubectl get -n test-env svc|grep rabbit
rabbitmq               ClusterIP   None             <none>        15672/TCP,5672/TCP              30m

[root@k8s01 rabbitmq]# kubectl exec -n test-env -it rabbitmq-0 bash
root@rabbitmq-0:/# rabbitmqctl cluster_status
Cluster status of node rabbit@rabbitmq-0.rabbitmq.test-env.svc.cluster.local ...
[{nodes,[{disc,['rabbit@rabbitmq-0.rabbitmq.test-env.svc.cluster.local',
                'rabbit@rabbitmq-1.rabbitmq.test-env.svc.cluster.local',
                'rabbit@rabbitmq-2.rabbitmq.test-env.svc.cluster.local']}]},
 {running_nodes,['rabbit@rabbitmq-2.rabbitmq.test-env.svc.cluster.local',
                 'rabbit@rabbitmq-1.rabbitmq.test-env.svc.cluster.local',
                 'rabbit@rabbitmq-0.rabbitmq.test-env.svc.cluster.local']},
 {cluster_name,<<"rabbit@rabbitmq-0.rabbitmq.test-env.svc.cluster.local">>},
 {partitions,[]},
 {alarms,[{'rabbit@rabbitmq-2.rabbitmq.test-env.svc.cluster.local',[]},
          {'rabbit@rabbitmq-1.rabbitmq.test-env.svc.cluster.local',[]},
          {'rabbit@rabbitmq-0.rabbitmq.test-env.svc.cluster.local',[]}]}]

添加用户修改密码

[root@k8s01 rabbitmq]# kubectl exec -n test-env -it rabbitmq-0 bash
#查看rabbitmq
root@rabbitmq-0:/# rabbitmqctl  list_users
Listing users ...
user	tags
guest	[administrator]                    
#修改rabbitmq用户密码（rabbitmqctl  change_password  Username  'Newpassword'）
root@rabbitmq-0:/# rabbitmqctl change_password guest 'guest123456'
Changing password for user "guest" ...          
#创建新用户
root@rabbitmq-0:/# rabbitmqctl add_user admin admin
Adding user "admin" ...
#授权用户超级权限
root@rabbitmq-0:/# rabbitmqctl set_user_tags admin administrator
Setting tags for user "admin" to [administrator] ...

创建ingress

[root@k8s01 ~]# cat /home/test-ing.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-ing
  namespace: test-env
  annotations:
    kubernetes.io/ingress.class: traefik            
    traefik.ingress.kubernetes.io/router.entrypoints: web
spec:
  rules:
  - host: rabbitmq.k8s.com
    http:
      paths:
      - path: /
        backend:
          serviceName: rabbitmq
          servicePort: 15672

[root@k8s01 ~]# kubectl apply -f /home/test-ing.yaml
ingress/test-ing created

本地添加hosts，登录rabbitmq.k8s.com使用初始账号密码guest/guest登录即可（如未修改密码）
mqo.png

mqp.png

节点伸缩

可以直接使用scale命令行来伸缩工作节点

[root@k8s01 ~]# kubectl scale statefulset rabbitmq --replicas=4 -n test-env
文章作者： 鲜花的主人
本文链接：
版权声明： 本站所有文章除特别声明外，均采用 CC BY-NC-SA 4.0 许可协议。转载请注明来自 爱吃可爱多！
Kubernetes RabbitMQ RabbitMQ Kubernetes
喜欢就支持一下吧
打赏
微信 微信
支付宝 支付宝