K8S部署RocketMQ集群（双主双从+同步模式） |
您所在的位置：网站首页 › rocketmq端口配置 › K8S部署RocketMQ集群（双主双从+同步模式） |
K8S部署RocketMQ集群（双主双从+同步模式）
|
broker-a 主1
broker-b 主2
broker-a-s 从1
broker-a-s 从2
NameServer（注册中心）
rocketmq-externals（可视化web界面）
二、部署过程记录 1、使用NFS配置StatefulSet的动态持久化存储StatefulSet使用volumeClaimTemplates 1.1）在NFS服务器端（172.16.60.238）通过nfs创建rocketmq的共享目录 [root@k8s-harbor01 ~]# mkdir -p /data/storage/k8s/rocketmq 使用NFS作为StatefulSet持久化存储的操作记录，分别需要创建nfs-provisioner的rbac、storageclass、nfs-client-provisioner和statefulset的pod。 1.2）创建nfs的rbac [root@k8s-master01 ~]# mkdir -p /opt/k8s/k8s_project/rocketmq [root@k8s-master01 ~]# cd /opt/k8s/k8s_project/rocketmq [root@k8s-master01 rocketmq]# [root@k8s-master01 rocketmq]# vim nfs-rbac.yaml --- apiVersion: v1 kind: ServiceAccount metadata: name: nfs-provisioner namespace: wiseco --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: nfs-provisioner-runner namespace: wiseco rules: - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "create", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["events"] verbs: ["watch", "create", "update", "patch"] - apiGroups: [""] resources: ["services", "endpoints"] verbs: ["get","create","list", "watch","update"] - apiGroups: ["extensions"] resources: ["podsecuritypolicies"] resourceNames: ["nfs-provisioner"] verbs: ["use"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: run-nfs-provisioner subjects: - kind: ServiceAccount name: nfs-provisioner namespace: wiseco roleRef: kind: ClusterRole name: nfs-provisioner-runner apiGroup: rbac.authorization.k8s.io创建并查看： [root@k8s-master01 rocketmq]# kubectl apply -f nfs-rbac.yaml serviceaccount/nfs-provisioner created clusterrole.rbac.authorization.k8s.io/nfs-provisioner-runner created clusterrolebinding.rbac.authorization.k8s.io/run-nfs-provisioner created [root@k8s-master01 rocketmq]# kubectl get sa -n wiseco|grep nfs nfs-provisioner 1 5s [root@k8s-master01 rocketmq]# kubectl get clusterrole -n wiseco|grep nfs nfs-provisioner-runner 2021-01-28T08:15:49Z [root@k8s-master01 rocketmq]# kubectl get clusterrolebinding -n wiseco|grep nfs run-nfs-provisioner ClusterRole/nfs-provisioner-runner
1.3）创建rocketmq集群的storageclass [root@k8s-master01 rocketmq]# vim rocketmq-nfs-class.yaml apiVersion: storage.k8s.io/v1beta1 kind: StorageClass metadata: name: rocketmq-nfs-storage namespace: wiseco provisioner: rocketmq/nfs reclaimPolicy: Retain
查看并创建 [root@k8s-master01 rocketmq]# kubectl apply -f rocketmq-nfs-class.yaml storageclass.storage.k8s.io/rocketmq-nfs-storage created [root@k8s-master01 rocketmq]# kubectl get sc -n wiseco NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE rocketmq-nfs-storage rocketmq/nfs Retain Immediate false 2s
1.4）创建rocketmq集群的nfs-client-provisionerPROVISIONER_NAME的值一定要和StorageClass中的provisioner相等。 [root@k8s-master01 rocketmq]# ll total 8 -rw-r--r-- 1 root root 1216 Jan 28 16:15 nfs-rbac.yaml -rw-r--r-- 1 root root 161 Jan 28 16:19 rocketmq-nfs-class.yaml [root@k8s-master01 rocketmq]# vim rocketmq-nfs.yml apiVersion: apps/v1 kind: Deployment metadata: name: rocketmq-nfs-client-provisioner namespace: wiseco spec: replicas: 1 selector: matchLabels: app: rocketmq-nfs-client-provisioner strategy: type: Recreate template: metadata: labels: app: rocketmq-nfs-client-provisioner spec: serviceAccount: nfs-provisioner containers: - name: rocketmq-nfs-client-provisioner image: registry.cn-hangzhou.aliyuncs.com/open-ali/nfs-client-provisioner imagePullPolicy: IfNotPresent volumeMounts: - name: nfs-client-root mountPath: /persistentvolumes env: - name: PROVISIONER_NAME value: rocketmq/nfs - name: NFS_SERVER value: 172.16.60.238 - name: NFS_PATH value: /data/storage/k8s/rocketmq volumes: - name: nfs-client-root nfs: server: 172.16.60.238 path: /data/storage/k8s/rocketmq
查看并创建 [root@k8s-master01 rocketmq]# kubectl apply -f rocketmq-nfs.yml deployment.apps/rocketmq-nfs-client-provisioner created [root@k8s-master01 rocketmq]# kubectl get pods -n wiseco|grep nfs rocketmq-nfs-client-provisioner-59d8bbd947-kz59r 1/1 Running 0 4s
2、创建RocketMQ集群（双主双从同步模式） 本案例采用jdk1.8.0_192的基础镜像 [root@k8s-master01 rocketmq]# ll total 12 -rw-r--r-- 1 root root 1216 Feb 1 11:11 nfs-rbac.yaml -rw-r--r-- 1 root root 161 Feb 1 11:12 rocketmq-nfs-class.yaml -rw-r--r-- 1 root root 1027 Feb 1 11:12 rocketmq-nfs.yml [root@k8s-master01 rocketmq]# mkdir image [root@k8s-master01 rocketmq]# cd image [root@k8s-master01 image]# mkdir rocketmq_image [root@k8s-master01 image]# cd rocketmq_image [root@k8s-master01 rocketmq_image]#
2.1）制作rocketmq的image镜像rocketmq下载地址：https://mirrors.tuna.tsinghua.edu.cn/apache/rocketmq/ 需要注意一点：pod可能会启动报错：/usr/local/rocketmq-4.8.0/bin/runbroker.sh: line 90: 18 Killed $JAVA ${JAVA_OPT} $@原因分析：/usr/local/rocketmq-4.8.0/bin/runbroker.sh脚本文件里的内存设置的太大, pod因内存不足导致启动失败！解决办法：将runbroker.sh脚本文件里的内存设置调整小点： 将 JAVA_OPT="${JAVA_OPT} -server -Xms8g -Xmx8g -Xmn4g" 改为 JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m"rocketmq镜像制作过程： [root@k8s-master01 rocketmq_image]# wget https://mirrors.tuna.tsinghua.edu.cn/apache/rocketmq/4.8.0/rocketmq-all-4.8.0-bin-release.zip [root@k8s-master01 rocketmq_image]# ll total 13564 -rw-r--r-- 1 root root 339 Feb 2 11:52 Dockerfile -rw-r--r-- 1 root root 13881969 Dec 19 19:54 rocketmq-all-4.8.0-bin-release.zip 由于ADD指令只能解压tar包，所以这里将zip包调整为tar包 [root@k8s-master01 rocketmq_image]# unzip rocketmq-all-4.8.0-bin-release.zip [root@k8s-master01 rocketmq_image]# ll total 13564 -rw-r--r-- 1 root root 339 Feb 2 11:52 Dockerfile drwxr-xr-x 6 root root 103 Dec 9 19:46 rocketmq-all-4.8.0-bin-release -rw-r--r-- 1 root root 13881969 Dec 19 19:54 rocketmq-all-4.8.0-bin-release.zip 将runbroker.sh脚本文件里的内存设置调整小点 [root@k8s-master01 rocketmq_image]# vim rocketmq-all-4.8.0-bin-release/bin/runbroker.sh ......... ......... JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m" ......... ......... [root@k8s-master01 rocketmq_image]# tar -zvcf rocketmq-all-4.8.0-bin-release.tar.gz rocketmq-all-4.8.0-bin-release [root@k8s-master01 rocketmq_image]# rm -rf rocketmq-all-4.8.0-bin-release rocketmq-all-4.8.0-bin-release.zip [root@k8s-master01 rocketmq_image]# ll total 13508 -rw-r--r-- 1 root root 278 Feb 2 10:38 Dockerfile -rw-r--r-- 1 root root 13827784 Feb 2 11:27 rocketmq-all-4.8.0-bin-release.tar.gz Dockerfile镜像文件内容： [root@k8s-master01 rocketmq_image]# cat Dockerfile FROM 172.16.60.238/wiseco/jdk1.8.0_192 RUN rm -f /etc/localtime \ && ln -sv /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \ && echo "Asia/Shanghai" > /etc/timezone ENV LANG en_US.UTF-8 ADD rocketmq-all-4.8.0-bin-release.tar.gz /usr/local/ RUN mv /usr/local/rocketmq-all-4.8.0-bin-release /usr/local/rocketmq-4.8.0 \ && mkdir -p /data/rocketmq/store CMD ["/bin/bash"] 制作镜像并上传到Harbor仓库 [root@k8s-master01 images]# docker build -t 172.16.60.238/wiseco/rocketmq-4.8.0 . [root@k8s-master01 images]# docker push 172.16.60.238/wiseco/rocketmq-4.8.0
2.2）制作nameserver的image镜像 [root@k8s-master01 image]# pwd /opt/k8s/k8s_project/rocketmq/image [root@k8s-master01 image]# ll total 0 drwxr-xr-x 2 root root 69 Feb 2 16:54 rocketmq_image [root@k8s-master01 image]# mkdir nameserver_image [root@k8s-master01 image]# cd nameserver_image/ 将上面rocketmq_image目录下重新打好的包rocketmq-all-4.8.0-bin-release.tar.gz 拷贝过来 [root@k8s-master01 nameserver_image]# cp ../rocketmq_image/rocketmq-all-4.8.0-bin-release.tar.gz ./ [root@k8s-master01 nameserver_image]# ll total 13512 -rw-r--r-- 1 root root 516 Feb 2 23:45 Dockerfile -rw-r--r-- 1 root root 17 Feb 2 23:35 namesrv.properties -rw-r--r-- 1 root root 13828045 Feb 2 23:34 rocketmq-all-4.8.0-bin-release.tar.gz 自定义nameserver的端口号 [root@k8s-master01 nameserver_image]# cat namesrv.properties listenPort=20901 Dockerfile镜像文件内容： [root@k8s-master01 nameserver_image]# cat Dockerfile FROM 172.16.60.238/wiseco/jdk1.8.0_192 RUN rm -f /etc/localtime \ && ln -sv /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \ && echo "Asia/Shanghai" > /etc/timezone ENV LANG en_US.UTF-8 ADD rocketmq-all-4.8.0-bin-release.tar.gz /usr/local/ RUN mv /usr/local/rocketmq-all-4.8.0-bin-release /usr/local/rocketmq-4.8.0 COPY namesrv.properties /usr/local/rocketmq-4.8.0/conf/ WORKDIR /usr/local/rocketmq-4.8.0 CMD ["/usr/local/rocketmq-4.8.0/bin/mqnamesrv","-c","/usr/local/rocketmq-4.8.0/conf/namesrv.properties"]制作镜像并上传到Harbor仓库 [root@k8s-master01 nameserver_image]# docker build -t 172.16.60.238/wiseco/rocketmq_namesrv:v1 . [root@k8s-master01 nameserver_image]# docker push 172.16.60.238/wiseco/rocketmq_namesrv:v1
2.3）制作rocketmq的web可视化界面的image镜像下载地址：https://pan.baidu.com/s/19PROiNivWBiOIV5NkOBBqA提取密码：r1cf [root@k8s-master01 image]# pwd /opt/k8s/k8s_project/rocketmq/image [root@k8s-master01 image]# ll total 0 drwxr-xr-x 2 root root 55 Feb 2 15:54 nameserver_image drwxr-xr-x 2 root root 69 Feb 2 16:54 rocketmq_image [root@k8s-master01 image]# mkdir rocketmq-externals_image [root@k8s-master01 image]# cd rocketmq-externals_image/ [root@k8s-master01 rocketmq-externals_image]# 使用上面的百度云盘地址下载rocketmq-externals.tar.gz包 或是通过下面的url地址下载rocketmq-externals [root@k8s-master01 rocketmq-externals_image]# git clone https://github.com/apache/rocketmq-externals.git 下载maven通过上面的百度云盘地址下载rocketmq-externals.tar.gz包 [root@k8s-master01 rocketmq-externals_image]# wget http://mirrors.hust.edu.cn/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz 这里选择 [root@k8s-master01 rocketmq-externals_image]# ll total 29364 -rw-r--r-- 1 root root 8491533 Jul 3 2020 apache-maven-3.3.9-bin.tar.gz -rw-r--r-- 1 root root 383 Feb 2 14:18 Dockerfile -rw-r--r-- 1 root root 21562027 Feb 2 14:18 rocketmq-externals.tar.gz [root@k8s-master01 rocketmq-externals_image]# tar -zvxf rocketmq-externals.tar.gz [root@k8s-master01 rocketmq-externals_image]# vim rocketmq-externals/rocketmq-console/src/main/resources/application.properties server.contextPath= server.port=8080 #默认访问端口是8080 #spring.application.index=true spring.application.name=rocketmq-console spring.http.encoding. spring.http.encoding.enabled=true spring.http.encoding.force=true logging.config=classpath:logback.xml #if this value is empty,use env value rocketmq.config.namesrvAddr NAMESRV_ADDR | now, you can set it in ops page.default localhost:9876 rocketmq.config.namesrvAddr=mq-namesrv-0.mq-namesrv.wiseco.svc.cluster.local:20901 #如果nameserver是多台集群机器，则后面就配置多个ip+port，即"rocketmq.config.namesrvAddr=ip1:port;ip2:port" #if you use rocketmq version < 3.5.8, rocketmq.config.isVIPChannel should be false.default true rocketmq.config.isVIPChannel=true #注意这个参数，是否设置为false或true取决于rocketmq的版本号。rocketmq低于3.5.8版本,设置为false，默认为true！ #rocketmq-console's data path:dashboard/monitor rocketmq.config.dataPath=/tmp/rocketmq-console/data #set it false if you don't want use dashboard.default true rocketmq.config.enableDashBoardCollect=true 删除之前的包，重新打包 [root@k8s-master01 rocketmq-externals_image]# rm -rf rocketmq-externals.tar.gz [root@k8s-master01 rocketmq-externals_image]# tar -zvcf rocketmq-externals.tar.gz rocketmq-externals [root@k8s-master01 rocketmq-externals_image]# rm -rf rocketmq-externals [root@k8s-master01 rocketmq-externals_image]# ll total 29360 -rw-r--r-- 1 root root 8491533 Jul 3 2020 apache-maven-3.3.9-bin.tar.gz -rw-r--r-- 1 root root 383 Feb 2 14:18 Dockerfile -rw-r--r-- 1 root root 21562566 Feb 2 14:24 rocketmq-externals.tar.gz Dockerfile镜像文件内容： [root@k8s-master01 rocketmq-externals_image]# cat Dockerfile FROM 172.16.60.238/wiseco/jdk1.8.0_192 RUN rm -f /etc/localtime \ && ln -sv /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \ && echo "Asia/Shanghai" > /etc/timezone ENV LANG en_US.UTF-8 ENV MAVEN_HOME /usr/local/maven ENV PATH $PATH:$MAVEN_HOME/bin ADD rocketmq-externals.tar.gz /usr/local ADD apache-maven-3.3.9-bin.tar.gz /usr/local RUN mv /usr/local/apache-maven-3.3.9 /usr/local/maven WORKDIR /usr/local/rocketmq-externals/rocketmq-console/ RUN mvn clean package -Dmaven.test.skip=true WORKDIR /usr/local/rocketmq-externals/rocketmq-console/target EXPOSE 8080 CMD ["nohup","java","-jar","rocketmq-console-ng-1.0.0.jar","&"] 制作镜像并上传到Harbor仓库 [root@k8s-master01 rocketmq-externals_image]# docker build -t 172.16.60.238/wiseco/rocketmq-externals:v1 . [root@k8s-master01 rocketmq-externals_image]# docker push 172.16.60.238/wiseco/rocketmq-externals:v1
3、准备配置文件，创建configmapRocketMQ默认提供的配置文件都是最基本的，很多配置都是默认值，在生产环境中我们需要根据实际情况进行修改。本案例中使用的Pod的namespace命名空间是 wiseco。broker的端口是20911、nameserver的端口是20901 [root@k8s-master01 rocketmq]# ll total 12 drwxr-xr-x 2 root root 24 Jan 28 18:04 images -rw-r--r-- 1 root root 1216 Jan 28 16:15 nfs-rbac.yaml -rw-r--r-- 1 root root 161 Jan 28 16:19 rocketmq-nfs-class.yaml -rw-r--r-- 1 root root 1027 Jan 28 16:32 rocketmq-nfs.yml 创建config目录 [root@k8s-master01 rocketmq]# mkdir config [root@k8s-master01 rocketmq]# cd config 在config目录下编辑： RocketMQ"双主双从集群模式"的四个配置文件。 以及nameserver配置文件 [root@k8s-master01 config]# ll total 24 -rw-r--r-- 1 root root 502 Feb 1 14:37 broker-a.properties -rw-r--r-- 1 root root 496 Feb 1 14:37 broker-a-s.properties -rw-r--r-- 1 root root 502 Feb 1 14:37 broker-b.properties -rw-r--r-- 1 root root 496 Feb 1 14:38 broker-b-s.propertiesbroker-a.properties 配置文件： [root@k8s-master01 config]# cat broker-a.properties brokerClusterName=rocketmq-cluster brokerName=broker-a brokerId=0 namesrvAddr=mq-namesrv-0.mq-namesrv.wiseco.svc.cluster.local:20901 defaultTopicQueueNums=4 autoCreateTopicEnable=true autoCreateSubscriptionGroup=true listenPort=20911 deleteWhen=04 fileReservedTime=120 mapedFileSizeCommitLog=1073741824 mapedFileSizeConsumeQueue=300000 diskMaxUsedSpaceRatio=88 storePathRootDir=/data/rocketmq/store maxMessageSize=65536 brokerRole=MASTER broker-a-s.properties 配置文件： [root@k8s-master01 config]# cat broker-a-s.properties brokerClusterName=rocketmq-cluster brokerName=broker-a brokerId=1 namesrvAddr=mq-namesrv-0.mq-namesrv.wiseco.svc.cluster.local:20901 defaultTopicQueueNums=4 autoCreateTopicEnable=true autoCreateSubscriptionGroup=true listenPort=20911 deleteWhen=04 fileReservedTime=120 mapedFileSizeCommitLog=1073741824 mapedFileSizeConsumeQueue=300000 diskMaxUsedSpaceRatio=88 storePathRootDir=/data/rocketmq/store maxMessageSize=65536 brokerRole=SLAVE flushDiskType=SYNC_FLUSH broker-b.properties 配置文件： [root@k8s-master01 config]# cat broker-b.properties brokerClusterName=rocketmq-cluster brokerName=broker-b brokerId=0 namesrvAddr=mq-namesrv-0.mq-namesrv.wiseco.svc.cluster.local:20901 defaultTopicQueueNums=4 autoCreateTopicEnable=true autoCreateSubscriptionGroup=true listenPort=20911 deleteWhen=04 fileReservedTime=120 mapedFileSizeCommitLog=1073741824 mapedFileSizeConsumeQueue=300000 diskMaxUsedSpaceRatio=88 storePathRootDir=/data/rocketmq/store maxMessageSize=65536 brokerRole=MASTER flushDiskType=SYNC_FLUSH broker-b-s.properties 配置文件： [root@k8s-master01 config]# cat broker-b-s.properties brokerClusterName=rocketmq-cluster brokerName=broker-b brokerId=1 namesrvAddr=mq-namesrv-0.mq-namesrv.wiseco.svc.cluster.local:20901 defaultTopicQueueNums=4 autoCreateTopicEnable=true autoCreateSubscriptionGroup=true listenPort=20911 deleteWhen=04 fileReservedTime=120 mapedFileSizeCommitLog=1073741824 mapedFileSizeConsumeQueue=300000 diskMaxUsedSpaceRatio=88 storePathRootDir=/data/rocketmq/store maxMessageSize=65536 brokerRole=SLAVE flushDiskType=SYNC_FLUSH 依据上面rocketmq的4个配置文件创建configmap存储卷 [root@k8s-master01 config]# pwd /opt/k8s/k8s_project/rocketmq/config [root@k8s-master01 config]# ll total 16 -rw-r--r-- 1 root root 464 Feb 2 23:55 broker-a.properties -rw-r--r-- 1 root root 462 Feb 2 23:59 broker-a-s.properties -rw-r--r-- 1 root root 464 Feb 2 23:55 broker-b.properties -rw-r--r-- 1 root root 462 Feb 2 23:55 broker-b-s.properties [root@k8s-master01 config]# kubectl create configmap rocketmq-config --from-file=broker-a.properties --from-file=broker-b.properties --from-file=broker-a-s.properties --from-file=broker-b-s.properties -n wiseco [root@k8s-master01 config]# kubectl get cm -n wiseco|grep rocketmq rocketmq-config 4 25m [root@k8s-master01 config]# kubectl describe cm rocketmq-config -n wiseco Name: rocketmq-config Namespace: wiseco Labels: Annotations: Data ==== broker-b-s.properties: ---- brokerClusterName=rocketmq-cluster brokerName=broker-b brokerId=1 namesrvAddr=mq-namesrv-0.mq-namesrv.wiseco.svc.cluster.local:20901 defaultTopicQueueNums=4 autoCreateTopicEnable=true autoCreateSubscriptionGroup=true listenPort=20911 deleteWhen=04 fileReservedTime=120 mapedFileSizeCommitLog=1073741824 mapedFileSizeConsumeQueue=300000 diskMaxUsedSpaceRatio=88 storePathRootDir=/data/rocketmq/store maxMessageSize=65536 brokerRole=SLAVE flushDiskType=SYNC_FLUSH broker-b.properties: ---- brokerClusterName=rocketmq-cluster brokerName=broker-b brokerId=0 namesrvAddr=mq-namesrv-0.mq-namesrv.wiseco.svc.cluster.local:20901 defaultTopicQueueNums=4 autoCreateTopicEnable=true autoCreateSubscriptionGroup=true listenPort=20911 deleteWhen=04 fileReservedTime=120 mapedFileSizeCommitLog=1073741824 mapedFileSizeConsumeQueue=300000 diskMaxUsedSpaceRatio=88 storePathRootDir=/data/rocketmq/store maxMessageSize=65536 brokerRole=MASTER flushDiskType=SYNC_FLUSH broker-a-s.properties: ---- brokerClusterName=rocketmq-cluster brokerName=broker-a brokerId=1 namesrvAddr=mq-namesrv-0.mq-namesrv.wiseco.svc.cluster.local:20901 defaultTopicQueueNums=4 autoCreateTopicEnable=true autoCreateSubscriptionGroup=true listenPort=20911 deleteWhen=04 fileReservedTime=120 mapedFileSizeCommitLog=1073741824 mapedFileSizeConsumeQueue=300000 diskMaxUsedSpaceRatio=88 storePathRootDir=/data/rocketmq/store maxMessageSize=65536 brokerRole=SLAVE flushDiskType=SYNC_FLUSH broker-a.properties: ---- brokerClusterName=rocketmq-cluster brokerName=broker-a brokerId=0 namesrvAddr=mq-namesrv-0.mq-namesrv.wiseco.svc.cluster.local:20901 defaultTopicQueueNums=4 autoCreateTopicEnable=true autoCreateSubscriptionGroup=true listenPort=20911 deleteWhen=04 fileReservedTime=120 mapedFileSizeCommitLog=1073741824 mapedFileSizeConsumeQueue=300000 diskMaxUsedSpaceRatio=88 storePathRootDir=/data/rocketmq/store maxMessageSize=65536 brokerRole=MASTER flushDiskType=SYNC_FLUSH Events: 4、部署rocketmq容器实例 启动命令-c可以指定配置文件，NameSvrAddr使用配置文件内指定的，配置文件使用前面的configmap挂载进来，subPath可以指定具体文件和目录。使用StatefulSet控制器部署rocketmq-a和rocketmq-b 两个master的pod。 [root@k8s-master01 rocketmq]# pwd /opt/k8s/k8s_project/rocketmq [root@k8s-master01 rocketmq]# ll total 12 drwxr-xr-x 2 root root 118 Feb 2 23:59 config drwxr-xr-x 5 root root 84 Feb 3 00:09 image -rw-r--r-- 1 root root 1216 Feb 1 11:11 nfs-rbac.yaml -rw-r--r-- 1 root root 161 Feb 1 11:12 rocketmq-nfs-class.yaml -rw-r--r-- 1 root root 1027 Feb 1 11:12 rocketmq-nfs.yml [root@k8s-master01 rocketmq]# mkdir deploy [root@k8s-master01 rocketmq]# cd deploy [root@k8s-master01 deploy]# ll total 24 -rw-r--r-- 1 root root 2283 Feb 2 16:57 broker-a-deployment.yaml -rw-r--r-- 1 root root 2403 Feb 2 16:57 broker-a-s-deployment.yaml -rw-r--r-- 1 root root 2283 Feb 2 16:57 broker-b-deployment.yaml -rw-r--r-- 1 root root 2403 Feb 2 16:57 broker-b-s-deployment.yaml -rw-r--r-- 1 root root 1439 Feb 2 23:46 namesrv-deployment.yaml -rw-r--r-- 1 root root 1371 Feb 2 18:36 rocketmq-externals-deployment.yaml broker-a-deployment.yaml 文件内容 [root@k8s-master01 deploy]# cat broker-a-deployment.yaml apiVersion: v1 kind: Service metadata: labels: app: broker-a name: broker-a namespace: wiseco spec: type: NodePort ports: - port: 20911 targetPort: 20911 name: broker-port nodePort: 30911 selector: app: broker-a --- apiVersion: apps/v1 kind: StatefulSet metadata: name: broker-a namespace: wiseco spec: serviceName: broker-a replicas: 1 selector: matchLabels: app: broker-a template: metadata: labels: app: broker-a spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: "app" operator: In values: - broker-a topologyKey: "kubernetes.io/hostname" containers: - name: broker-a image: 172.16.60.238/wiseco/rocketmq-4.8.0 imagePullPolicy: Always command: ["sh","-c","/usr/local/rocketmq-4.8.0/bin/mqbroker -c /usr/local/rocketmq-4.8.0/conf/broker-a.properties"] volumeMounts: - mountPath: /root/logs name: rocketmq-data subPath: mq-brokeroptlogs - mountPath: /data/rocketmq name: rocketmq-data subPath: mq-brokeroptstore - name: broker-config mountPath: /usr/local/rocketmq-4.8.0/conf/broker-a.properties subPath: broker-a.properties lifecycle: postStart: exec: command: ["/bin/sh","-c","touch /tmp/health"] livenessProbe: exec: command: ["test","-e","/tmp/health"] initialDelaySeconds: 5 timeoutSeconds: 5 periodSeconds: 10 readinessProbe: tcpSocket: port: 20911 initialDelaySeconds: 15 timeoutSeconds: 5 periodSeconds: 20 volumes: - name: broker-config configMap: name: rocketmq-config volumeClaimTemplates: - metadata: name: rocketmq-data annotations: volume.beta.kubernetes.io/storage-class: "rocketmq-nfs-storage" spec: accessModes: - ReadWriteMany resources: requests: storage: 10Gi broker-a-s-deployment.yaml 文件内容 [root@k8s-master01 deploy]# cat broker-a-s-deployment.yaml apiVersion: v1 kind: Service metadata: labels: app: broker-a-s name: broker-a-s namespace: wiseco spec: type: NodePort ports: - port: 20911 targetPort: 20911 name: broker-port nodePort: 30912 selector: app: broker-a-s --- apiVersion: apps/v1 kind: StatefulSet metadata: name: broker-a-s namespace: wiseco spec: serviceName: broker-a-s replicas: 1 selector: matchLabels: app: broker-a-s template: metadata: labels: app: broker-a-s spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: "app" operator: In values: - broker-a-s topologyKey: "kubernetes.io/hostname" containers: - name: broker-a-s image: 172.16.60.238/wiseco/rocketmq-4.8.0 imagePullPolicy: Always command: ["sh","-c","/usr/local/rocketmq-4.8.0/bin/mqbroker -c /usr/local/rocketmq-4.8.0/conf/broker-a-s.properties"] volumeMounts: - mountPath: /root/logs name: rocketmq-data subPath: mq-brokeroptlogs - mountPath: /data/rocketmq name: rocketmq-data subPath: mq-brokeroptstore - name: broker-config mountPath: /usr/local/rocketmq-4.8.0/conf/broker-a-s.properties subPath: broker-a-s.properties lifecycle: postStart: exec: command: ["/bin/sh","-c","touch /tmp/health"] livenessProbe: exec: command: ["test","-e","/tmp/health"] initialDelaySeconds: 5 timeoutSeconds: 5 periodSeconds: 10 readinessProbe: tcpSocket: port: 20911 initialDelaySeconds: 15 timeoutSeconds: 5 periodSeconds: 20 volumes: - name: broker-config configMap: name: rocketmq-config items: - key: broker-a-s.properties path: broker-a-s.properties volumeClaimTemplates: - metadata: name: rocketmq-data annotations: volume.beta.kubernetes.io/storage-class: "rocketmq-nfs-storage" spec: accessModes: - ReadWriteMany resources: requests: storage: 10Gi broker-b-deployment.yaml 文件内容 [root@k8s-master01 deploy]# cat broker-b-deployment.yaml apiVersion: v1 kind: Service metadata: labels: app: broker-b name: broker-b namespace: wiseco spec: type: NodePort ports: - port: 20911 targetPort: 20911 name: broker-port nodePort: 30913 selector: app: broker-b --- apiVersion: apps/v1 kind: StatefulSet metadata: name: broker-b namespace: wiseco spec: serviceName: broker-b replicas: 1 selector: matchLabels: app: broker-b template: metadata: labels: app: broker-b spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: "app" operator: In values: - broker-b topologyKey: "kubernetes.io/hostname" containers: - name: broker-b image: 172.16.60.238/wiseco/rocketmq-4.8.0 imagePullPolicy: Always command: ["sh","-c","/usr/local/rocketmq-4.8.0/bin/mqbroker -c /usr/local/rocketmq-4.8.0/conf/broker-b.properties"] volumeMounts: - mountPath: /root/logs name: rocketmq-data subPath: mq-brokeroptlogs - mountPath: /data/rocketmq name: rocketmq-data subPath: mq-brokeroptstore - name: broker-config mountPath: /usr/local/rocketmq-4.8.0/conf/broker-b.properties subPath: broker-b.properties lifecycle: postStart: exec: command: ["/bin/sh","-c","touch /tmp/health"] livenessProbe: exec: command: ["test","-e","/tmp/health"] initialDelaySeconds: 5 timeoutSeconds: 5 periodSeconds: 10 readinessProbe: tcpSocket: port: 20911 initialDelaySeconds: 15 timeoutSeconds: 5 periodSeconds: 20 volumes: - name: broker-config configMap: name: rocketmq-config volumeClaimTemplates: - metadata: name: rocketmq-data annotations: volume.beta.kubernetes.io/storage-class: "rocketmq-nfs-storage" spec: accessModes: - ReadWriteMany resources: requests: storage: 10Gi broker-b-s-deployment.yaml 文件内容 [root@k8s-master01 deploy]# cat broker-b-s-deployment.yaml apiVersion: v1 kind: Service metadata: labels: app: broker-b-s name: broker-b-s namespace: wiseco spec: type: NodePort ports: - port: 20911 targetPort: 20911 name: broker-port nodePort: 30914 selector: app: broker-b-s --- apiVersion: apps/v1 kind: StatefulSet metadata: name: broker-b-s namespace: wiseco spec: serviceName: broker-b-s replicas: 1 selector: matchLabels: app: broker-b-s template: metadata: labels: app: broker-b-s spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: "app" operator: In values: - broker-b-s topologyKey: "kubernetes.io/hostname" containers: - name: broker-b-s image: 172.16.60.238/wiseco/rocketmq-4.8.0 imagePullPolicy: Always command: ["sh","-c","/usr/local/rocketmq-4.8.0/bin/mqbroker -c /usr/local/rocketmq-4.8.0/conf/broker-b-s.properties"] volumeMounts: - mountPath: /root/logs name: rocketmq-data subPath: mq-brokeroptlogs - mountPath: /data/rocketmq name: rocketmq-data subPath: mq-brokeroptstore - name: broker-config mountPath: /usr/local/rocketmq-4.8.0/conf/broker-b-s.properties subPath: broker-b-s.properties lifecycle: postStart: exec: command: ["/bin/sh","-c","touch /tmp/health"] livenessProbe: exec: command: ["test","-e","/tmp/health"] initialDelaySeconds: 5 timeoutSeconds: 5 periodSeconds: 10 readinessProbe: tcpSocket: port: 20911 initialDelaySeconds: 15 timeoutSeconds: 5 periodSeconds: 20 volumes: - name: broker-config configMap: name: rocketmq-config items: - key: broker-b-s.properties path: broker-b-s.properties volumeClaimTemplates: - metadata: name: rocketmq-data annotations: volume.beta.kubernetes.io/storage-class: "rocketmq-nfs-storage" spec: accessModes: - ReadWriteMany resources: requests: storage: 10Gi namesrv-deployment.yaml文件内容 [root@k8s-master01 deploy]# cat namesrv-deployment.yaml apiVersion: v1 kind: Service metadata: labels: app: mq-namesrv name: mq-namesrv namespace: wiseco spec: type: NodePort ports: - port: 20901 targetPort: 20901 name: namesrv-port nodePort: 30915 selector: app: mq-namesrv --- apiVersion: apps/v1 kind: StatefulSet metadata: name: mq-namesrv namespace: wiseco spec: serviceName: mq-namesrv replicas: 1 selector: matchLabels: app: mq-namesrv template: metadata: labels: app: mq-namesrv spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: "app" operator: In values: - mq-namesrv topologyKey: "kubernetes.io/hostname" containers: - name: mq-namesrv image: 172.16.60.238/wiseco/rocketmq_namesrv:v1 imagePullPolicy: Always lifecycle: postStart: exec: command: ["/bin/sh","-c","touch /tmp/health"] livenessProbe: exec: command: ["test","-e","/tmp/health"] initialDelaySeconds: 5 timeoutSeconds: 5 periodSeconds: 10 readinessProbe: tcpSocket: port: 20901 initialDelaySeconds: 15 timeoutSeconds: 5 periodSeconds: 20 rocketmq-externals-deployment.yaml 文件内容 [root@k8s-master01 deploy]# cat rocketmq-externals-deployment.yaml apiVersion: v1 kind: Service metadata: labels: app: mq-externals name: mq-externals namespace: wiseco spec: type: NodePort ports: - port: 8080 targetPort: 8080 name: console-port nodePort: 30916 selector: app: mq-externals --- apiVersion: apps/v1 kind: Deployment metadata: name: mq-externals namespace: wiseco spec: replicas: 1 selector: matchLabels: app: mq-externals template: metadata: labels: app: mq-externals spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: "app" operator: In values: - mq-externals topologyKey: "kubernetes.io/hostname" containers: - name: mq-externals image: 172.16.60.238/wiseco/rocketmq-externals:v1 imagePullPolicy: Always lifecycle: postStart: exec: command: ["/bin/sh","-c","touch /tmp/health"] livenessProbe: exec: command: ["test","-e","/tmp/health"] initialDelaySeconds: 5 timeoutSeconds: 5 periodSeconds: 10 readinessProbe: tcpSocket: port: 8080 initialDelaySeconds: 15 部署rockermq相应pod [root@k8s-master01 deploy]# kubectl apply -f .
查看pod、svc [root@k8s-master01 deploy]# kubectl get pods -n wiseco NAME READY STATUS RESTARTS AGE broker-a-0 1/1 Running 0 35m broker-a-s-0 1/1 Running 0 35m broker-b-0 1/1 Running 0 35m broker-b-s-0 1/1 Running 0 35m mq-externals-57c9ddddcb-cffj5 1/1 Running 0 35m mq-namesrv-0 1/1 Running 0 35m rocketmq-nfs-client-provisioner-59d8bbd947-k294r 1/1 Running 1 37h [root@k8s-master01 deploy]# kubectl get svc -n wiseco NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE broker-a NodePort 10.254.52.116 20911:30911/TCP 35m broker-a-s NodePort 10.254.32.251 20911:30912/TCP 35m broker-b NodePort 10.254.103.228 20911:30913/TCP 35m broker-b-s NodePort 10.254.39.178 20911:30914/TCP 35m mq-externals NodePort 10.254.33.18 8080:30916/TCP 35m mq-namesrv NodePort 10.254.221.168 20901:30915/TCP
查看PV、PVC [root@k8s-master01 deploy]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-4d31cdfb-94c3-4973-b3c3-e62ce88ae29e 10Gi RWX Delete Bound wiseco/rocketmq-data-broker-a-s-0 rocketmq-nfs-storage 40m pvc-77d7358e-ee27-49f2-bf29-7c31fa6250b1 10Gi RWX Delete Bound wiseco/rocketmq-data-broker-a-0 rocketmq-nfs-storage 40m pvc-d1d06c6b-2b3d-43e3-88cf-0d56e2eb8bd7 10Gi RWX Delete Bound wiseco/rocketmq-data-broker-b-0 rocketmq-nfs-storage 40m pvc-f05d79c8-c2b0-4f00-b300-7db23a3a0c28 10Gi RWX Delete Bound wiseco/rocketmq-data-broker-b-s-0 rocketmq-nfs-storage 40m [root@k8s-master01 deploy]# kubectl get pvc -n wiseco NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE rocketmq-data-broker-a-0 Bound pvc-77d7358e-ee27-49f2-bf29-7c31fa6250b1 10Gi RWX rocketmq-nfs-storage 40m rocketmq-data-broker-a-s-0 Bound pvc-4d31cdfb-94c3-4973-b3c3-e62ce88ae29e 10Gi RWX rocketmq-nfs-storage 40m rocketmq-data-broker-b-0 Bound pvc-d1d06c6b-2b3d-43e3-88cf-0d56e2eb8bd7 10Gi RWX rocketmq-nfs-storage 40m rocketmq-data-broker-b-s-0 Bound pvc-f05d79c8-c2b0-4f00-b300-7db23a3a0c28 10Gi RWX rocketmq-nfs-storage 40m查看NFS共享存储 NFS服务器（172.16.60.238），查看共享目录/data/storage/k8s/rocketmq [root@k8s-harbor01 ~]# cd /data/storage/k8s/rocketmq/ [root@k8s-harbor01 rocketmq]# ll total 0 drwxrwxrwx 4 root root 55 Feb 3 00:02 wiseco-rocketmq-data-broker-a-0-pvc-77d7358e-ee27-49f2-bf29-7c31fa6250b1 drwxrwxrwx 4 root root 55 Feb 3 00:02 wiseco-rocketmq-data-broker-a-s-0-pvc-4d31cdfb-94c3-4973-b3c3-e62ce88ae29e drwxrwxrwx 4 root root 55 Feb 3 00:02 wiseco-rocketmq-data-broker-b-0-pvc-d1d06c6b-2b3d-43e3-88cf-0d56e2eb8bd7 drwxrwxrwx 4 root root 55 Feb 3 00:02 wiseco-rocketmq-data-broker-b-s-0-pvc-f05d79c8-c2b0-4f00-b300-7db23a3a0c28 [root@k8s-harbor01 rocketmq]# ll wiseco-rocketmq-data-broker-a-0-pvc-77d7358e-ee27-49f2-bf29-7c31fa6250b1/ total 0 drwxrwxrwx 3 root root 26 Feb 3 00:02 mq-brokeroptlogs drwxrwxrwx 3 root root 19 Feb 3 00:02 mq-brokeroptstore [root@k8s-harbor01 rocketmq]# ll wiseco-rocketmq-data-broker-a-0-pvc-77d7358e-ee27-49f2-bf29-7c31fa6250b1/mq-brokeroptstore/ total 0 drwxr-xr-x 3 root root 63 Feb 3 00:02 store [root@k8s-harbor01 rocketmq]# ll wiseco-rocketmq-data-broker-a-0-pvc-77d7358e-ee27-49f2-bf29-7c31fa6250b1/mq-brokeroptstore/store/ total 8 -rw-r--r-- 1 root root 0 Feb 3 00:02 abort -rw-r--r-- 1 root root 4096 Feb 3 00:43 checkpoint drwxr-xr-x 2 root root 193 Feb 3 00:44 config -rw-r--r-- 1 root root 4 Feb 3 00:02 lock 5、rockermq的可视化界面 访问http://node节点ip:30916 
再次查看可视化界面 
7、客户端连接RockerMQ集群 RocketMQ可以令客户端找到Name Server, 然后通过Name Server再找到Broker。如下所示有多种配置方式，优先级由高到低，高优先级会覆盖低优先级； 代码中指定Name Server地址，多个namesrv地址之间用分号分割： producer.setNamesrvAddr("mq-namesrv-0.mq-namesrv.wiseco.svc.cluster.local:20901"); consumer.setNamesrvAddr("mq-namesrv-0.mq-namesrv.wiseco.svc.cluster.local:20901");
|
【本文地址】
今日新闻 |
推荐新闻 |