SuifengFlying

公告

欢迎来到我的博客，这里记录生活与技术。

了解更多

465 字

2 分钟

Kafka集群部署

2025-11-21

BigData

/

Kafka

/

Deployment

Kafka集群部署#

系统配置调整#

关闭防火墙#

1
sudo systemctl stop ufw
2
sudo systemctl disable ufw

资源配置调整#

编辑/etc/security/limits.conf文件

1
sudo vi /etc/security/limits.conf

添加以下内容：

1
* soft nofile 65536
2
* hard nofile 65536
3
* soft nproc 65536
4
* hard nproc 65536

安装与配置Kafka集群#

下载并解压#

1
cd /export/software
2
wget -P /export/software/ https://repo.huaweicloud.com/apache/kafka/3.4.0/kafka_2.12-3.4.0.tgz
3
tar -zxvf /export/software/kafka_2.12-3.4.0.tgz -C /export/servers/
4
cd /export/servers
5
mv /export/servers/kafka_2.12-3.4.0 /export/servers/kafka

配置Kafka服务器参数#

编辑修改/export/servers/kafka/config/server.properties文件

1
sudo chown -R hadooper:hadooper /export/servers/kafka/
2
vi /export/servers/kafka/config/server.properties

hadoop01：

1
broker.id=1
2
listeners=PLAINTEXT://hadoop01:9092
3
advertised.listeners=PLAINTEXT://hadoop01:9092
4
log.dirs=/export/servers/kafka/kafka-logs
5
num.partitions=3
6
zookeeper.connect=hadoop01:2181,hadoop02:2181,hadoop03:2181
7
default.replication.factor=3
8
offsets.topic.replication.factor=3

1
vi /export/servers/kafka/bin/kafka-server-start.sh

添加如下内容于脚本开始处：

1
export JAVA_HOME=/export/servers/jdk

1
vi /export/servers/kafka/bin/kafka-server-stop.sh

添加如下内容于脚本开始处：

1
export JAVA_HOME=/export/servers/jdk

删除并替换hive中原有的kafka：

1
rm /export/servers/hive-4.0.1/lib/kafka-clients-2.5.0.jar
2
cp /export/servers/kafka/libs/kafka-clients-3.4.0.jar /export/servers/hive-4.0.1/lib/

于hadoop01节点分发至hadoop02、hadoop03节点

1
scp -r /export/servers/kafka hadoop02:/tmp/
2
scp -r /export/servers/kafka hadoop03:/tmp/

进入hadoop02:

1
sudo cp -r /tmp/kafka /export/servers/
2
rm -r /tmp/kafka

编辑修改/export/servers/kafka/config/server.properties文件

1
sudo chown -R hadooper:hadooper /export/servers/kafka/
2
vi /export/servers/kafka/config/server.properties

hadoop02:

1
broker.id=2
2
listeners=PLAINTEXT://hadoop02:9092
3
advertised.listeners=PLAINTEXT://hadoop02:9092
4
log.dirs=/export/servers/kafka/kafka-logs
5
num.partitions=3
6
zookeeper.connect=hadoop01:2181,hadoop02:2181,hadoop03:2181
7
default.replication.factor=3
8
offsets.topic.replication.factor=3

进入hadoop03:

1
sudo cp -r /tmp/kafka /export/servers/
2
rm -r /tmp/kafka

编辑修改/export/servers/kafka/config/server.properties文件

1
sudo chown -R hadooper:hadooper /export/servers/kafka/
2
vi /export/servers/kafka/config/server.properties

hadoop03:

1
broker.id=3
2
listeners=PLAINTEXT://hadoop03:9092
3
advertised.listeners=PLAINTEXT://hadoop03:9092
4
log.dirs=/export/servers/kafka/kafka-logs
5
num.partitions=3
6
zookeeper.connect=hadoop01:2181,hadoop02:2181,hadoop03:2181
7
default.replication.factor=3
8
offsets.topic.replication.factor=3

启动集群验证#

启动ZooKeeper集群：在各节点上启动ZooKeeper服务#

启动节点

1
zkServer.sh start

检查节点角色（Leader/Follower）

1
zkServer.sh status

启动Kafka集群#

按顺序在每个Kafka节点上，使用以下命令启动Kafka服务：

1
cd /export/servers/kafka
2
mkdir -p /export/servers/kafka/logs
3
mkdir -p /export/servers/kafka/kafka-logs
4
/export/servers/kafka/bin/kafka-server-start.sh -daemon /export/servers/kafka/config/server.properties

测试#

1
jps

出现同时出现QuorumPeerMain和Kafka则代表服务启动成功,如下图所示

测试集群工作状态#

在hadoop01上创建topic:

1
/export/servers/kafka/bin/kafka-topics.sh --create --bootstrap-server hadoop01:9092,hadoop02:9092,hadoop03:9092 --replication-factor 3 --partitions 3 --topic my-test-topic

查看topic描述确认分区和副本分配情况#

1
/export/servers/kafka/bin/kafka-topics.sh --describe --bootstrap-server hadoop01:9092 --topic my-test-topic

验证生产消费端是否消息互通：#

hadoop01(生产者):

1
/export/servers/kafka/bin/kafka-console-producer.sh --bootstrap-server hadoop01:9092 --topic my-test-topic

hadoop02(消费者):

1
/export/servers/kafka/bin/kafka-console-consumer.sh --bootstrap-server hadoop02:9092 --topic my-test-topic --from-beginning

hadoo03(消费者):

1
/export/servers/kafka/bin/kafka-console-consumer.sh --bootstrap-server hadoop03:9092 --topic my-test-topic --from-beginning

测试集群状态#

在hadoop01输入数据，在hadoop02、hadoop03查看数据，若全部互通则代表kafka集群正常运行，如下图所示