SuifengFlying

公告

欢迎来到我的博客，这里记录生活与技术。

了解更多

552 字

3 分钟

Spark集群部署

2025-01-15

BigData

/

Spark

/

Deployment

Spark集群部署#

Spark的安装并配置启动#

下载Spark安装包#

1
wget -P /export/software/ https://repo.huaweicloud.com/apache/spark/spark-3.4.3/spark-3.4.3-bin-hadoop3.tgz

解压Spark安装包#

1
tar -zxvf /export/software/spark-3.4.3-bin-hadoop3.tgz -C /export/servers
2
# 重命名:
3
cd /export/servers
4
mv /export/servers/spark-3.4.3-bin-hadoop3 /export/servers/spark

修改配置文件#

进入spark/conf目录修改Spark的配置文件spark-env.sh,将spark-env.sh.template配置模板文件复制一份并命名为spark-env.sh,命令如下:

1
cd /export/servers/spark/conf
2
sudo cp /export/servers/spark/conf/spark-env.sh.template /export/servers/spark/conf/spark-env.sh

修改spark-env.sh文件#

1
sudo vim /export/servers/spark/conf/spark-env.sh

在该文件中添加以下内容:

1
#配置java环境变量
2
export JAVA_HOME=/export/servers/jdk
3
#指定Master的IP
4
export SPARK_MASTER_HOST=hadoop01
5
#指定Master的端口
6
export SPARK_MASTER_PORT=7077
7
export SPARK_MASTER_WEBUI_PORT=8082

复制workers.template文件，并重命名为workers#

1
sudo cp /export/servers/spark/conf/workers.template /export/servers/spark/conf/workers

编辑workers配置文件#

1
sudo vim /export/servers/spark/conf/workers

1
hadoop01
2
hadoop02
3
hadoop03

分发文件#

1
scp -r /export/servers/spark/ hadoop02:/tmp/
2
scp -r /export/servers/spark/ hadoop03:/tmp/

进入hadoop02:

1
sudo cp -r /tmp/spark /export/servers
2
rm -r /tmp/spark

进入hadoop03:

1
sudo cp -r /tmp/spark /export/servers
2
rm -r /tmp/spark

启动Spark集群#

分别添加权限#

1
cd /export/servers
2
sudo chown -R hadooper:hadooper /export/servers/spark/

分步启动Spark#

hadoop01:

1
cd /export/servers/spark/sbin/
2
/export/servers/spark/sbin/start-master.sh

hadoop02,hadoop03:

1
cd /export/servers/spark/sbin/
2
/export/servers/spark/sbin/start-slave.sh spark://hadoop01:7077

此步的hadoop02和hadoop03启动时如若出现问题，首先检查主机名是否正确！如若不正确，按照网络上的教程进行修改，此处不做过多赘述

输入$SPARK_HOME/sbin/start-all.sh启动(上述步骤也是启动，为按服务器逐步启动，该启动指令为批量启动),如下如所示即为成功,进一步验证打开本机浏览器输入（hadoop01的IP地址）:8082如果界面正常打开如下所示即代表成功

使Spark适配YARN#

配置Spark以识别YARN#

进入配置目录/export/servers/spark/conf修改spark-env.sh文件

1
cd /export/servers/spark/conf
2
sudo vim /export/servers/spark/conf/spark-env.sh

添加如下内容:

1
# 设置Hadoop配置文件目录（请根据你的实际路径修改）
2
export HADOOP_CONF_DIR=/export/servers/hadoop/etc/hadoop

将以下内容注释掉（用#注释）：

1
export SPARK_MASTER_HOST=hadoop01
2
export SPARK_MASTER_PORT=7077
3
export SPARK_MASTER_WEBUI_PORT=8082

调整YARN配置#

进入配置目录

1
cd /export/servers/hadoop/etc/hadoop

修改yarn-site.xml文件

1
sudo vim /export/servers/hadoop/etc/hadoop/yarn-site.xml

添加内容如下(注意缩进要与文件中其他内容保持一致):

1
  <property>
2
        <name>yarn.nodemanager.pmem-check-enabled</name>
3
        <value>false</value>
4
    </property>
5
    <property>
6
        <name>yarn.nodemanager.vmem-check-enabled</name>
7
    <value>false</value>
8
    </property>

分发文件#

hadoop01:

1
scp -r /export/servers/hadoop/etc/hadoop hadoop02:/tmp/

hadoop02:

1
sudo rm -rf /export/servers/hadoop/etc/hadoop
2
sudo cp -r /tmp/hadoop /export/servers/hadoop/etc/hadoop
3
rm -r /tmp/hadoop

hadoop01:

1
scp -r /export/servers/hadoop/etc/hadoop hadoop03:/tmp/

hadoop03:

1
sudo rm -rf /export/servers/hadoop/etc/hadoop
2
sudo cp -r /tmp/hadoop /export/servers/hadoop/etc/hadoop
3
rm -r /tmp/hadoop

分发完成后每个节点都需要重启YARN集群先输入source /etc/profile加载配置进入/export/servers/spark/sbin目录

1
cd /export/servers/spark/sbin

重新启动yarn

1
/export/servers/spark/sbin/stop-yarn.sh
2
/export/servers/spark/sbin/start-yarn.sh

Spark集群部署

https://mizuki.mysqil.com/posts/spark部署/

作者

SuifengFlying

发布于

2025-01-15

许可协议

CC BY-NC-SA 4.0

Big Data Deployment

11 / 12

Mahout集群部署 ZooKeeper的配置

欢迎来到 SuifengFlying!

Spark集群部署#

Spark的安装并配置启动#

下载Spark安装包#

解压Spark安装包#

修改配置文件#

修改spark-env.sh文件#

复制workers.template文件，并重命名为workers#

编辑workers配置文件#

分发文件#

启动Spark集群#

分别添加权限#

分步启动Spark#

使Spark适配YARN#

配置Spark以识别YARN#

调整YARN配置#

分发文件#

Big Data Deployment

Related articles