大数据的持续搭建一(容器化)

容器化搭建可以看前面的搭建hadoop,我的3个节点是在3个容器内搭建的分别为master,slave1,slave2
此后我直接说节点

一.spark的搭建

1.将需要的包导入master节点并且解压

[root@localhost ~]# docker cp spark-3.1.1-bin-hadoop3.2.tgz master:/root[root@master ~]# tar -zxvf spark-3.1.1-bin-hadoop3.2.tgz -C /opt/module/

2.配置环境变量

(1)添加如下几行

[root@master module]# vim /etc/profileexportSPARK_HOME=/opt/module/spark-3.1.1-bin-hadoop3.2exportPATH=$PATH:$SPARK_HOME/bin

(2)使环境变量生效

[root@master module]# source /etc/profile

(3)验证是否成功

[root@master module]# spark-submit --versionWelcome to ____ __ / __/__ ___ _____/ /__ _\\/ _\/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\version3.1.1 /_/ Using Scala version2.12.10, Java HotSpot(TM)64-Bit Server VM,1.8.0_371 Branch HEAD Compiled by user ubuntu on2021-02-22T01:33:19Z Revision 1d550c4e90275ab418b9161925049239227f3dc9 Url https://github.com/apache/spark Type--helpformoreinformation.

3.配置相关环境

[root@master hadoop]# cd /opt/module/spark-3.1.1-bin-hadoop3.2/conf/[root@master conf]# cp spark-env.sh.template spark-env.sh[root@master conf]# cp workers.template workers[root@master conf]# vim spark-env.sh #添加如下几行exportJAVA_HOME=/opt/module/jdk1.8.0_371exportHADOOP_HOME=/opt/module/hadoop-3.2.1exportYARN_CONF_DIR=/opt/module/hadoop-3.2.1/etc/hadoopexportHADOOP_CONF_DIR=/opt/module/hadoop-3.2.1/etc/hadoop[root@master conf]# vim workers #添加3行,删除原来的localhostmaster slave1 slave2

4.把master节点的传给其他两个节点

[root@master conf]# scp -r /etc/profile slave1:/etc/profile[root@master conf]# scp -r /etc/profile slave2:/etc/profile[root@master conf]# scp -r /opt/module/spark-3.1.1-bin-hadoop3.2/ slave1:/opt/module/[root@master conf]# scp -r /opt/module/spark-3.1.1-bin-hadoop3.2/ slave2:/opt/module/

5.测试

[root@master spark-3.1.1-bin-hadoop3.2]# ./bin/run-example SparkPi 10Assignments Map()2025-10-2017:58:46,196 INFO scheduler.TaskSetManager: Finished task5.0instage0.0(TID5)in47ms on master(executor driver)(6/10)2025-10-2017:58:46,198 INFO executor.Executor: Running task7.0instage0.0(TID7)2025-10-2017:58:46,211 INFO executor.Executor: Finished task6.0instage0.0(TID6).957bytes result sent to driver2025-10-2017:58:46,213 INFO scheduler.TaskSetManager: Starting task8.0instage0.0(TID8)(master, executor driver, partition8, PROCESS_LOCAL,4578bytes)taskResourceAssignments Map()2025-10-2017:58:46,214 INFO scheduler.TaskSetManager: Finished task6.0instage0.0(TID6)in36ms on master(executor driver)(7/10)2025-10-2017:58:46,217 INFO executor.Executor: Running task8.0instage0.0(TID8)2025-10-2017:58:46,233 INFO executor.Executor: Finished task7.0instage0.0(TID7).957bytes result sent to driver2025-10-2017:58:46,238 INFO scheduler.TaskSetManager: Starting task9.0instage0.0(TID9)(master, executor driver, partition9, PROCESS_LOCAL,4578bytes)taskResourceAssignments Map()2025-10-2017:58:46,239 INFO scheduler.TaskSetManager: Finished task7.0instage0.0(TID7)in44ms on master(executor driver)(8/10)2025-10-2017:58:46,239 INFO executor.Executor: Running task9.0instage0.0(TID9)2025-10-2017:58:46,255 INFO executor.Executor: Finished task8.0instage0.0(TID8).957bytes result sent to driver2025-10-2017:58:46,263 INFO scheduler.TaskSetManager: Finished task8.0instage0.0(TID8)in51ms on master(executor driver)(9/10)2025-10-2017:58:46,273 INFO executor.Executor: Finished task9.0instage0.0(TID9).957bytes result sent to driver2025-10-2017:58:46,274 INFO scheduler.TaskSetManager: Finished task9.0instage0.0(TID9)in37ms on master(executor driver)(10/10)2025-10-2017:58:46,276 INFO scheduler.TaskSchedulerImpl: Removed TaskSet0.0, whose tasks have all completed, from pool2025-10-2017:58:46,285 INFO scheduler.DAGScheduler: ResultStage0(reduce at SparkPi.scala:38)finishedin1.230s2025-10-2017:58:46,292 INFO scheduler.DAGScheduler: Job0is finished. Cancelling potential speculative or zombie tasksforthis job2025-10-2017:58:46,292 INFO scheduler.TaskSchedulerImpl: Killing all running tasksinstage0: Stage finished2025-10-2017:58:46,296 INFO scheduler.DAGScheduler: Job0finished: reduce at SparkPi.scala:38, took1.313123s Pi is roughly3.1379711379711382025-10-2017:58:46,315 INFO server.AbstractConnector: Stopped Spark@53cdecf6{HTTP/1.1,(http/1.1)}{0.0.0.0:4040}2025-10-2017:58:46,321 INFO ui.SparkUI: Stopped Spark web UI at http://master:40402025-10-2017:58:46,345 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!2025-10-2017:58:46,390 INFO memory.MemoryStore: MemoryStore cleared2025-10-2017:58:46,391 INFO storage.BlockManager: BlockManager stopped2025-10-2017:58:46,407 INFO storage.BlockManagerMaster: BlockManagerMaster stopped2025-10-2017:58:46,411 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:OutputCommitCoordinator stopped!2025-10-2017:58:46,444 INFO spark.SparkContext: Successfully stopped SparkContext2025-10-2017:58:46,449 INFO util.ShutdownHookManager: Shutdown hook called2025-10-2017:58:46,449 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-46db54ca-ca2c-4114-b9fb-58f0fc3a62b42025-10-2017:58:46,452 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-97095aab-a35e-4bb4-9edb-9177e4b23cb7

输出如下说明没问题

二.Hbase的搭建

1.将需要的包导入master节点并且解压

[root@localhost ~]# docker cp hbase-2.2.3-bin.tar.gz master:/root[root@master ~]# tar -zxvf hbase-2.2.3-bin.tar.gz -C /opt/module/[root@master ~]# cd /opt/module/[root@master module]# mv hbase-2.2.3 hbase

2.配置环境变量

[root@master ~]# vim /etc/profileexportHBASE_HOME=/opt/module/hbaseexportPATH=$PATH:$HBASE_HOME/bin[root@master module]# source /etc/profile[root@master module]# hbase versionSLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found bindingin[jar:file:/opt/module/hadoop-3.2.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found bindingin[jar:file:/opt/module/hbase/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is oftype[org.slf4j.impl.Log4jLoggerFactory]HBase2.2.3 Source code repository git://hao-OptiPlex-7050/home/hao/open_source/hbaserevision=6a830d87542b766bd3dc4cfdee28655f62de3974 Compiled by hao on2020? 01?10? ???18:27:51 CST Fromsourcewith checksum 097925184b85f6995e20da5462b10f3f

3.配置相关环境

[root@master conf]# vim hbase-env.shexportJAVA_HOME=/opt/module/jdk1.8.0_371exportHBASE_MANAGES_ZK=false[root@master conf]# vim hbase-site.xml<configuration><property><name>hbase.rootdir</name><value>hdfs://master:9000/hbase</value></property><property><name>hbase.cluster.distributed</name><value>true</value></property><property><name>hbase.zookeeper.quorum</name><value>master,slave1,slave2</value></property><property><name>hbase.unsafe.stream.capability.enforce</name><value>false</value></property></configuration>[root@master conf]# vim regionserversmaster slave1 slave2[root@master conf]# cp /opt/module/hadoop-3.2.1/etc/hadoop/core-site.xml /opt/module/hbase/conf/[root@master conf]# cp /opt/module/hadoop-3.2.1/etc/hadoop/hdfs-site.xml /opt/module/hbase/conf/

4把master节点的传给其他两个节点

跟前面一样这里就不写了之后都是

5.启动

[root@master conf]# start-hbase.shSLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found bindingin[jar:file:/opt/module/hadoop-3.2.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found bindingin[jar:file:/opt/module/hbase/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is oftype[org.slf4j.impl.Log4jLoggerFactory]running master, logging to /opt/module/hbase/logs/hbase-root-master-master.out slave2: running regionserver, logging to /opt/module/hbase/bin/../logs/hbase-root-regionserver-slave2.out slave1: running regionserver, logging to /opt/module/hbase/bin/../logs/hbase-root-regionserver-slave1.out master: running regionserver, logging to /opt/module/hbase/bin/../logs/hbase-root-regionserver-master.out[root@master conf]# hbase shell # 启动 HBase 交互式命令行工具SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found bindingin[jar:file:/opt/module/hadoop-3.2.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found bindingin[jar:file:/opt/module/hbase/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is oftype[org.slf4j.impl.Log4jLoggerFactory]HBase Shell Use"help"to get list of supported commands. Use"exit"to quit this interactive shell. For Reference, please visit: http://hbase.apache.org/2.0/book.html#shellVersion2.2.3, r6a830d87542b766bd3dc4cfdee28655f62de3974,2020年 01月10日 星期五18:27:51 CST Took0.0034seconds hbase(main):001:0>

三.hudi,maven的搭建

1.将需要的包导入master节点并且解压

[root@localhost ~]# docker cp apache-maven-3.6.3-bin.tar.gz master:/root[root@localhost ~]# docker cp hudi-0.11.0.src.tgz master:/root[root@master ~]# tar -zxvf apache-maven-3.6.3-bin.tar.gz -C /opt/module/[root@master ~]# tar -zxvf hudi-0.11.0.src.tgz -C /opt/module/[root@master modulel# mv apache-maven-3.6.3/ maven

2.配置环境变量

[root@master ~]# vim /etc/profileexportMAVEN_HOME=/opt/module/mavenexportPATH=$PATH:$MAVEN_HOME/bin[root@master module]# source /etc/profile[root@master ~]# mvn -vApache Maven3.6.3(cecedd343002696d0abb50b32b541b8a6ba2883f)Maven home: /opt/module/maven Java version:1.8.0_371, vendor: Oracle Corporation, runtime: /opt/module/jdk1.8.0_371/jre Default locale: en_US, platform encoding: ANSI_X3.4-1968 OS name:"linux", version:"3.10.0-1160.el7.x86_64", arch:"amd64", family:"unix"

3.配置相关环境

[root@master ~]# cd /opt/module/maven/conf/[root@master conf]# vim settings.xml/local<localRepository>/path/to/local/repo</localRepository>有需要在这里改仓库位置 /mirrors<mirror><id>nexus-aliyun</id><mirrorOf>central</mirrorOf><name>Nexus aliyun</name><url>http://maven.aliyun.com/nexus/content/groups/public</url></mirror>[root@master conf]# cd /opt/module/hudi-0.11.0/[root@master hudi-0.11.0]# vim pom.xml/repo<id>nexus-aliyun</id><name>nexus-aliyun</name><url>http://maven.aliyun.com/nexus/content/groups/public</url>/hadoop<hadoop.version>3.2.1</hadoop.version><hive.version>3.1.2</hive.version>[root@master hudi-0.11.0]# cd hudi-common/src/main/java/org/apache/hudi/common/table/log/block/[root@master block]# vim HoodieParquetDataBlock.java/try try(FSDataOutputStream outputStream=new FSDataOutputStream(baos,satas:null)){

4.启动

[root@master block]# spark-shell \>--jars/opt/software/hudi-spark3.1-bundle_2.12-0.12.0.jar\>--conf'spark.serializer=org.apache.spark.serializer.KryoSerializer'\>--conf'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension'2025-10-2020:14:29,222 INFO repl.Main: Created Spark session with Hive support Spark context Web UI available at http://master:4040 Spark context available as'sc'(master=local[*], appid=local-1760991268524). Spark session available as'spark'.Welcome to ____ __ / __/__ ___ _____/ /__ _\\/ _\/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\version3.1.1 /_/ Using Scala version2.12.10(Java HotSpot(TM)64-Bit Server VM, Java1.8.0_371)Typeinexpressions to have them evaluated. Type :helpformoreinformation. scala>

四.flink的搭建

1.将需要的包导入master节点并且解压

[root@localhost ~]# docker cp flink-1.14.0-bin-scala_2.12.tgz master:/root[root@master ~]# tar -zxvf flink-1.14.0-bin-scala_2.12.tgz -C /opt/module/[root@master ~]# cd /opt/moduleroot@master module]#:mv flink-1.14.0 flink

2.配置环境变量

[root@master ~]# vim /etc/profileexportFLINK_HOME=/opt/module/flinkexportPATH=$PATH:$FLINK_HOME/binexportHADOOP_CLASSPATH=$(hadoop classpath)[root@master module]# source /etc/profile[root@master ~]# flink --versionSLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found bindingin[jar:file:/opt/module/flink/lib/log4j-slf4j-impl-2.14.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found bindingin[jar:file:/opt/module/hadoop-3.2.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is oftype[org.apache.logging.slf4j.Log4jLoggerFactory]Version:1.14.0, Commit ID: 460b386

3.配置相关环境

[root@master ~]# cd /opt/module/flink/conf/[root@master conf]# vim flink-conf.yamlclassloader.check-leaked-classloader:falsejobmanager.rpc.address: master[root@master conf]# vim mastersmaster:8081[root@master conf]# vim workersmaster[root@master conf]# vim zoo.cfgserver.1=master:2888:3888

4把master节点的传给其他两个节点

5.测试

[root@master conf]# flink run -m yarn-cluster -p 2 -yjm 2G -ytm 2G $FLINK_HOME/examples/batch/WordCount.jar(rather,1)(regard,1)(remember,1)(resolution,1)(respect,1)(rub,1)(sins,1)(sleep,5)(slings,1)(so,1)(soft,1)(spurns,1)(suffer,1)(sweat,1)(take,1)(that,7)(their,1)(there,2)(thought,1)(time,1)(to,15)(under,1)(undiscover,1)(we,4)(weary,1)(when,2)(whips,1)(who,2)(whose,1)(will,1)(wish,1)(would,2)(wrong,1)(you,1)