Hadoop集群安装与配置

Posted 2021-02-18 果园里的码农

1、安装包下载

下载地址：https://hadoop.apache.org/releases.html

 2、解压

tar -zxvf hadoop-3.2.1.tar.gz -C ~/soft/app/

3、修改配置文件

3.1、配置hadoop-env.sh

export JAVA_HOME=/root/opt/jdk1.8.0_221
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root

3.2、配置core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://centos1:9000</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/root/opt/hadoop/data/tmp</value>
    </property>
</configuration>

3.3、配置hdfs-site.xml

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/root/opt/hadoop/data/namenode</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>/root/opt/hadoop/data/datanode</value>
    </property>
</configuration>

3.4、配置mapred-site.xml

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.admin.user.env</name>
        <value>HADOOP_MAPRED_HOME=$HADOOP_COMMON_HOME</value>
    </property>
    <property>
        <name>yarn.app.mapreduce.am.env</name>
        <value>HADOOP_MAPRED_HOME=$HADOOP_COMMON_HOME</value>
    </property>
</configuration>

3.5、配置yarn-site.xml

<configuration>
    <property>
         <name>yarn.resourcemanager.hostname</name>
        <value>centos3</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-service</name>
        <value>mapreduce_shuffle</value>
    </property>
    
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>centos1:8032</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>centos1:8030</value>
    </property>
    <property>
     <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>centos1:8031</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>centos1:8033</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>centos1:8088</value>
    </property>
</configuration>

3.6、配置workers

centos1
centos2
centos3

4、将配置好的hadoop分发到各个节点

scp -r /root/app/hadoop centos2:/root/app/
scp -r /root/app/hadoop centos3:/root/app/

5、配置环境变量

vim /etc/profile

export HADOOP_HOME=/root/soft/app/hadoop/hadoop-3.2.1
export PATH=$JAVA_HOME/bin:$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

source /etc/profile

6、集群初始化与启动

1. 格式化：

   [root@centos1 hadoop-3.2.1]# bin/hdfs namenode -format 

2. 启动：

   [root@bs1 hadoop-3.2.1]# sbin/start-dfs.sh
   [root@bs1 hadoop-3.2.1]# sbin/start-yarn.sh

3. 验证：

   [root@centos1 hadoop-3.2.1]# jps