Hadoop 2.9 多节点

Posted 2023-04-15

技术标签:

【中文标题】Hadoop 2.9 多节点【英文标题】：Hadoop 2.9 MultiNodes 【发布时间】：2018-10-18 15:37:48 【问题描述】：

我有 3 台服务器 Centos 7（防火墙和 selinux 已禁用） chadoop1（主）、chadoop2（从）和chadoop3（从）

当我启动服务时，节点没有启动，我在 jps 上看到，不显示 DataNode 和 NodeManager。

所有配置在节点上都是 rsync（从属除外）

我尝试重新格式化，显示 OK，但同样的问题。

我的目录是：/opt/hadoop

配置：

hdfs-site.xml

<configuration>
    <property>
            <name>dfs.data.dir</name>
            <value>/opt/hadoop/dfs/name/data</value>
            <final>true</final>
    </property>
    <property>
            <name>dfs.name.dir</name>
            <value>/opt/hadoop/dfs/name</value>
            <final>true</final>
    </property>
    <property>
            <name>dfs.replication</name>
            <value>2</value>
    </property>

核心站点.xml

<configuration>

<property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:8020/</value>
    <description>NameNode URI</description>
</property>

<property>
  <name>io.file.buffer.size</name>
  <value>131072</value>
  <description>Buffer size</description>
</property>

mapred-site.xml

<configuration>

<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
    <description>MapReduce framework name</description>
</property>

<property>
  <name>mapreduce.jobhistory.address</name>
  <value>localhost:10020</value>
  <description>Default port is 10020.</description>
</property>

<property>
  <name>mapreduce.jobhistory.webapp.address</name>
  <value>localhost:19888</value>
  <description>Default port is 19888.</description>
</property>

<property>
  <name>mapreduce.jobhistory.intermediate-done-dir</name>
  <value>/mr-history/tmp</value>
  <description>Directory where history files are written by MapReduce jobs.</description>
</property>

<property>
  <name>mapreduce.jobhistory.done-dir</name>
  <value>/mr-history/done</value>
  <description>Directory where history files are managed by the MR JobHistory Server.</description>
</property>

yarn-site.xml

<configuration>

<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
    <description>MapReduce framework name</description>
</property>

<property>
  <name>mapreduce.jobhistory.address</name>
  <value>localhost:10020</value>
  <description>Default port is 10020.</description>
</property>

<property>
  <name>mapreduce.jobhistory.webapp.address</name>
  <value>localhost:19888</value>
  <description>Default port is 19888.</description>
</property>

<property>
  <name>mapreduce.jobhistory.intermediate-done-dir</name>
  <value>/mr-history/tmp</value>
  <description>Directory where history files are written by MapReduce jobs.</description>
</property>

<property>
  <name>mapreduce.jobhistory.done-dir</name>
  <value>/mr-history/done</value>
  <description>Directory where history files are managed by the MR JobHistory Server.</description>
</property>

slave（仅在 master 上，在 slave 中具有 localhost）

chadoop3
chadoop4

启动服务

[hadoop@chadoop1 hadoop]$ start-dfs.sh
 Starting namenodes on [localhost]
 localhost: starting namenode, logging to /opt/hadoop/logs/hadoop-hadoop- 
 namenode-chadoop1.out
 chadoop4: starting datanode, logging to /opt/hadoop/logs/hadoop-hadoop- 
 datanode-chadoop4.out
 chadoop3: starting datanode, logging to /opt/hadoop/logs/hadoop-hadoop-                    
 datanode-chadoop3.out
 Starting secondary namenodes [0.0.0.0]
 0.0.0.0: starting secondarynamenode, logging to /opt/hadoop/logs/hadoop- 
 hadoop-secondarynamenode-chadoop1.out

 [hadoop@chadoop1 hadoop]$ jps
 5603 Jps
 5492 SecondaryNameNode
 5291 NameNode
 [hadoop@chadoop1 hadoop]$ start-yarn.sh
 starting yarn daemons
 starting resourcemanager, logging to /opt/hadoop/logs/yarn-hadoop-               
 resourcemanager-chadoop1.out
 chadoop3: starting nodemanager, logging to /opt/hadoop/logs/yarn-hadoop- 
 nodemanager-chadoop3.out
 chadoop4: starting nodemanager, logging to /opt/hadoop/logs/yarn-hadoop- 
 nodemanager-chadoop4.out
 [hadoop@chadoop1 hadoop]$ jps
 5492 SecondaryNameNode
 5658 ResourceManager
 5914 Jps
 5291 NameNode

【问题讨论】：

请在问题中添加/opt/hadoop/logs/hadoop-hadoop-datanode-*.out 文件。注意：切勿在 XML 中使用 localhost。始终使用外部主机或 IP 名称感谢本地主机信息。 hadoop-hadoop-datanode-chadoop1.log（结束文件）2018-10-18 09:03:35,211 INFO org.apache.hadoop.hdfs.server.datanode.DataNode：获得块池 BP-1848867256-127.0 的最终确定命令。 0.1-153978980$ 2018-10-18 11:14:03,411 错误 org.apache.hadoop.hdfs.server.datanode.DataNode：收到信号 15：SIGTERM 2018-10-18 11:14:03,787 信息 org.apache.hadoop .hdfs.server.datanode.DataNode: SHUTDOWN_MSG: SHUTDOWN_MSG: 在 localhost/127.0.0.1 关闭 DataNode RECEIVED SIGNAL 15: SIGTERM 表示它从某个外部进程关闭，不一定是任何错误。您可以查看其他 2 台机器的错误日志 【参考方案1】：