关于hadoop单机配置问题执行hadoop namenode -format并start-all.sh后用jps查看并没有启动成功

Posted 2023-05-02

[hadoop@localhost hadoop]$ start-all.sh
starting namenode, logging to /var/log/hadoop/hadoop/hadoop-hadoop-namenode-localhost.localdomain.out
hadoop@localhost's password:
localhost: starting datanode, logging to /var/log/hadoop/hadoop/hadoop-hadoop-datanode-localhost.localdomain.out
hadoop@localhost's password:
localhost: starting secondarynamenode, logging to /var/log/hadoop/hadoop/hadoop-hadoop-secondarynamenode-localhost.localdomain.out
localhost: Exception in thread "main" java.lang.NullPointerException
localhost: at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:136)
localhost: at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:176)
localhost: at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:206)
localhost: at org.apache.hadoop.hdfs.server.namenode.NameNode.getServiceAddress(NameNode.java:200)
localhost: at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:161)
localhost: at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
localhost: at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
starting jobtracker, logging to /var/log/hadoop/hadoop/hadoop-hadoop-jobtracker-localhost.localdomain.out
hadoop@localhost's password:
localhost: starting tasktracker, logging to /var/log/hadoop/hadoop/hadoop-hadoop-tasktracker-localhost.localdomain.out
[hadoop@localhost hadoop]$ jps
9997 Jps

参考技术A “hadoop@localhost's password:”
ssh互信没有建立起来。
执行
ssh-keygen -t dsa出现提示都回车就行。
cd ~/.ssh
cat id_dsa.pub >> authorized_keys
然后ssh localhost如果不需要密码就是设置成功了。本回答被提问者采纳

Hadoop单机Hadoop测试环境搭建

Hadoop单机Hadoop测试环境搭建：
1. 安装jdk，并配置环境变量,配置ssh免密码登录
2. 下载安装包hadoop-2.7.3.tar.gz
3. 配置/etc/hosts   127.0.0.1 YARN001
4. 解压缩hadoop-2.7.3.tar.gtz到/home/zhangzhenghai/cluster目录下
5. 配置etc/hadoop/hadoop-env.sh 配置JAVA_HOME环境变量
export JAVA_HOME=/home/zhangzhenghai/soft/jdk1.8.0_121
6. 配置四个配置文件
etc/hadoop/mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>
etc/hadoop/core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://YARN001:8020</value>
    </property>
</configuration>
etc/hadoop/hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/home/zhangzhenghai/cluster/hadoop-2.7.3/dfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>/home/zhangzhenghai/cluster/hadoop-2.7.3/dfs/data</value>
    </property>
</configuration>
etc/hadoop/yarn-site.xml
<?xml version="1.0"?>
<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
</configuration>
7. etc/hadoop/slaves配置文件
$more etc/hadoop/slaves
YARN001
8. 格式化NameNode（仅在搭建的时候格式化；若常规启动集群需要跳过此步骤）
bin/hadoop namenode -format
9. 启动方法三种，建议一个一个启动
sbin/hadoop-daemon.sh start namenode
sbin/hadoop-daemon.sh start datanode
sbin/yarn-daemon.sh start resourcemanager
sbin/yarn-daemon.sh start nodemanager
10. 检查各进程是否启动jps
28819 DataNode
29348 NodeManager
29065 ResourceManager
28269 NameNode
29487 Jps
11. web检查NameNode是否启动
http://localhost:50070
http://test:8088
12. 验证机群，跑一个测试用例<br>　　bin/yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar pi 2 10