Hadoop学习之二Hadoop伪分布式安装

Posted 2021-01-31 cac2020

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Hadoop学习之二Hadoop伪分布式安装相关的知识，希望对你有一定的参考价值。

环境
　　虚拟机：VMware 10
　　Linux版本：CentOS-6.5-x86_64
　　客户端：Xshell4
　　FTP：Xftp4
jdk8
hadoop-3.1.1

伪分布式就一台机器：主节点和从节点都在一个机器上，这里我们使用：node1 192.168230.11

一、平台和软件
平台：GNU/Linux
软件：JDK+SSH+rsync+hadoop3.1.1
修改主机/etc/hosts和/etc/sysconfig/network：【切记】
192.168.230.11 node1
参考：https://www.cnblogs.com/heruiguo/p/7943006.html

1.安装JDK
参考：https://www.cnblogs.com/cac2020/p/9683212.html

2.安装ssh:
(1)查看是否已经安装

[root@node1 bin]# type ssh
ssh is hashed (/usr/bin/ssh)

(2)如果未安装进行安装

yum install -y ssh

3.安装rsync(远程同步):
(1)查看是否已经安装

[root@node1 bin]# type rsync
-bash: type: rsync: not found

(2)如果未安装进行安装

yum install -y rsync

4.免密登录(主要用于全分布模式)
(1)实现localhost免密登录
#1)生成公钥和私钥对

[root@node1 ~]# ssh-keygen -t rsa -P \'\' -f ~/.ssh/id_rsa
Generating public/private rsa key pair.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
18:e5:c1:eb:55:be:9c:27:3a:bc:c1:51:dd:8e:2c:f3 root@node1
The key\'s randomart image is:
+--[ RSA 2048]----+
| .o |
| o.. .. . |
| . .. o. . .|
| o. .... o |
| ..S...ooo .|
| .. .=+. |
| .o. oE |
| +. |
| .o |
+-----------------+

#2)将生成公钥id_rsa.pub拷贝至当前目录下隐藏目录.ssh下文件authorized_keys

[root@node1 ~]# cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

#3).ssh权限改为700

[root@node1 ~]# chmod 0700 ~/.ssh

#4)authorized_keys权限改为0600

[root@node1 ~]# chmod 0600 ~/.ssh/authorized_keys

#5)登录localhost

[root@node1 ~]# echo $$
1209
[root@node1 ~]# ssh localhost
Last login: Tue Jan 1 16:17:35 2019 from 192.168.230.1
[root@node1 ~]# echo $$
1234

(2)实现node1访问node2(192.168.230.12)免密登录

#1)通过命令ssh-copy-id将node1的公钥追加到node2 注意使用 ssh-copy-id命令一定要在/root/.ssh目录下

[root@node1 .ssh]# ssh-copy-id -i ~/.ssh/id_rsa.pub 192.168.230.12
root@192.168.230.12\'s password: 
Now try logging into the machine, with "ssh \'192.168.230.12\'", and check in:

.ssh/authorized_keys

to make sure we haven\'t added extra keys that you weren\'t expecting.

#注意如果提示 -bash: ssh-copy-id: command not found //提示命令不存在

#需要安装yum -y install openssh-clients

#2)node2修改权限
#.ssh权限改为700

[root@node2 ~]# chmod 0700 ~/.ssh

#3)authorized_keys权限改为0600

[root@node2 ~]# chmod 0600 ~/.ssh/authorized_keys

#4)免密登录

[root@node1 ~]# ssh 192.168.230.12
Last login: Tue Jan 1 16:57:09 2019 from 192.168.230.11
[root@node2 ~]#

5.安装Hadoop

[root@node1 src]# tar -xf hadoop-3.1.1.tar.gz -C /usr/local

二、配置Hadoop
1、修改/usr/local/hadoop-3.1.1/etc/hadoop/hadoop-env.sh 设置JAVA环境变量、角色用户
在最后添加如下设置：

export JAVA_HOME=/usr/local/jdk1.8.0_65
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root

2、修改/usr/local/hadoop-3.1.1/etc/hadoop/core-site.xml 配置主节点相关信息

(1)fs.defaultFS 主节点通讯信息（hadoop3默认端口改为9820）
(2)hadoop.tmp.dir 设置namenode元数据和datanode block数据的目录

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://node1:9820</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/data/hadoop</value>
    </property>
</configuration>

3、修改/usr/local/hadoop-3.1.1/etc/hadoop/hdfs-site.xml 配置从节点相关信息
(1)dfs.replication 副本数
(2)dfs.namenode.secondary.http-address 二级namenode （hadoop默认端口改为9868）

<property>
    <name>dfs.replication</name>
    <value>1</value>
</property>
<property>
    <name>dfs.namenode.secondary.http-address</name>
    <value>node1:9868</value>
</property>

4、修改workers(hadoop2.X叫slave)配置从节点（DATANODE）信息
配置:node1

三、启动Hadoop
1、格式化namenode
1)将hadoop.tmp.dir指定的目录格式化，如果没有先创建，然后将元信息fsimge存到该目录下
2)生成全局唯一的集群ID 所以搭建集群时只执行一次

[root@node1 /]# /usr/local/hadoop-3.1.1/bin/hdfs namenode -format
2019-01-02 10:33:06,541 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = node1/192.168.230.11
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 3.1.1
STARTUP_MSG:   classpath = /usr/local/hadoop-3.1.1/etc/hadoop:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/jackson-core-2.7.8.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/kerby-xdr-1.0.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/snappy-java-1.0.5.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/jersey-servlet-1.19.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/commons-lang3-3.4.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/jetty-http-9.3.19.v20170502.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/kerby-asn1-1.0.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/accessors-smart-1.2.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/netty-3.10.5.Final.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/commons-io-2.5.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/kerb-common-1.0.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/curator-client-2.12.0.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/kerby-util-1.0.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/asm-5.0.4.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/stax2-api-3.1.4.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/commons-math3-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/kerb-server-1.0.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/json-smart-2.3.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/hadoop-annotations-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/kerby-pkix-1.0.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/htrace-core4-4.1.0-incubating.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/jettison-1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/commons-net-3.6.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/javax.servlet-api-3.1.0.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/woodstox-core-5.0.3.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/gson-2.2.4.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/jetty-servlet-9.3.19.v20170502.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/commons-compress-1.4.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/commons-lang-2.6.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/curator-framework-2.12.0.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/jetty-util-9.3.19.v20170502.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/httpcore-4.4.4.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/kerb-core-1.0.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/kerby-config-1.0.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/jsr311-api-1.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/kerb-admin-1.0.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/jsch-0.1.54.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/jetty-io-9.3.19.v20170502.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/metrics-core-3.2.4.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/token-provider-1.0.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/paranamer-2.3.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/jaxb-api-2.2.11.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/kerb-crypto-1.0.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/curator-recipes-2.12.0.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/commons-beanutils-1.9.3.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/jsr305-3.0.0.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/slf4j-api-1.7.25.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/commons-collections-3.2.2.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/jetty-security-9.3.19.v20170502.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/jetty-server-9.3.19.v20170502.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/jetty-xml-9.3.19.v20170502.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/jetty-webapp-9.3.19.v20170502.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/jul-to-slf4j-1.7.25.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/re2j-1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/commons-codec-1.11.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/guava-11.0.2.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/xz-1.0.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/jersey-core-1.19.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/jackson-annotations-2.7.8.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/jcip-annotations-1.0-1.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/commons-configuration2-2.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/nimbus-jose-jwt-4.41.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/kerb-simplekdc-1.0.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/hadoop-auth-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/kerb-util-1.0.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/kerb-identity-1.0.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/httpclient-4.5.2.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/jackson-databind-2.7.8.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/kerb-client-1.0.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/jersey-json-1.19.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/avro-1.7.7.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/zookeeper-3.4.9.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/lib/jersey-server-1.19.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/hadoop-kms-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/hadoop-common-3.1.1-tests.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/hadoop-nfs-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/common/hadoop-common-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/jackson-core-2.7.8.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/netty-all-4.0.52.Final.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/kerby-xdr-1.0.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/snappy-java-1.0.5.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/jersey-servlet-1.19.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/commons-lang3-3.4.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/jetty-http-9.3.19.v20170502.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/kerby-asn1-1.0.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/accessors-smart-1.2.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/netty-3.10.5.Final.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/commons-io-2.5.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/kerb-common-1.0.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/curator-client-2.12.0.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/kerby-util-1.0.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/asm-5.0.4.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/stax2-api-3.1.4.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/commons-math3-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/kerb-server-1.0.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/json-smart-2.3.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/hadoop-annotations-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/kerby-pkix-1.0.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/htrace-core4-4.1.0-incubating.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/jettison-1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/commons-net-3.6.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/javax.servlet-api-3.1.0.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/woodstox-core-5.0.3.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/gson-2.2.4.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/jetty-servlet-9.3.19.v20170502.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/commons-compress-1.4.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/json-simple-1.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/curator-framework-2.12.0.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/jetty-util-9.3.19.v20170502.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/httpcore-4.4.4.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/kerb-core-1.0.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/kerby-config-1.0.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/jsr311-api-1.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/kerb-admin-1.0.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/jsch-0.1.54.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/jetty-io-9.3.19.v20170502.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/token-provider-1.0.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/paranamer-2.3.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/jetty-util-ajax-9.3.19.v20170502.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/okio-1.6.0.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/jaxb-api-2.2.11.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/kerb-crypto-1.0.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/curator-recipes-2.12.0.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/commons-beanutils-1.9.3.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/commons-collections-3.2.2.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/jetty-security-9.3.19.v20170502.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/jetty-server-9.3.19.v20170502.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/jetty-xml-9.3.19.v20170502.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/jetty-webapp-9.3.19.v20170502.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/re2j-1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/commons-codec-1.11.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/guava-11.0.2.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/xz-1.0.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/jersey-core-1.19.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/jackson-annotations-2.7.8.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/jcip-annotations-1.0-1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/commons-configuration2-2.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/nimbus-jose-jwt-4.41.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/kerb-simplekdc-1.0.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/okhttp-2.7.5.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/hadoop-auth-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/kerb-util-1.0.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/kerb-identity-1.0.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/httpclient-4.5.2.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/jackson-databind-2.7.8.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/kerb-client-1.0.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/jersey-json-1.19.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/avro-1.7.7.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/zookeeper-3.4.9.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/lib/jersey-server-1.19.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/hadoop-hdfs-native-client-3.1.1-tests.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/hadoop-hdfs-client-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/hadoop-hdfs-nfs-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/hadoop-hdfs-client-3.1.1-tests.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/hadoop-hdfs-rbf-3.1.1-tests.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/hadoop-hdfs-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/hadoop-hdfs-native-client-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/hadoop-hdfs-httpfs-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/hadoop-hdfs-3.1.1-tests.jar:/usr/local/hadoop-3.1.1/share/hadoop/hdfs/hadoop-hdfs-rbf-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/usr/local/hadoop-3.1.1/share/hadoop/mapreduce/lib/junit-4.11.jar:/usr/local/hadoop-3.1.1/share/hadoop/mapreduce/hadoop-mapreduce-client-uploader-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/mapreduce/hadoop-mapreduce-client-app-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/mapreduce/hadoop-mapreduce-client-core-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/mapreduce/hadoop-mapreduce-client-common-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.1-tests.jar:/usr/local/hadoop-3.1.1/share/hadoop/mapreduce/hadoop-mapreduce-client-nativetask-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn:/usr/local/hadoop-3.1.1/share/hadoop/yarn/lib/jersey-guice-1.19.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/lib/dnsjava-2.1.7.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/lib/jackson-jaxrs-base-2.7.8.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/lib/java-util-1.9.0.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/lib/jackson-module-jaxb-annotations-2.7.8.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/lib/snakeyaml-1.16.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/lib/HikariCP-java7-2.4.12.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/lib/metrics-core-3.2.4.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/lib/ehcache-3.3.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/lib/fst-2.50.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/lib/jackson-jaxrs-json-provider-2.7.8.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/lib/guice-servlet-4.0.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/lib/json-io-2.5.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/lib/swagger-annotations-1.5.4.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/lib/guice-4.0.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/lib/objenesis-1.0.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/lib/jersey-client-1.19.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/hadoop-yarn-server-router-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/hadoop-yarn-server-common-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/hadoop-yarn-services-core-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/hadoop-yarn-services-api-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/hadoop-yarn-common-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/hadoop-yarn-client-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/hadoop-yarn-api-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/hadoop-yarn-server-web-proxy-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/hadoop-yarn-registry-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/hadoop-yarn-server-nodemanager-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.1.1.jar:/usr/local/hadoop-3.1.1/share/hadoop/yarn/hadoop-yarn-server-tests-3.1.1.jar
STARTUP_MSG:   build = https://github.com/apache/hadoop -r 2b9a8c1d3a2caf1e733d57f346af3ff0d5ba529c; compiled by \'leftnoteasy\' on 2018-08-02T04:26Z
STARTUP_MSG:   java = 1.8.0_65
************************************************************/
2019-01-02 10:33:06,647 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
2019-01-02 10:33:06,681 INFO namenode.NameNode: createNameNode [-format]
2019-01-02 10:33:17,758 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Formatting using clusterid: CID-501237ab-3212-4483-a36e-a48f31345438
2019-01-02 10:33:23,541 INFO namenode.FSEditLog: Edit logging is async:true
2019-01-02 10:33:23,759 INFO namenode.FSNamesystem: KeyProvider: null
2019-01-02 10:33:23,778 INFO namenode.FSNamesystem: fsLock is fair: true
2019-01-02 10:33:23,802 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
2019-01-02 10:33:23,925 INFO namenode.FSNamesystem: fsOwner             = root (auth:SIMPLE)
2019-01-02 10:33:23,925 INFO namenode.FSNamesystem: supergroup          = supergroup
2019-01-02 10:33:23,926 INFO namenode.FSNamesystem: isPermissionEnabled = true
2019-01-02 10:33:23,926 INFO namenode.FSNamesystem: HA Enabled: false
2019-01-02 10:33:24,298 INFO common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
2019-01-02 10:33:24,405 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit: configured=1000, counted=60, effected=1000
2019-01-02 10:33:24,405 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
2019-01-02 10:33:24,419 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
2019-01-02 10:33:24,421 INFO blockmanagement.BlockManager: The block deletion will start around 2019 Jan 02 10:33:24
2019-01-02 10:33:24,424 INFO util.GSet: Computing capacity for map BlocksMap
2019-01-02 10:33:24,424 INFO util.GSet: VM type       = 64-bit
2019-01-02 10:33:24,482 INFO util.GSet: 2.0% max memory 239.8 MB = 4.8 MB
2019-01-02 10:33:24,482 INFO util.GSet: capacity      = 2^19 = 524288 entries
2019-01-02 10:33:24,582 INFO blockmanagement.BlockManager: dfs.block.access.token.enable = false
2019-01-02 10:33:24,622 INFO Configuration.deprecation: No unit for dfs.namenode.safemode.extension(30000) assuming MILLISECONDS
2019-01-02 10:33:24,622 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2019-01-02 10:33:24,622 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.min.datanodes = 0
2019-01-02 10:33:24,622 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.extension = 30000
2019-01-02 10:33:24,622 INFO blockmanagement.BlockManager: defaultReplication         = 1
2019-01-02 10:33:24,740 INFO blockmanagement.BlockManager: maxReplication             = 512
2019-01-02 10:33:24,740 INFO blockmanagement.BlockManager: minReplication             = 1
2019-01-02 10:33:24,741 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
2019-01-02 10:33:24,741 INFO blockmanagement.BlockManager: redundancyRecheckInterval  = 3000ms
2019-01-02 10:33:24,741 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
2019-01-02 10:33:24,741 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
2019-01-02 10:33:24,922 INFO util.GSet: Computing capacity for map INodeMap
2019-01-02 10:33:24,922 INFO util.GSet: VM type       = 64-bit
2019-01-02 10:33:24,922 INFO util.GSet: 1.0% max memory 239.8 MB = 2.4 MB
2019-01-02 10:33:24,922 INFO util.GSet: capacity      = 2^18 = 262144 entries
2019-01-02 10:33:24,930 INFO namenode.FSDirectory: ACLs enabled? false
2019-01-02 10:33:24,930 INFO namenode.FSDirectory: POSIX ACL inheritance enabled? true
2019-01-02 10:33:24,931 INFO namenode.FSDirectory: XAttrs enabled? true
2019-01-02 10:33:24,932 INFO namenode.NameNode: Caching file names occurring more than 10 times
2019-01-02 10:33:24,978 INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: false, skipCaptureAccessTimeOnlyChange: false, snapshotDiffAllowSnapRootDescendant: true, maxSnapshotLimit: 65536
2019-01-02 10:33:24,986 INFO snapshot.SnapshotManager: SkipList is disabled
2019-01-02 10:33:24,995 INFO util.GSet: Computing capacity for map cachedBlocks
2019-01-02 10:33:24,995 INFO util.GSet: VM type       = 64-bit
2019-01-02 10:33:24,996 INFO util.GSet: 0.25% max memory 239.8 MB = 613.8 KB
2019-01-02 10:33:24,996 INFO util.GSet: capacity      = 2^16 = 65536 entries
2019-01-02 10:33:25,022 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
2019-01-02 10:33:25,022 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
2019-01-02 10:33:25,022 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
2019-01-02 10:33:25,047 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
2019-01-02 10:33:25,048 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
2019-01-02 10:33:25,090 INFO util.GSet: Computing capacity for map NameNodeRetryCache
2019-01-02 10:33:25,091 INFO util.GSet: VM type       = 64-bit
2019-01-02 10:33:25,091 INFO util.GSet: 0.029999999329447746% max memory 239.8 MB = 73.7 KB
2019-01-02 10:33:25,091 INFO util.GSet: capacity      = 2^13 = 8192 entries
2019-01-02 10:33:25,285 INFO namenode.FSImage: Allocated new BlockPoolId: BP-600017715-192.168.230.11-1546396405239
2019-01-02 10:33:25,421 INFO common.Storage: Storage directory /data/hadoop/dfs/name has been successfully formatted.
2019-01-02 10:33:25,584 INFO namenode.FSImageFormatProtobuf: Saving image file /data/hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
2019-01-02 10:33:26,036 INFO namenode.FSImageFormatProtobuf: Image file /data/hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 386 bytes saved in 0 seconds .
2019-01-02 10:33:26,198 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
2019-01-02 10:33:26,297 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at node1/192.168.230.11
************************************************************/

2、启动hadoop

[root@node1 sbin]# /usr/local/hadoop-3.1.1/sbin/start-dfs.sh Starting namenodes on [node1] node1: Warning: Permanently added \'node1,192.168.230.11\' (RSA) to the list of known hosts. Starting datanodes Starting secondary namenodes [node1] 2019-01-02 10:52:31,822 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [root@node1 sbin]# jps 1344 NameNode 1616 SecondaryNameNode 1450 DataNode 1803 Jps [root@node1 sbin]# ss -nal State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 *:9867 *:* LISTEN 0 128 192.168.230.11:9868 *:* LISTEN 0 128 *:9870

以上是关于Hadoop学习之二Hadoop伪分布式安装的主要内容，如果未能解决你的问题，请参考以下文章

Hadoop学习之八MapReduce开发

Hadoop学习之hadoop安装JDK安装集群启动(完全分布式)

大数据学习之MapReduce基础与Yarn集群安装09

0.1_hadoop安装（伪分布式）

hadoop安装伪分布

Hadoop 伪分布模式安装