HDFS 搭建记录

Posted 2020-08-04 番茄瓜园

1. 三台服务：

172.17.0.62（namenode）

172.17.0.68（datanode）

172.17.0.76（datanode）

/etc/hosts包含的内容：

三台都包含的域名映射：（注意：域名中不要有下划线（_），否则 datanode起不来，呵呵：。。）

172.17.0.76  tomcattomcatmobileplatformtest.bjdv   #不要有下划线（_）

172.17.0.62  tomcatmobileplatform.bjdv

172.17.0.68  mobileplatformweblogic.bjdv

#各个服务器映射自己的主机名

127.0.0.1 cd6d8c70882c

 

2. namenode与datanode配置互相的不用密码的ssh，以及登陆localhost的不用密码的ssh

每台服务器上执行一下：#localhost的ssh

1）ssh-keygen -t dsa -P ‘‘ -f /root/.ssh/id_dsa

2）cat /root/id_dsa.pub >> /root/.ssh/authorized_keys

3）chmod 0600 ~/.ssh/authorized_keys

把namenode和datanode上的id_dsa.pub拷到对方，并在对方上执行一下2，3  #namenode与datanode互相的ssh

 

3.下载并拷贝hadoop(本文使用的是hadoop-2.7.2)到三台服务器。

4.配置文件：

etc/hadoop/core-site.xml：

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
        <name>fs.defaultFS</name>
        <value>hdfs://172.17.0.62:9000</value>
</property>
</configuration>

etc/hadoop/hadoop-env.sh：

加入JAVA_HOME,HADOOP_PID_DIR

export JAVA_HOME=/root/jdk1.7.0_60
export HADOOP_PID_DIR=/george/h/pids

etc/hadoop/hdfs-site.xml:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/george/namenode</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>/george/datanode</value>
    </property>
</configuration>

etc/hadoop/slaves:

172.17.0.68
172.17.0.76

 

namenode服务器上：

# sbin/start-dfs.sh

检查服务

# jps
27831 Jps
27353 NameNode
27584 SecondaryNameNode

 

# bin/hdfs dfsadmin -report
-bash: ./hadoop-2.7.2/bin/hdfs: No such file or directory
[email protected]:/george/hadoop-2.7.2# ./bin/hdfs dfsadmin -report
Configured Capacity: 20869324800 (19.44 GB)
Present Capacity: 14557261824 (13.56 GB)
DFS Remaining: 14557155328 (13.56 GB)
DFS Used: 106496 (104 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

-------------------------------------------------
Live datanodes (2):

Name: 172.17.0.68:50010 (mobileplatformweblogic.bjdv)
Hostname: localhost
Decommission Status : Normal
Configured Capacity: 10434662400 (9.72 GB)
DFS Used: 49152 (48 KB)
Non DFS Used: 4756729856 (4.43 GB)
DFS Remaining: 5677883392 (5.29 GB)
DFS Used%: 0.00%
DFS Remaining%: 54.41%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Mon Aug 29 14:55:37 CST 2016


Name: 172.17.0.76:50010 (tomcattomcatmobileplatformtest.bjdv)
Hostname: localhost
Decommission Status : Normal
Configured Capacity: 10434662400 (9.72 GB)
DFS Used: 57344 (56 KB)
Non DFS Used: 1555333120 (1.45 GB)
DFS Remaining: 8879271936 (8.27 GB)
DFS Used%: 0.00%
DFS Remaining%: 85.09%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Mon Aug 29 14:55:38 CST 2016

（ NameNode 的webUI信息 似乎不正确，只能看到一个Datanode， http://172.17.0.62:50070/dfshealth.html， 上面的 bin/hdfs dfsadmin -report 是正确的）

测试：

# bin/hadoop fs -put LICENSE.txt /

 

datanode服务器上： 

检查服务

# jps 
21622 DataNode
21774 Jps

 

日志查看：

# tail -f logs/hadoop-root-datanode-cd6d8c70882c.log

 

datanode上：

没有压缩的情况下，往hdfs上传文件后在其中的一个datanode上可以看到（如：blk_1073741828），dfs.replication设置为1，所以在所有datanode上只有一份文件。

# ls -l datanode/current/BP-333745209-172.17.0.62-1472441549732/current/finalized/subdir0/subdir0/
total 8
-rw-r--r-- 1 root root 1366 Aug 29 14:55 blk_1073741828
-rw-r--r-- 1 root root   19 Aug 29 14:55 blk_1073741828_1004.meta

 

以上是自己搭建hdfs时测试与总结，有不对的地方，欢迎指正。