Windows配置本地Hadoop运行环境

Posted 凯文の博客

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了Windows配置本地Hadoop运行环境相关的知识,希望对你有一定的参考价值。

 

很多人喜欢用Windows本地开发Hadoop程序,这里是一个在Windows下配置Hadoop的教程。

首先去官网下载hadoop,这里需要下载一个工具winutils,这个工具是编译hadoop用的,下载完之后解压hadoop文件,然后把winutils.exe放到hadoop文件的bin目录下面

然后在hadoop/etc/hadoop下修改以下文件:

core-site.xml:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000/</value>
  </property>
  <property>
    <name>io.native.lib.available</name>
    <value>false</value>
  </property>
  <property>
    <name>hadoop.native.lib</name>
    <value>false</value>
  </property>
  <property>
    <name>io.compression.codecs</name>
    <value>org.apache.hadoop.io.compress.GzipCodec,
           org.apache.hadoop.io.compress.DefaultCodec,
           com.hadoop.compression.lzo.LzoCodec,
           com.hadoop.compression.lzo.LzopCodec,
           org.apache.hadoop.io.compress.BZip2Codec,
           org.apache.hadoop.io.compress.SnappyCodec
        </value>
</property>
<property>
    <name>io.compression.codec.lzo.class</name>
    <value>com.hadoop.compression.lzo.LzoCodec</value>
</property>

</configuration>

hdfs-site.xml:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>  
       <property>  
               <name>dfs.replication</name>  
                <value>1</value>  
       </property>
       <property>
                <name>dfs.namenode.name.dir</name>  
               <value>file:///D:/Hadoop/namenode</value>  
       </property>
       <property>
               <name>dfs.datanode.data.dir</name>  
               <value>file:///D:/Hadoop/datanode</value>  
       </property>
</configuration>  

mapred-site.xml:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapred.compress.map.output</name>
        <value>true</value>
    </property>
    <property>
        <name>mapred.map.output.compression.codec</name>
        <value>com.hadoop.compression.lzo.LzoCodec</value>
    </property> 
    <property> 
        <name>mapred.child.env</name> 
        <value>LD_LIBRARY_PATH=</value>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapred.compress.map.output</name>
        <value>true</value>
    </property>
    <property>
        <name>mapred.map.output.compression.codec</name>
        <value>com.hadoop.compression.lzo.LzoCodec</value>
    </property> 
    <property> 
        <name>mapred.child.env</name> 
        <value>LD_LIBRARY_PATH=D:\\hadoop-2.7.3-win64\\lib</value> 
    </property>
</configuration>

然后cmd到hadoop的bin目录下执行:

hdfs namenode -format

然后在sbin目录下执行:

start-all.cmd

 

然后浏览器打开http://localhost:8088:

执行hadoop命令:hadoop fs -ls /

空的,新建一个文件夹:hadoop fs -mkdir /data 

然后查看:hadoop fs -ls /

这样就hadoop的本地伪分布式环境就配置好了。

 

以上是关于Windows配置本地Hadoop运行环境的主要内容,如果未能解决你的问题,请参考以下文章

Hadoop环境搭建

windows 本地配置hadoop客户端

MapReduce两种执行环境介绍:本地测试环境,服务器环境

Windows开发Flink,Spark环境配置

Hadoop 7MapReduce执行环境配置

windows下本地调试hadoop代码,远程调试hadoop节点。