Installing Hadoop in Docker

Posted 2020-12-02 xiaobin-hlj80

一、Running in the host

1. pull centos

$docker pull centos:7

假设得到的容器ID为：8164e2537625

$docker run -it -v /docker:/root/store1 centos:7 /bin/bash

二、Container Running

1. install tools

[[email protected] /]yum install -y wget vim openssh-server openssh-clients net-tools

2. install jdk

    1) down

    download to tmp folder:

[[email protected] tmp]wget --no-check-certificate -c --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u171-b11/512cd62ec5174c3487ac17c61aaa89e8/jdk-8u171-linux-x64.tar.gz

   unzip:

[[email protected] tmp]mkdir /usr/java
[[email protected] tmp]tar zxvf jdk-8u171-linux-x64.tar.gz -C /usr/java

 

    2) setup

[[email protected] tmp]echo ‘export JAVA_HOME=/usr/java/jdk1.8.0_171‘ >> /etc/bashrc
[[email protected] tmp]echo ‘export CLASSPATH=$JAVA_HOME/lib/:.‘ >> /etc/bashrc
[[email protected] tmp]echo ‘export PATH=$PATH:$JAVA_HOME/bin‘ >> /etc/bashrc
[[email protected] tmp]source /etc/bashrc

   3) test

[[email protected] tmp]java -version

 

3. install hadoop

    1) down

     http://hadoop.apache.org/releases.html

     Select a mirrors download：

[[email protected] tmp]wget http://mirrors.hust.edu.cn/apache/hadoop/common/hadoop-2.8.4/hadoop-2.8.4.tar.gz

    unzip:

[[email protected] tmp]mkdir /usr/local/hadoop
[[email protected] tmp]tar zxvf hadoop-2.8.4.tar.gz -C /usr/local/hadoop/

 

    2) setup

[[email protected] tmp]echo ‘export HADOOP_HOME=/usr/local/hadoop/hadoop-2.8.4‘ >> /etc/bashrc
[[email protected] tmp]echo ‘export HADOOP_CONFIG_HOME=$HADOOP_HOME/etc/hadoop‘ >> /etc/bashrc
[[email protected] tmp]echo ‘export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin‘ >> /etc/bashrc
[[email protected] tmp]source /etc/bashrc

 

    3) config hadoop

[[email protected] tmp]cd $HADOOP_CONFIG_HOME

 

    (1) core-site.xml

<configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/usr/local/hadoop/hadoop-2.8.4/tmp</value>
        <description>A base for other temporary dirctories.</description>
    </property>

    <property>
        <name>fs.default.name</name>
        <value>hdfs://master:9000</value>
        <final>true</final>
        <description>The name of the default file system.
            A URI whose scheme and authority determine the FileSystem implemntation.
            The uri‘s scheme determines the config property (fs.SCHEME.impl) naming the FileSystem implemnetation class.
            The uri‘s authority is used to determine the host, port, etc. for a filesystem.
        </description>
    </property>
</configuration>

 

        (2) hdfs-site.xml

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>2</value>
        <final>true</final>
        <description>Default block replication.
            The actual number of replications can be specified when the file is created.
            The default is used if replication is not specified in create time.
        </description>
    </property>

    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/usr/local/hadoop/hadoop-2.8.4/namenode</value>
        <final>true</final>
    </property>

    <property>
        <name>dfs.datenode.data.dir</name>
        <value>/usr/local/hadoop/hadoop-2.8.4/datanode</value>
        <final>true</final>
    </property>
</configuration>

 

        (3) mapred-site.xml

        copy template:

[[email protected] hadoop]cp mapred-site.xml.template mapred-site.xml

       write it:

<configuration>
    <property>
        <name>maperd.job.tracker</name>
        <value>master:9001</value>
        <description>The host and port that the MapReduce job tracker runs at.
            IF "local", then jobs are run in-process as a single map and reduce task</description>
    </property>
</configuration>

 

 

 

Reference:

    1. 《Downloading Java JDK on Linux via wget is shown license page instead》

    2. 《vmware虚拟机redhat7.2下docker容器安装hadoop - v2.7.2》 

    3. 《基于 Docker 安装 Hadoop 2.9.0 - 陈刚》