Hadoop-自动安装脚本
Posted Triumph-CP
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Hadoop-自动安装脚本相关的知识，希望对你有一定的参考价值。
Hadoop-自动安装脚本
自动安装jdk
创建shellfile文件夹
mkdir shellfile
创建autoinstall.sh
vim autoinstall.sh
#! /bin/bash
echo 'auto install begining....'
给予权限
chmod 777 ./autoinstall.sh
执行
./autoinstall.sh
执行后显示
继续编写autoinstall.sh
#! /bin/bash
echo 'auto install begining....'
#global var
jdk=true
if [ "$jdk" = true ];then
echo 'jdk install set true'
echo 'setup jdk 8'
tar -zxf /opt/install/jdk-8u321-linux-x64.tar.gz -C /opt/soft/
mv /opt/soft/jdk1.8.0_321 /opt/soft/jdk180
sed -i '73a\\export PATH=$PATH:$JAVA_HOME/bin' /etc/profile
sed -i '73a\\export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar' /etc/profile
sed -i '73a\\export JAVA_HOME=/opt/soft/jdk180' /etc/profile
sed -i '73a\\#jdk' /etc/profile
echo 'setip jdk 8 success!!!'
fi
hadoop自动安装脚本
#! /bin/bash
echo 'auto install hadoop begining....'
#global var
hadoop=true
if [ "$hadoop" = true ];then
echo 'hadoop install set true'
echo 'setup hadoop 3'
tar -zxf /opt/install/hadoop-3.1.3.tar.gz -C /opt/soft/
mv /opt/soft/hadoop-3.1.3 /opt/soft/hadoop313
sed -i '73a\\export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop' /etc/profile
sed -i '73a\\export HADOOP_LIBEXEC_DIR=$HADOOP_HOME/libexec' /etc/profile
sed -i '73a\\export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native' /etc/profile
sed -i '73a\\export HADOOP_INSTALL=$HADOOP_HOME' /etc/profile
sed -i '73a\\export HADOOP_YARN_HOME=$HADOOP_HOME' /etc/profile
sed -i '73a\\export HADOOP_HDFS_HOME=$HADOOP_HOME' /etc/profile
sed -i '73a\\export HADOOP_COMMON_HOME=$HADOOP_HOME' /etc/profile
sed -i '73a\\export HADOOP_MAPRED_HOME=$HADOOP_HOME' /etc/profile
sed -i '73a\\export YARN_NODEMANAGER_USER=root' /etc/profile
sed -i '73a\\export YARN_RESOURCEMANAGER_USER=root' /etc/profile
sed -i '73a\\export HDFS_ZKFC_USER=root' /etc/profile
sed -i '73a\\export HDFS_JOURNALNODE_USER=root' /etc/profile
sed -i '73a\\export HDFS_SECONDARYNAMENODE_USER=root' /etc/profile
sed -i '73a\\export HDFS_DATANODE_USER=root' /etc/profile
sed -i '73a\\export HDFS_NAMENODE_USER=root' /etc/profile
sed -i '73a\\export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/lib' /etc/profile
sed -i '73a\\export HADOOP_HOME=/opt/soft/hadoop313' /etc/profile
sed -i '73a\\#HADOOP_HOME' /etc/profile
#core-site
sed -i '19a\\ </property>' /opt/soft/hadoop313/etc/hadoop/core-site.xml
sed -i '19a\\ <value>*</value>' /opt/soft/hadoop313/etc/hadoop/core-site.xml
sed -i '19a\\ <name>hadoop.proxyuser.root.groups</name>' /opt/soft/hadoop313/etc/hadoop/core-site.xml
sed -i '19a\\ <property>' /opt/soft/hadoop313/etc/hadoop/core-site.xml
sed -i '19a\\ </property>' /opt/soft/hadoop313/etc/hadoop/core-site.xml
sed -i '19a\\ <value>*</value>' /opt/soft/hadoop313/etc/hadoop/core-site.xml
sed -i '19a\\ <name>hadoop.proxyuser.root.hosts</name>' /opt/soft/hadoop313/etc/hadoop/core-site.xml
sed -i '19a\\ <property>' /opt/soft/hadoop313/etc/hadoop/core-site.xml
sed -i '19a\\ </property>' /opt/soft/hadoop313/etc/hadoop/core-site.xml
sed -i '19a\\ <description>读写队列缓存：128k</description>' /opt/soft/hadoop313/etc/hadoop/core-site.xml
sed -i '19a\\ <value>131072</value>' /opt/soft/hadoop313/etc/hadoop/core-site.xml
sed -i '19a\\ <name>io.file.buffer.size</name>' /opt/soft/hadoop313/etc/hadoop/core-site.xml
sed -i '19a\\ <property>' /opt/soft/hadoop313/etc/hadoop/core-site.xml
sed -i '19a\\ </property>' /opt/soft/hadoop313/etc/hadoop/core-site.xml
sed -i '19a\\ <value>root</value>' /opt/soft/hadoop313/etc/hadoop/core-site.xml
sed -i '19a\\ <name>hadoop.http.staticuser.user</name>' /opt/soft/hadoop313/etc/hadoop/core-site.xml
sed -i '19a\\ <property>' /opt/soft/hadoop313/etc/hadoop/core-site.xml
sed -i '19a\\ </property>' /opt/soft/hadoop313/etc/hadoop/core-site.xml
sed -i '19a\\ <description>namenode上本地的hadoop临时文件夹</description>' /opt/soft/hadoop313/etc/hadoop/core-site.xml
sed -i '19a\\ <value>/opt/soft/hadoop313/data</value>' /opt/soft/hadoop313/etc/hadoop/core-site.xml
sed -i '19a\\ <name>hadoop.tmp.dir</name>' /opt/soft/hadoop313/etc/hadoop/core-site.xml
sed -i '19a\\ <property>' /opt/soft/hadoop313/etc/hadoop/core-site.xml
sed -i '19a\\ </property>' /opt/soft/hadoop313/etc/hadoop/core-site.xml
sed -i '19a\\ <value>hdfs://cp145:9000</value>' /opt/soft/hadoop313/etc/hadoop/core-site.xml
sed -i '19a\\ <name>fs.defaultFS</name>' /opt/soft/hadoop313/etc/hadoop/core-site.xml
sed -i '19a\\ <property>' /opt/soft/hadoop313/etc/hadoop/core-site.xml
#hadoop-env.sh
sed -i '53a\\export JAVA_HOME=/opt/soft/jdk180' /opt/soft/hadoop313/etc/hadoop/hadoop-env.sh
#hdfs-site.xml
sed -i '19a\\ </property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xml
sed -i '19a\\ <description>关闭权限验证</description>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xml
sed -i '19a\\ <value>false</value>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xml
sed -i '19a\\ <name>dfs.permissions.enabled</name>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xml
sed -i '19a\\ <property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xml
sed -i '19a\\ </property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xml
sed -i '19a\\ <description>datanode上数据块的物理存储位置目录</description>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xml
sed -i '19a\\ <value>/opt/soft/hadoop313/data/dfs/data</value>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xml
sed -i '19a\\ <name>dfs.datanode.data.dir</name>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xml
sed -i '19a\\ <property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xml
sed -i '19a\\ </property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xml
sed -i '19a\\ <description>namenode上存储hdfsq名字空间元数据的目录</description>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xml
sed -i '19a\\ <value>/opt/soft/hadoop313/data/dfs/name</value>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xml
sed -i '19a\\ <name>dfs.namenode.name.dir</name>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xml
sed -i '19a\\ <property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xml
sed -i '19a\\ </property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xml
sed -i '19a\\ <description>hadoop中每一个block文件的备份数量</description>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xml
sed -i '19a\\ <value>1</value>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xml
sed -i '19a\\ <name>dfs.replication</name>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xml
sed -i '19a\\ <property>' /opt/soft/hadoop313/etc/hadoop/hdfs-site.xml
#mapred-site.xml
sed -i '19a\\ </property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xml
sed -i '19a\\ <value>1024</value>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xml
sed -i '19a\\ <name>mapreduce.reduce.memory.mb</name>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xml
sed -i '19a\\ <property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xml
sed -i '19a\\ </property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xml
sed -i '19a\\ <value>1024</value>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xml
sed -i '19a\\ <name>mapreduce.map.memory.mb</name>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xml
sed -i '19a\\ <property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xml
sed -i '19a\\ </property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xml
sed -i '19a\\ <value>cp145:19888</value>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xml
sed -i '19a\\ <name>mapreduce.jobhistory.webapp.address</name>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xml
sed -i '19a\\ <property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xml
sed -i '19a\\ </property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xml
sed -i '19a\\ <value>cp145:10020</value>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xml
sed -i '19a\\ <name>mapreduce.jobhistory.address</name>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xml
sed -i '19a\\ <property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xml
sed -i '19a\\ </property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xml
sed -i '19a\\ <value>/opt/soft/hadoop313/etc/hadoop:/opt/soft/hadoop313/share/hadoop/common/lib/*:/opt/soft/hadoop313/share/hadoop/common/*:/opt/soft/hadoop313/share/hadoop/hdfs/*:/opt/soft/hadoop313/share/hadoop/hdfs/lib/*:/opt/soft/hadoop313/share/hadoop/mapreduce/*:/opt/soft/hadoop313/share/hadoop/mapreduce/lib/*:/opt/soft/hadoop313/share/hadoop/yarn/*:/opt/soft/hadoop313/share/hadoop/yarn/lib/*</value>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xml
sed -i '19a\\ <name>mapreduce.application.classpath</name>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xml
sed -i '19a\\ <property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xml
sed -i '19a\\ </property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xml
sed -i '19a\\ <final>true</final>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xml
sed -i '19a\\ <description>job执行框架：local,classic or yarn</description>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xml
sed -i '19a\\ <value>yarn</value>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xml
sed -i '19a\\ <name>mapreduce.framework.name</name>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xml
sed -i '19a\\ <property>' /opt/soft/hadoop313/etc/hadoop/mapred-site.xml
#yarn-site.xml
sed -i '17a\\ </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
sed -i '17a\\ <value>/opt/soft/hadoop313/yarndata/log</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
sed -i '17a\\ <name>yarn.nodemanager.log-dirs</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
sed -i '17a\\ <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
sed -i '17a\\ </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
sed -i '17a\\ <value>/opt/soft/hadoop313/yarndata/yarn</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
sed -i '17a\\ <name>yarn.nodemanager.local-dirs</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
sed -i '17a\\ <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
sed -i '17a\\ </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
sed -i '17a\\ <value>mapreduce_shuffle</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
sed -i '17a\\ <name>yarn.nodemanager.aux-services</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
sed -i '17a\\ <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
sed -i '17a\\ </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
sed -i '17a\\ <value>cp145:8042</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
sed -i '17a\\ <name>yarn.nodemanager.webapp.address</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
sed -i '17a\\ <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
sed -i '17a\\ </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
sed -i '17a\\ <value>cp145:8050</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
sed -i '17a\\ <name>yarn.nodemanager.address</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
sed -i '17a\\ <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
sed -i '17a\\ </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
sed -i '17a\\ <value>cp145:8040</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
sed -i '17a\\ <name>yarn.nodemanager.localizer.address</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
sed -i '17a\\ <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
sed -i '17a\\ </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
sed -i '17a\\ <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
sed -i '17a\\ <name>yarn.resourcemanager.scheduler.class</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
sed -i '17a\\ <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
sed -i '17a\\ </property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
sed -i '17a\\ <value>20000</value>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
sed -i '17a\\ <name>yarn.resourcemanager.connect.retry-interval.ms</name>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
sed -i '17a\\ <property>' /opt/soft/hadoop313/etc/hadoop/yarn-site.xml
echo 'setup hadoop 3 success!!!'
fi
记得将cp145改成自己的ip、路径改成自己的路径
写完初始化
[root@cp145 hadoop]# source /etc/profile
[root@cp145 hadoop]# echo $HADOOP_HOME
/opt/soft/hadoop313
[root@cp145 hadoop]# hdfs namenode -format
start-all.sh即可成功
以上是关于Hadoop-自动安装脚本的主要内容，如果未能解决你的问题，请参考以下文章