sqoop 安装

Posted hello-wei

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了sqoop 安装相关的知识,希望对你有一定的参考价值。

Sqoop是一款开源的工具,主要用于在Hadoop(Hive)与传统的数据库(mysql、postgresql...)间进行数据的传递,可以将一个关系型数据库(例如 : MySQL ,Oracle ,Postgres等)中的数据导进到Hadoop的HDFS中,也可以将HDFS的数据导进到关系型数据库中。Sqoop项目开始于2009年,最早是作为Hadoop的一个第三方模块存在,后来为了让使用者能够快速部署,也为了让开发人员能够更快速的迭代开发,Sqoop独立成为一个Apache项目。
总之Sqoop是一个转换工具,用于在关系型数据库与HDFS之间进行数据转换。

技术图片

 

sqoop 安装步骤如下:

1.下载,指定到目录下

下载路径:https://mirrors.tuna.tsinghua.edu.cn/apache/sqoop/1.4.7/

选择版本:sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz

安装在master主节点上。

解压:gunzip -d sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz

          tar -xvf sqoop-1.4.7.bin__hadoop-2.6.0.tar

mv sqoop-1.4.7.bin__hadoop-2.6.0 sqoop-1.4.7

cd sqoop-1.4.7/

cp sqoop-env-template.sh  sqoop-env.sh

vi sqoop-env.sh     --根据具体内容填写

# Set Hadoop-specific environment variables here.

#Set path to where bin/hadoop is available
#export HADOOP_COMMON_HOME=/home/hadoop/hadoop-2.7.3

#Set path to where hadoop-*-core.jar is available
#export HADOOP_MAPRED_HOME=/home/hadoop/hadoop-2.7.3

#set the path to where bin/hbase is available
#export HBASE_HOME=/home/hadoop/hbase

#Set the path to where bin/hive is available
#export HIVE_HOME=/home/hadoop/hive

#Set the path for where zookeper config dir is
#export ZOOCFGDIR=/home/hadoop/zookeeper

2.添加环境变量:

vi .bash_profile

export SQOOP_HOME=/home/hadoop/sqoop-1.4.7
export PATH=$PATH:${SQOOP_HOME}/bin
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export CLASSPATH=$CLASSPATH:${SQOOP_HOME}/lib

使文件生效:

source .bash_profile

 

3.复制相关依赖包$SQOOP_HOME/lib

下载MySQL的依赖包

mysql-connector-java-5.1.46-bin.jar 点击打开链接

上传解压后,把mysql-connector-java-5.1.46-bin.jar 移动到/home/hadoop/sqoop-1.4.7/lib 下

cd /home/hadoop/hadoop-2.7.3/share/hadoop/common

cp hadoop-common-2.7.3.jar /home/hadoop/sqoop-1.4.7/lib/

 

4.修改$SQOOP_HOME/bin/configure-sqoop

注释掉HCatalog,Accumulo检查(除非你准备使用HCatalog,Accumulo等HADOOP上的组件)    

## Moved to be a runtime check in sqoop.
#if [ ! -d "${HCAT_HOME}" ]; then
#  echo "Warning: $HCAT_HOME does not exist! HCatalog jobs will fail."
#  echo Please set $HCAT_HOME to the root of your HCatalog installation.
#fi


#if[ ! -d "${ACCUMULO_HOME}" ]; then
#  echo "Warning: $ACCUMULO_HOME does notexist! Accumulo imports will fail."
#  echo Please set $ACCUMULO_HOME to the rootof your Accumulo installation.
#fi
 

#Add HCatalog to dependency list
#if[ -e "${HCAT_HOME}/bin/hcat" ]; then
# TMP_SQOOP_CLASSPATH=${SQOOP_CLASSPATH}:`${HCAT_HOME}/bin/hcat-classpath`
#  if [ -z "${HIVE_CONF_DIR}" ]; then
#   TMP_SQOOP_CLASSPATH=${TMP_SQOOP_CLASSPATH}:${HIVE_CONF_DIR}
#  fi
#  SQOOP_CLASSPATH=${TMP_SQOOP_CLASSPATH}
#fi
 

#Add Accumulo to dependency list
#if[ -e "$ACCUMULO_HOME/bin/accumulo" ]; then
#  for jn in `$ACCUMULO_HOME/bin/accumuloclasspath | grep file:.*accumulo.*jar |cut -d: -f2`; do
#    SQOOP_CLASSPATH=$SQOOP_CLASSPATH:$jn
#  done
#  for jn in `$ACCUMULO_HOME/bin/accumuloclasspath | grep file:.*zookeeper.*jar |cut -d: -f2`; do
#    SQOOP_CLASSPATH=$SQOOP_CLASSPATH:$jn
#  done
#fi

 

测试与mysql的连接

首先确保mysqld在运行:

[[email protected] ~]# service mysqld status
mysqld (pid  3052) is running...

然后测试是否连通:

[[email protected] ~]$ sqoop list-databases --connect jdbc:mysql://127.0.0.1:3306/?useSSL=false --username root -P
19/02/18 17:38:32 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
Enter password:
19/02/18 17:38:45 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
information_schema
hive
mysql
performance_schema
sys

 

输入密码后如果能显示你mysql上的数据库则表示已经连通。

 

<完>

以上是关于sqoop 安装的主要内容,如果未能解决你的问题,请参考以下文章

Sqoop环境搭建 01sqoop-1.4.7 安装配置CentOS Linux release 7.5.1804(附Sqoop1最新版+Sqoop2最新版安装包+MySQL驱动包资源)

Sqoop--安装配置

安装sqoop,给bin/sqoop-list-databases: line 26: /Users/apple/sqoop: 是一个目录

安装sqoop

Sqoop2安装

Sqoop的安装部署