安装hadoop下的sqoop1.99.3及配置问题全解决
Posted
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了安装hadoop下的sqoop1.99.3及配置问题全解决相关的知识,希望对你有一定的参考价值。
2016年8月27日,解决sqoop
先下载的是sqoop1.99.7,结果配置的时候,出现了没有server/conf目录,以及启动时出现无法配置错误
./bin/sqoop.sh server start
[email protected]:/opt/sqoop-1.99.7$ ./bin/sqoop.sh server start
Setting conf dir: ./bin/../conf
Sqoop home directory: /opt/sqoop-1.99.7
Starting the Sqoop2 server...
Sqoop2 server started.
[email protected]:/opt/sqoop-1.99.7$ 0 [main] INFO org.apache.sqoop.core.SqoopServer - Initializing Sqoop server.
58 [main] INFO org.apache.sqoop.core.PropertiesConfigurationProvider - Starting config file poller thread
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:38)
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:36)
at org.apache.hadoop.security.UserGroupInformation$UgiMetrics.create(UserGroupInformation.java:120)
at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:236)
at org.apache.sqoop.security.authentication.SimpleAuthenticationHandler.secureLogin(SimpleAuthenticationHandler.java:39)
at org.apache.sqoop.security.AuthenticationManager.initialize(AuthenticationManager.java:98)
at org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:57)
at org.apache.sqoop.server.SqoopJettyServer.<init>(SqoopJettyServer.java:67)
at org.apache.sqoop.server.SqoopJettyServer.main(SqoopJettyServer.java:177)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration.Configuration
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 9 more
解决办法是下载sqoop1.99.3,然后哪找如下配置,搞掂:
2.解压文件到工作目录:
hadoop@hadoopMaster:$ sudo tar -xvf /opt/hn/hadoop_family/sqoop-1.99.3-bin-hadoop200.tar.gz
hadoop@hadoopMaster:mv /opt/hn/hadoop_family/sqoop-1.99.3-bin-hadoop200 /usr/local/sqoop
3.修改环境变量:
hadoop@hadoopMaster:~$ vim /etc/profile
添加如下内容:
#sqoop
export SQOOP_HOME=/usr/sqoop-1.99.3
export PATH=$SQOOP_HOME/bin:$PATH
export CATALINA_HOME=$SQOOP_HOME/server
export LOGDIR=$SQOOP_HOME/logs
保存退出即时生效:
source /etc/profile
4.修改sqoop配置:
hadoop@hadoopMaster:~$ vim /usr/local/sqoop/server/conf/sqoop.properties
#修改指向我的hadoop的配置文件目录
org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/usr/local/hadoop-2.6.2/etc/hadoop
#把hadoop目录下的jar包都引进来
hadoop@hadoopMaster:~$ vim /usr/local/sqoop/server/conf/catalina.properties
common.loader=/usr/local/hadoop/share/hadoop/common/*.jar,/usr/local/hadoop/share/hadoop/common/lib/*.jar,/usr/local/hadoop/share/hadoop/hdfs/*.jar,/usr/local/hadoop/share/hadoop/hdfs/lib/*.jar,/usr/local/hadoop/share/hadoop/mapreduce/*.jar,/usr/local/hadoop/share/hadoop/mapreduce/lib/*.jar,/usr/local/hadoop/share/hadoop/tools/*.jar,/usr/local/hadoop/share/hadoop/tools/lib/*.jar,/usr/local/hadoop/share/hadoop/yarn/*.jar,/usr/local/hadoop/share/hadoop/yarn/lib/*.jar,/usr/local/hadoop/share/hadoop/httpfs/tomcat/lib/*.jar
5.下载mysql驱动包
mysql-connector-java-5.1.16-bin.jar
6.启动/停止sqoop200
hadoop@hadoopMaster:/usr/local/sqoop/bin$ ./sqoop.sh server start/stop
跑起来试试:
[email protected]:/opt/sqoop-1.99.3$ ./bin/sqoop.sh server start
Sqoop home directory: /opt/sqoop-1.99.3
Setting SQOOP_HTTP_PORT: 12000
Setting SQOOP_ADMIN_PORT: 12001
Using CATALINA_OPTS:
Adding to CATALINA_OPTS: -Dsqoop.http.port=12000 -Dsqoop.admin.port=12001
Using CATALINA_BASE: /opt/sqoop-1.99.3/server
Using CATALINA_HOME: /opt/sqoop-1.99.3/server
Using CATALINA_TMPDIR: /opt/sqoop-1.99.3/server/temp
Using JRE_HOME: /usr/java/jdk1.7.0_79/jre
Using CLASSPATH: /opt/sqoop-1.99.3/server/bin/bootstrap.jar
以上是关于安装hadoop下的sqoop1.99.3及配置问题全解决的主要内容,如果未能解决你的问题,请参考以下文章
甘道夫Sqoop1.99.3基础操作--导入Oracle的数据到HDFS
无法让 Sqoop 1.99.3 在 64 位 Centos 6.5 上与 Apache Hadoop 2.4.0 一起使用