Sqoop 与 SQL Anywhere 17 连接

Posted 2023-04-18

【中文标题】Sqoop 与 SQL Anywhere 17 连接

【英文标题】：Sqoop connect with SQL Anywhere 17

【发布时间】：2019-10-22 13:32:04

【问题描述】：

我正在尝试让 sqoop 与 sqlanywhere17 的 sajdbc4.jar 一起工作。我不断收到命令库错误，无法找到 jar 所需的 .so 相关文件。

现在，我在/usr/hdp/current/sqoop-client/lib/ 内部创建了一个指向sajdbc4.jar -> /opt/sqlanywhere17/java/sajdbc4.jar 的符号链接。

ln -s  /opt/sqlanywhere17/java/sajdbc4.jar /usr/hdp/current/sqoop-client/lib/sajdbc4.jar 

我收到此错误：

线程“主”java.lang.UnsatisfiedLinkError 中的异常：java.library.path 中没有 dbjdbc17

关注this guide和this documentation之后：

cat /opt/sqlanywhere17/bin64/sa_config.sh >> ~/.bashrc
cat /opt/sqlanywhere17/bin64/sa_config.sh >> ~/.bash_profile

并检查echo $LD_LIBRARY_PATH ：

/opt/sqlanywhere17/lib64:/opt/sqlanywhere17/bin64/jre180/lib/amd64/client:/opt/sqlanywhere17/bin64/jre180/lib/amd64/server:/opt/sqlanywhere17/bin64/jre180/lib/amd64:/opt/sqlanywhere17/bin64/jre180/lib/amd64/native_threads:

并检查 java.library.path 是否有：/opt/sqlanywhere17/lib64

命令行：

  java -XshowSettings:properties

---

Property settings:
    awt.toolkit = sun.awt.X11.XToolkit
    com.sap.vm.codeline = 81_REL
    com.sap.vm.compressedoops = true
    com.sap.vm.profilingserver = true
    com.sap.vm.type = opt
    com.sap.vm.version = 10
    file.encoding = UTF-8
    file.encoding.pkg = sun.io
    file.separator = /
    java.awt.graphicsenv = sun.awt.X11GraphicsEnvironment
    java.awt.printerjob = sun.print.PSPrinterJob
    java.class.path = .
    java.class.version = 52.0
    java.endorsed.dirs = /opt/sqlanywhere17/bin64/jre180/lib/endorsed
    java.ext.dirs = /opt/sqlanywhere17/bin64/jre180/lib/ext-sap
        /opt/sqlanywhere17/bin64/jre180/lib/ext
        /usr/java/packages/lib/ext
    java.home = /opt/sqlanywhere17/bin64/jre180
    java.io.tmpdir = /tmp
    java.library.path = /opt/sqlanywhere17/lib64
        /opt/sqlanywhere17/bin64/jre180/lib/amd64/client
        /opt/sqlanywhere17/bin64/jre180/lib/amd64/server
        /opt/sqlanywhere17/bin64/jre180/lib/amd64
        /opt/sqlanywhere17/bin64/jre180/lib/amd64/native_threads
         
        /usr/java/packages/lib/amd64
        /usr/lib64
        /lib64
        /lib
        /usr/lib

---

完全错误：

    19/10/22 08:30:05 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.5.3.0-37
19/10/22 08:30:07 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.
19/10/22 08:30:07 INFO manager.SqlManager: Using default fetchSize of 1000
Exception in thread "main" java.lang.UnsatisfiedLinkError: no dbjdbc17 in java.library.path
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
        at java.lang.Runtime.loadLibrary0(Runtime.java:870)
        at java.lang.System.loadLibrary(System.java:1122)
        at sap.jdbc4.sqlanywhere.IDriver.try_load(IDriver.java:451)
        at sap.jdbc4.sqlanywhere.IDriver.<clinit>(IDriver.java:413)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:264)
        at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:873)
        at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
        at org.apache.sqoop.tool.EvalSqlTool.run(EvalSqlTool.java:64)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:225)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:243)
[root@server1 dag_scripts]# echo $LD_LIBRARY_PATH
/opt/sqlanywhere17/lib64:/opt/sqlanywhere17/bin64/jre180/lib/amd64/client:/opt/sqlanywhere17/bin64/jre180/lib/amd64/server:/opt/sqlanywhere17/bin64/jre180/lib/amd64:/opt/sqlanywhere17/bin64/jre180/lib/amd64/native_threads:

---

---

【参考方案1】：

这个解决方案似乎暂时解决了这个问题：

ln -s /opt/sqlanywhere17/java/sajdbc4.jar /usr/hdp/current/sqoop-client/lib/sajdbc4.jar

根据 32 位/64 位要求选择 /opt/sqlanywhere17/lib32 或 /opt/sqlanywhere17/lib64/

创建指向当前 JDK 路径和 JVM 路径的符号链接

如果您以 root 身份安装，则打开访问 /opt/sqlanywhere17/java/ 的权限

确保用户是“hadoop”组的一部分 - 对于 sqoop 文件夹权限 (.env)

将默认导出添加到运行 sqoop 命令的任何用户

 ln -s /opt/sqlanywhere17/lib64/* /usr/jdk64/jdk1.8.0_112/jre/lib/amd64/

 ln -s /opt/sqlanywhere17/lib64/* /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-1.el7_7.x86_64/jre/lib/amd64/

 chmod 755 -R /opt/sqlanywhere17/*
 usermod -a -G hadoop user

 #For user executing sqoop:
 cat /opt/sqlanywhere17/bin64/sa_config.sh >> ~/.bashrc

---

用于测试的 Sqoop 命令：

对于 sajdbc4.jar：

sqoop eval --connect 'jdbc:sqlanywhere:Host=192.168.1.11:1234;ServerName=DemoServer;DatabaseName=demo' --driver sap.jdbc4.sqlanywhere.IDriver --username testuser --P --query "SELECT * from database.table"

对于 jconn4：

你也可以把jconn4.jar放到sqoop的lib目录下。

/usr/hdp/current/sqoop-client/lib/sajdbc4.jar

https://wiki.scn.sap.com/wiki/display/SYBCON/Software+Developer+Kit+for+ASE+driver+versions+for+Windows

sqoop eval --connect 'jdbc:sybase:Tds:192.168.1.11:1234' --driver com.sybase.jdbc4.jdbc.SybDriver --username testuser --P --query "SELECT * from database.table"

---

对于在 DBCPConnectionPool 中运行 SqlAnywhere17 的 NiFi：

删除所有符号链接和.bashrc 配置文件设置，而是将 sqlanywhere (/opt/sqlanywhere17/bin64/sa_config.sh) 的配置附加到“nifi-env.sh 的 NiFi 节点高级配置片段（安全阀）”中。我能够让库正确加载。连接现在可以在 NiFi 中使用！

唯一有趣的错误是，如果我使用驱动程序创建 2 个 DBCPConnectionPool，其中一个查询将失败并显示： failed to process session due to Native Library /opt/sqlanywhere17/lib64/libdbjdbc17.so.1 already loaded in another classloader; Processor Administratively Yielded for 1 sec: java.lang.UnsatisfiedLinkError: Native Library /opt/sqlanywhere17/lib64/libdbjdbc17.so.1 already loaded in another classloader

所以目前我们使用 sajdbc4.jar 使用 1 个连接，而我们的解决方法是使用 jconn4.jar 作为另一个。