使用 KafkaConnect JdbcDriver 和 Oracle 11g(Strimzi 运算符)时出现无用的 ORA 错误

Posted

技术标签:

【中文标题】使用 KafkaConnect JdbcDriver 和 Oracle 11g(Strimzi 运算符)时出现无用的 ORA 错误【英文标题】:Non useful ORA error when working with KafkaConnect JdbcDriver with Oracle 11g (Strimzi Operator) 【发布时间】:2020-10-26 22:09:09 【问题描述】:

我正在使用 srimzi Kafka 运算符与 Confluent 集群一起使用来自 Confluent 的 JdbcSourceConnector 来实现 Oracle2Kafka 类型的 KafkaConnector。

KafkaConnector 规范

# connector 
connection.url: jdbc:oracle:thin:@HOST:PORT/SERVICE
connection.user: USER
connection.password: PASS
dialect.name: OracleDatabaseDialect
topic.prefix: test-topic-
mode: bulk
db.timezone: Europe/Madrid
table.whitelist: TEST_TABLE

但我在 srimzi-cluster-operator 日志中收到以下 错误

io.strimzi.operator.cluster.operator.assembly.ConnectRestException: PUT /connectors/confluent-cluster-int-20200706-02/config returned 400 (Bad Request): Connector configuration is invalid and contains the following 2 error(s):
Invalid value java.sql.SQLException: ORA-00604: error occurred at recursive SQL level 1
ORA-06502: PL/SQL: numeric or value error: character string buffer too small
ORA-06512: at line 43
 for configuration Couldn't open connection to jdbc:oracle:thin:@***:1534/***
Invalid value java.sql.SQLException: ORA-00604: error occurred at recursive SQL level 1
ORA-06502: PL/SQL: numeric or value error: character string buffer too small
ORA-06512: at line 43
 for configuration Couldn't open connection to jdbc:oracle:thin:@***:1534/***
You can also find the above list of errors at the endpoint `/connector-plugins/connectorType/config/validate`
    at io.strimzi.operator.cluster.operator.assembly.KafkaConnectApiImpl.lambda$null$2(KafkaConnectApi.java:208) ~[io.strimzi.cluster-operator-0.18.0.jar:0.18.0]

如果我修改源代码,我会得到一个非常具体的堆栈跟踪

2020-07-14 09:33:46,636 ERROR SQLException (io.confluent.connect.jdbc.source.JdbcSourceConnectorConfig) [qtp742672280-21]
java.sql.SQLException: ORA-00604: error occurred at recursive SQL level 1
ORA-06502: PL/SQL: numeric or value error: character string buffer too small
ORA-06512: at line 43

    at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:509)
    at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:456)
    at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:451)
    at oracle.jdbc.driver.T4CTTIfun.processError(T4CTTIfun.java:1123)
    at oracle.jdbc.driver.T4CTTIoauthenticate.processError(T4CTTIoauthenticate.java:552)
    at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:553)
    at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:269)
    at oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:501)
    at oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:1292)
    at oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:1025)
    at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:747)
    at oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:793)
    at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:57)
    at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:747)
    at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:562)
    at java.sql.DriverManager.getConnection(DriverManager.java:664)
    at java.sql.DriverManager.getConnection(DriverManager.java:208)
    at io.confluent.connect.jdbc.dialect.GenericDatabaseDialect.getConnection(GenericDatabaseDialect.java:223)
    at io.confluent.connect.jdbc.source.JdbcSourceConnectorConfig$TableRecommender.validValues(JdbcSourceConnectorConfig.java:606)
    at io.confluent.connect.jdbc.source.JdbcSourceConnectorConfig$CachingRecommender.validValues(JdbcSourceConnectorConfig.java:653)
    at org.apache.kafka.common.config.ConfigDef.validate(ConfigDef.java:607)
    at org.apache.kafka.common.config.ConfigDef.validate(ConfigDef.java:622)
    at org.apache.kafka.common.config.ConfigDef.validate(ConfigDef.java:530)
    at org.apache.kafka.common.config.ConfigDef.validateAll(ConfigDef.java:513)
    at org.apache.kafka.common.config.ConfigDef.validate(ConfigDef.java:495)
    at org.apache.kafka.connect.connector.Connector.validate(Connector.java:135)

我已经检查过的一些事情

KafkaConnect 图像是自制的,带有 JdbcSourceConnector 所需的插件和 OracleDriver

Dockerfile

FROM strimzi/kafka:0.18.0-kafka-2.5.0
USER root:root
COPY ./kafka-connect-jdbc-5.4.0.jar /opt/kafka/plugins/
COPY ./ojdbc6.jar /opt/kafka/libs/
USER 1001

KafkaConnect 资源已成功部署,因为 KafkaConnect 主题正在融合集群上填充(connect-cluster-configs、connect-cluster-configs...)

Oracle 驱动程序似乎已成功加载。如果我在凭据或连接链中添加拼写错误,则该错误是不言自明且有意义的。我还尝试了其他版本的 oracle 驱动程序。

回到过去(4 个月前),同样的配置正在运行(在本地 srimzi-deployed-cluster 和 confluent 上)。现在本地集群工作正常,但融合的集群失败并出现描述的错误。

尝试了几个升级最新版本的srimzi operator和kafka-jdbc-connector

(编辑)按照 srimzi slack 中的建议,尝试了 KafkaConnect 的 PUT /connector-plugins/JdbcSourceConnector/config/validate 休息端点,并在白名单和黑名单字段中得到了相同的 2 个错误

结果

"value": 
  "name": "table.blacklist",
  "value": "",
  "recommended_values": [],
  "errors": ["Invalid value java.sql.SQLException: ORA-00604: error occurred at recursive SQL level 1\nORA-06502: PL/SQL: numeric or value error: character string buffer too small\nORA-06512: at line 43\n for configuration Couldn't open connection to jdbc:oracle:thin:@***:1534/***"],
"visible": true

(编辑)我试图将白名单字段留空,但错误是相同的。数据库似乎没有更改,并且连接链从源代码 spring-data 访问工作正常。

我没有想法,欢迎任何提示x)

【问题讨论】:

【参考方案1】:

我不知道你使用的工具,但是 - 你得到的错误意味着:

SQL> declare
  2    l_var varchar2(1);       -- note length, only 1 character
  3  begin
  4    l_var := 'Littlefoot';   -- I'm little, but can't fit
  5  end;
  6  /
declare
*
ERROR at line 1:
ORA-06502: PL/SQL: numeric or value error: character string buffer too small
ORA-06512: at line 4


SQL>

怎么办?

SQL> declare
  2    l_var varchar2(20);      -- is that enough?
  3  begin
  4    l_var := 'Littlefoot'; 
  5  end;
  6  /

PL/SQL procedure successfully completed.         --> Yes, it is!

SQL>

因此,请检查代码中的 尺寸

【讨论】:

是的,在原始 Oracle 上下文中,我已经检查了那个 ORA 错误的含义,但这对于只查询表的 KafkaConnect 来说没有意义,这就是为什么我把“无用的 ora标题中的错误”。我试图了解它的来源..

以上是关于使用 KafkaConnect JdbcDriver 和 Oracle 11g(Strimzi 运算符)时出现无用的 ORA 错误的主要内容,如果未能解决你的问题,请参考以下文章

KafkaConnect Offset存储设计

kafka connect 使用说明

Kafka核心API——Connect API

Kafka核心API——Connect API

使用 KafkaConnect JdbcDriver 和 Oracle 11g(Strimzi 运算符)时出现无用的 ORA 错误

Kafka Connect:使用 debezium 从 Postgres 流式传输更改到主题