gRPC over TLS 使用 Netty/Kortlin/Ktor 引发错误并阻止 gNMI 工作

Posted 2023-04-15

【中文标题】gRPC over TLS 使用 Netty/Kortlin/Ktor 引发错误并阻止 gNMI 工作

【英文标题】：gRPC over TLS using Netty/Kortlin/Ktor raises error and prevents gNMI to work

【发布时间】：2019-04-13 06:17:40

【问题描述】：

目前正在使用 gRPC/gNMI + Kotlin 和 Ktor HTTP 开发一个模块。 卡在安全的依赖库 tc-native netty-tcnative-boringssl-static 2.0.19-Final

Ktor 在 1.0.0-beta3 中

  // ktor
    implementation "io.ktor:ktor-server-core:$ktor_version"
    implementation "io.ktor:ktor-server-netty:$ktor_version"
    implementation "io.ktor:ktor-html-builder:$ktor_version"
    implementation "io.ktor:ktor-gson:$ktor_version"
    implementation "io.ktor:ktor-metrics:$ktor_version"
    implementation "io.ktor:ktor-locations:$ktor_version"
    implementation "ch.qos.logback:logback-classic:$logback_version"

    // tpl
    implementation "io.ktor:ktor-freemarker:$ktor_version"
    implementation "org.freemarker:freemarker:$freemarker_version"

    // Security
    //implementation "io.netty:netty-tcnative:$netty_tcnative_version"
    implementation "io.netty:netty-tcnative-boringssl-static:$netty_tcnative_version".

使用 gRPC 请求时

fun Route.routeFilesystem() 
get("/test") 
    call.respondHtml 
        head 
            title  +"gRPC/gNMI" 
        
        body 
            p 
                +"[ Capabilities Test ] "
                TLSConnection("xxx.xxx.xxx.xxx",8448).capabilities()
            
        
    

它给出以下错误（堆栈跟踪）：

[DEBUG] 2018-11-09 16:50:36.789 [nettyCallPool-4-4] DEBUG i.n.u.internal.NativeLibraryLoader - netty_tcnative_linux_x86_64 cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
java.lang.UnsatisfiedLinkError: no netty_tcnative_linux_x86_64 in java.library.path
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
        at java.lang.Runtime.loadLibrary0(Runtime.java:870)
        at java.lang.System.loadLibrary(System.java:1122)
        at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
        at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:243)
        at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:124)
        at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:85)
        at io.netty.handler.ssl.OpenSsl.loadTcNative(OpenSsl.java:440)
        at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:97)
        at io.grpc.netty.GrpcSslContexts.defaultSslProvider(GrpcSslContexts.java:244)
        at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:171)
        at io.grpc.netty.GrpcSslContexts.forClient(GrpcSslContexts.java:120)
        at sdn.client.cli.console.TLSConnection.<init>(TLSConnection.kt:38)
        at sdn.client.ui.ApplicationKt$routeFilesystem$1$1$2$1.invoke(Application.kt:125)
        at sdn.client.ui.ApplicationKt$routeFilesystem$1$1$2$1.invoke(Application.kt)
        at kotlinx.html.ApiKt.visit(api.kt:80)
        at kotlinx.html.Gen_tag_groupsKt.p(gen-tag-groups.kt:125)
        at kotlinx.html.Gen_tag_groupsKt.p$default(gen-tag-groups.kt:125)
        at .sdn.client.ui.ApplicationKt$routeFilesystem$1$1$2.invoke(Application.kt:123)
        at .sdn.client.ui.ApplicationKt$routeFilesystem$1$1$2.invoke(Application.kt)
        at kotlinx.html.ApiKt.visit(api.kt:80)
        at kotlinx.html.Gen_tags_hKt.body(gen-tags-h.kt:164)
        at kotlinx.html.Gen_tags_hKt.body$default(gen-tags-h.kt:164)
        at .sdn.client.ui.ApplicationKt$routeFilesystem$1$1.invoke(Application.kt:122)
        at sdn.client.ui.ApplicationKt$routeFilesystem$1$1.invoke(Application.kt)
        at kotlinx.html.ApiKt.visit(api.kt:80)
        at kotlinx.html.ApiKt.visitAndFinalize(api.kt:93)
        at kotlinx.html.Gen_consumer_tagsKt.html(gen-consumer-tags.kt:317)
        at kotlinx.html.Gen_consumer_tagsKt.html$default(gen-consumer-tags.kt:317)
        at io.ktor.html.HtmlContent.writeTo(RespondHtml.kt:32)
        at io.ktor.server.engine.BaseApplicationResponse$respondWriteChannelContent$2.invokeSuspend(BaseApplicationResponse.kt:155)
        at io.ktor.server.engine.BaseApplicationResponse$respondWriteChannelContent$2.invoke(BaseApplicationResponse.kt)
        at io.ktor.util.cio.ReadersKt.use(Readers.kt:33)
        at io.ktor.server.engine.BaseApplicationResponse.respondWriteChannelContent$suspendImpl(BaseApplicationResponse.kt:149)
        at io.ktor.server.engine.BaseApplicationResponse.respondWriteChannelContent(BaseApplicationResponse.kt)
        at io.ktor.server.engine.BaseApplicationResponse.respondOutgoingContent$suspendImpl(BaseApplicationResponse.kt:119)
        at io.ktor.server.engine.BaseApplicationResponse.respondOutgoingContent(BaseApplicationResponse.kt)
        at io.ktor.server.netty.NettyApplicationResponse.respondOutgoingContent$suspendImpl(NettyApplicationResponse.kt:37)
        at io.ktor.server.netty.NettyApplicationResponse.respondOutgoingContent(NettyApplicationResponse.kt)
        at io.ktor.server.engine.BaseApplicationResponse$$special$$inlined$apply$lambda$1.invokeSuspend(BaseApplicationResponse.kt:36)
        at io.ktor.server.engine.BaseApplicationResponse$$special$$inlined$apply$lambda$1.invoke(BaseApplicationResponse.kt)
        at io.ktor.util.pipeline.PipelineContext.proceed(PipelineContext.kt:53)
        at io.ktor.util.pipeline.PipelineContext.proceedWith(PipelineContext.kt:39)
        at io.ktor.server.engine.DefaultTransformKt$installDefaultTransformations$1.invokeSuspend(DefaultTransform.kt:26)
        at io.ktor.server.engine.DefaultTransformKt$installDefaultTransformations$1.invoke(DefaultTransform.kt)
        at io.ktor.util.pipeline.PipelineContext.proceed(PipelineContext.kt:53)
        at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:24)
        at io.ktor.html.RespondHtmlKt.respondHtml(RespondHtml.kt:37)
        at io.ktor.html.RespondHtmlKt.respondHtml$default(RespondHtml.kt:15)
        at sdn.client.ui.ApplicationKt$routeFilesystem$1.invokeSuspend(Application.kt:116)
        at .sdn.client.ui.ApplicationKt$routeFilesystem$1.invoke(Application.kt)
        at io.ktor.util.pipeline.PipelineContext.proceed(PipelineContext.kt:53)
        at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:24)
        at io.ktor.routing.Routing.executeResult(Routing.kt:110)
        at io.ktor.routing.Routing.interceptor(Routing.kt:29)
        at io.ktor.routing.Routing$Feature$install$1.invokeSuspend(Routing.kt:75)
        at io.ktor.routing.Routing$Feature$install$1.invoke(Routing.kt)
        at io.ktor.util.pipeline.PipelineContext.proceed(PipelineContext.kt:53)
        at io.ktor.features.ContentNegotiation$Feature$install$1.invokeSuspend(ContentNegotiation.kt:60)
        at io.ktor.features.ContentNegotiation$Feature$install$1.invoke(ContentNegotiation.kt)
        at io.ktor.util.pipeline.PipelineContext.proceed(PipelineContext.kt:53)
        at io.ktor.features.StatusPages$intercept$3.invokeSuspend(StatusPages.kt:88)
        at io.ktor.features.StatusPages$intercept$3.invoke(StatusPages.kt)
        at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:82)
        at kotlinx.coroutines.CoroutineScopeKt.coroutineScope(CoroutineScope.kt:162)
        at io.ktor.features.StatusPages.intercept(StatusPages.kt:87)
        at io.ktor.features.StatusPages$Feature$install$1.invokeSuspend(StatusPages.kt:121)
        at io.ktor.features.StatusPages$Feature$install$1.invoke(StatusPages.kt)
        at io.ktor.util.pipeline.PipelineContext.proceed(PipelineContext.kt:53)
        at io.ktor.features.CallLogging$Feature$install$2.invokeSuspend(CallLogging.kt:124)
        at io.ktor.features.CallLogging$Feature$install$2.invoke(CallLogging.kt)
        at io.ktor.util.pipeline.PipelineContext.proceed(PipelineContext.kt:53)
        at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:24)
        at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invokeSuspend(DefaultEnginePipeline.kt:80)
        at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invoke(DefaultEnginePipeline.kt)
        at io.ktor.util.pipeline.PipelineContext.proceed(PipelineContext.kt:53)
        at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:24)
        at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invokeSuspend(NettyApplicationCallHandler.kt:31)
        at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invoke(NettyApplicationCallHandler.kt)
        at kotlinx.coroutines.intrinsics.UndispatchedKt.startCoroutineUndispatched(Undispatched.kt:54)
        at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:111)
        at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:160)
        at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch(Builders.common.kt:54)
        at kotlinx.coroutines.BuildersKt.launch(Unknown Source)
        at io.ktor.server.netty.NettyApplicationCallHandler.handleRequest(NettyApplicationCallHandler.kt:22)
        at io.ktor.server.netty.NettyApplicationCallHandler.channelRead(NettyApplicationCallHandler.kt:16)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38)
        at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:464)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)
        Suppressed: java.lang.UnsatisfiedLinkError: no netty_tcnative_linux_x86_64 in java.library.path
                at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
                at java.lang.Runtime.loadLibrary0(Runtime.java:870)
                at java.lang.System.loadLibrary(System.java:1122)
                at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:498)
                at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:263)
                at java.security.AccessController.doPrivileged(Native Method)
                at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:255)
                at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:233)
                ... 90 common frames omitted

请问有什么办法解决这个问题吗？ 谢谢

【参考方案1】：

见https://github.com/netty/netty-tcnative/issues/331

这是一个 DEBUG 级别的打印，这表明它可能不是一个实际的错误，只是 netty 尝试加载但未能加载的东西。加载boringssl .so 以后可能会成功。