尝试使用此代码在 TLS 上运行 TLS 时，为啥会出现握手失败？

Posted 2023-02-23

【中文标题】尝试使用此代码在 TLS 上运行 TLS 时，为啥会出现握手失败？

【英文标题】：Why is there a handshake failure when trying to run TLS over TLS with this code?尝试使用此代码在 TLS 上运行 TLS 时，为什么会出现握手失败？

【发布时间】：2011-07-05 01:03:36

【问题描述】：

我尝试实现一个协议，该协议可以使用twisted.protocols.tls 在 TLS 上运行 TLS，这是一个使用内存 BIO 的 OpenSSL 接口。

我将它实现为一个协议包装器，它看起来很像一个常规的 TCP 传输，但它有 startTLS 和 stopTLS 分别添加和删除一层 TLS 的方法。这适用于 TLS 的第一层。如果我通过“本机”Twisted TLS 传输运行它，它也可以正常工作。但是，如果我尝试使用此包装器提供的 startTLS 方法添加第二个 TLS 层，则会立即出现握手错误，并且连接最终会处于某种未知的不可用状态。

让它工作的包装器和两个助手看起来像这样：

from twisted.python.components import proxyForInterface
from twisted.internet.error import ConnectionDone
from twisted.internet.interfaces import ITCPTransport, IProtocol
from twisted.protocols.tls import TLSMemoryBIOFactory, TLSMemoryBIOProtocol
from twisted.protocols.policies import ProtocolWrapper, WrappingFactory

class TransportWithoutDisconnection(proxyForInterface(ITCPTransport)):
    """
    A proxy for a normal transport that disables actually closing the connection.
    This is necessary so that when TLSMemoryBIOProtocol notices the SSL EOF it
    doesn't actually close the underlying connection.

    All methods except loseConnection are proxied directly to the real transport.
    """
    def loseConnection(self):
        pass


class ProtocolWithoutConnectionLost(proxyForInterface(IProtocol)):
    """
    A proxy for a normal protocol which captures clean connection shutdown
    notification and sends it to the TLS stacking code instead of the protocol.
    When TLS is shutdown cleanly, this notification will arrive.  Instead of telling
    the protocol that the entire connection is gone, the notification is used to
    unstack the TLS code in OnionProtocol and hidden from the wrapped protocol.  Any
    other kind of connection shutdown (SSL handshake error, network hiccups, etc) are
    treated as real problems and propagated to the wrapped protocol.
    """
    def connectionLost(self, reason):
        if reason.check(ConnectionDone):
            self.onion._stopped()
        else:
            super(ProtocolWithoutConnectionLost, self).connectionLost(reason)


class OnionProtocol(ProtocolWrapper):
    """
    OnionProtocol is both a transport and a protocol.  As a protocol, it can run over
    any other ITransport.  As a transport, it implements stackable TLS.  That is,
    whatever application traffic is generated by the protocol running on top of
    OnionProtocol can be encapsulated in a TLS conversation.  Or, that TLS conversation
    can be encapsulated in another TLS conversation.  Or **that** TLS conversation can
    be encapsulated in yet *another* TLS conversation.

    Each layer of TLS can use different connection parameters, such as keys, ciphers,
    certificate requirements, etc.  At the remote end of this connection, each has to
    be decrypted separately, starting at the outermost and working in.  OnionProtocol
    can do this itself, of course, just as it can encrypt each layer starting with the
    innermost.
    """
    def makeConnection(self, transport):
        self._tlsStack = []
        ProtocolWrapper.makeConnection(self, transport)


    def startTLS(self, contextFactory, client, bytes=None):
        """
        Add a layer of TLS, with SSL parameters defined by the given contextFactory.

        If *client* is True, this side of the connection will be an SSL client.
        Otherwise it will be an SSL server.

        If extra bytes which may be (or almost certainly are) part of the SSL handshake
        were received by the protocol running on top of OnionProtocol, they must be
        passed here as the **bytes** parameter.
        """
        # First, create a wrapper around the application-level protocol
        # (wrappedProtocol) which can catch connectionLost and tell this OnionProtocol 
        # about it.  This is necessary to pop from _tlsStack when the outermost TLS
        # layer stops.
        connLost = ProtocolWithoutConnectionLost(self.wrappedProtocol)
        connLost.onion = self
        # Construct a new TLS layer, delivering events and application data to the
        # wrapper just created.
        tlsProtocol = TLSMemoryBIOProtocol(None, connLost, False)
        tlsProtocol.factory = TLSMemoryBIOFactory(contextFactory, client, None)

        # Push the previous transport and protocol onto the stack so they can be
        # retrieved when this new TLS layer stops.
        self._tlsStack.append((self.transport, self.wrappedProtocol))

        # Create a transport for the new TLS layer to talk to.  This is a passthrough
        # to the OnionProtocol's current transport, except for capturing loseConnection
        # to avoid really closing the underlying connection.
        transport = TransportWithoutDisconnection(self.transport)

        # Make the new TLS layer the current protocol and transport.
        self.wrappedProtocol = self.transport = tlsProtocol

        # And connect the new TLS layer to the previous outermost transport.
        self.transport.makeConnection(transport)

        # If the application accidentally got some bytes from the TLS handshake, deliver
        # them to the new TLS layer.
        if bytes is not None:
            self.wrappedProtocol.dataReceived(bytes)


    def stopTLS(self):
        """
        Remove a layer of TLS.
        """
        # Just tell the current TLS layer to shut down.  When it has done so, we'll get
        # notification in *_stopped*.
        self.transport.loseConnection()


    def _stopped(self):
        # A TLS layer has completely shut down.  Throw it away and move back to the
        # TLS layer it was wrapping (or possibly back to the original non-TLS
        # transport).
        self.transport, self.wrappedProtocol = self._tlsStack.pop()

我有用于执行此操作的简单客户端和服务器程序，可从启动板 (bzr branch lp:~exarkun/+junk/onion) 获得。当我使用它两次调用上述startTLS 方法时，没有对stopTLS 的干预调用，出现此OpenSSL 错误：

OpenSSL.SSL.Error: [('SSL routines', 'SSL23_GET_SERVER_HELLO', 'unknown protocol')]

为什么会出错？

【问题讨论】：

为什么要在 TLS 中使用 TLS？

为什么不呢？ ;) 目前我实际上对此并没有真正的用途。另一个 *** 问题提示我探索它（不幸的是，该问题已被其作者删除）。但是，实现类似 Tor 的洋葱路由网络可能是一件有用的事情。

我正常的网络调试方法是'使用wireshark'，当你要查看的数据是加密的时候效果不好。能否让外层使用 TLS_RSA_WITH_NULL_MD5，以便进行有意义的数据包捕获？

好主意。我很快就会试一试。

您可能需要对其进行特殊配置。 'openssl ciphers' 不会列出它，但 'openssl ciphers NULL' 会列出。

【参考方案1】：

OnionProtocol至少有两个问题：

最内层TLSMemoryBIOProtocol变成wrappedProtocol，而它应该是最外层； ProtocolWithoutConnectionLost 不会从OnionProtocol 的堆栈中弹出任何TLSMemoryBIOProtocols，因为connectionLost 仅在FileDescriptors doRead 或doWrite 方法返回断开连接的原因之后调用。

如果不改变OnionProtocol 管理其堆栈的方式，我们无法解决第一个问题，而在我们弄清楚新的堆栈实现之前，我们无法解决第二个问题。不出所料，正确的设计是数据在 Twisted 中如何流动的直接结果，因此我们将从一些数据流分析开始。

Twisted 表示与twisted.internet.tcp.Server 或twisted.internet.tcp.Client 实例建立的连接。由于我们程序中唯一的交互发生在stoptls_client，因此我们只考虑进出Client 实例的数据流。

让我们用一个最小的LineReceiver 客户端来热身，它会回显从端口 9999 上从本地服务器接收到的行：

from twisted.protocols import basic
from twisted.internet import defer, endpoints, protocol, task

class LineReceiver(basic.LineReceiver):
    def lineReceived(self, line):
        self.sendLine(line)

def main(reactor):
    clientEndpoint = endpoints.clientFromString(
        reactor, "tcp:localhost:9999")
    connected = clientEndpoint.connect(
        protocol.ClientFactory.forProtocol(LineReceiver))
    def waitForever(_):
        return defer.Deferred()
    return connected.addCallback(waitForever)

task.react(main)

一旦建立连接，Client 就成为我们的LineReceiver 协议的传输并调解输入和输出：

来自服务器的新数据导致反应器调用Client 的doRead 方法，该方法又将收到的数据传递给LineReceiver 的dataReceived 方法。最后，当至少有一条线路可用时，LineReceiver.dataReceived 会调用 LineReceiver.lineReceived。

我们的应用程序通过调用LineReceiver.sendLine 将一行数据发送回服务器。这会在绑定到协议实例的传输上调用write，这与处理传入数据的Client 实例相同。 Client.write 安排数据由反应器发送，而Client.doWrite 实际通过套接字发送数据。

我们已准备好查看从不调用 startTLS 的 OnionClient 的行为：

OnionClients 被包裹在OnionProtocols 中，这是我们尝试嵌套 TLS 的关键。作为twisted.internet.policies.ProtocolWrapper的子类，OnionProtocol的实例是一种协议-传输三明治；它以 protocol 的形式呈现给较低级别​​的传输，并以 transport 的形式呈现给协议，它通过WrappingFactory 在连接时建立的伪装包装。

现在，Client.doRead 调用 OnionProtocol.dataReceived，它将数据代理到 OnionClient。作为OnionClient 的传输，OnionProtocol.write 接受从OnionClient.sendLine 发送的线路并将它们代理到Client，它的自己的 传输。这是ProtocolWrapper、它的封装协议和它自己的传输之间的正常交互，因此数据自然而然地往返于每一个之间。

OnionProtocol.startTLS 做了一些不同的事情。它试图在已建立协议-传输对之间插入一个新的ProtocolWrapper——恰好是TLSMemoryBIOProtocol。这似乎很简单：ProtocolWrapper 将上层协议存储为其wrappedProtocol attribute 和proxies write and other attributes down to its own transport。 startTLS 应该能够注入一个新的TLSMemoryBIOProtocol，通过在其自己的wrappedProtocol 和transport 上修补该实例来将OnionClient 包装到连接中：

def startTLS(self):
    ...
    connLost = ProtocolWithoutConnectionLost(self.wrappedProtocol)
    connLost.onion = self
    # Construct a new TLS layer, delivering events and application data to the
    # wrapper just created.
    tlsProtocol = TLSMemoryBIOProtocol(None, connLost, False)

    # Push the previous transport and protocol onto the stack so they can be
    # retrieved when this new TLS layer stops.
    self._tlsStack.append((self.transport, self.wrappedProtocol))
    ...
    # Make the new TLS layer the current protocol and transport.
    self.wrappedProtocol = self.transport = tlsProtocol

这是第一次调用startTLS 后的数据流：

正如预期的那样，传递给OnionProtocol.dataReceived 的新数据被路由到存储在_tlsStack 上的TLSMemoryBIOProtocol，后者将解密后的明文传递给OnionClient.dataReceived。 OnionClient.sendLine 还将其数据传递给TLSMemoryBIOProtocol.write，后者对其进行加密并将生成的密文发送给OnionProtocol.write，然后发送给Client.write。

不幸的是，该方案在第二次调用 startTLS 后失败。根本原因是这一行：

    self.wrappedProtocol = self.transport = tlsProtocol

每次调用startTLS 都会将wrappedProtocol 替换为innermost TLSMemoryBIOProtocol，即使Client.doRead 接收到的数据已被outermost 加密：

但是，transports 嵌套正确。 OnionClient.sendLine 只能调用其传输的write——即OnionProtocol.write——因此OnionProtocol 应将其transport 替换为最里面的TLSMemoryBIOProtocol，以确保写入连续嵌套在额外的加密层中。

那么解决办法就是保证数据依次通过_tlsStack上的firstTLSMemoryBIOProtocol到next一个，这样每一层加密的顺序与应用相反：

考虑到这一新要求，将_tlsStack 表示为列表似乎不太自然。幸运的是，线性表示传入的数据流建议了一种新的数据结构：

错误和正确的传入数据流都类似于一个单链表，wrappedProtocol 充当ProtocolWrapper 的下一个链接，protocol 充当Client 的下一个链接。该列表应从OnionProtocol 向下增长，并始终以OnionClient 结尾。出现该错误是因为违反了该排序不变量。

单链表很适合将协议推送到堆栈上，但很难将它们弹出，因为它需要从其头部向下遍历到节点才能删除。当然，每次接收到数据时都会发生这种遍历，因此关注的是额外遍历所隐含的复杂性，而不是最坏情况下的时间复杂度。幸运的是，列表实际上是双向链接的：

transport 属性将每个嵌套协议与其前身链接起来，因此transport.write 可以在最终通过网络发送数据之前依次进行较低级别的加密。我们有两个哨兵来帮助管理列表：Client 必须始终位于顶部，OnionClient 必须始终位于底部。

将两者放在一起，我们最终得到：

from twisted.python.components import proxyForInterface
from twisted.internet.interfaces import ITCPTransport
from twisted.protocols.tls import TLSMemoryBIOFactory, TLSMemoryBIOProtocol
from twisted.protocols.policies import ProtocolWrapper, WrappingFactory


class PopOnDisconnectTransport(proxyForInterface(ITCPTransport)):
    """
    LTLSMemoryBIOProtocol.loseConnection shuts down the TLS session
    and calls its own transport's CloseConnection.  A zero-length
    read also calls the transport's CloseConnection.  This proxy
    uses that behavior to invoke a Cpop callback when a session has
    ended.  The callback is invoked exactly once because
    CloseConnection must be idempotent.
    """
    def __init__(self, pop, **kwargs):
        super(PopOnDisconnectTransport, self).__init__(**kwargs)
        self._pop = pop

    def loseConnection(self):
        self._pop()
        self._pop = lambda: None


class OnionProtocol(ProtocolWrapper):
    """
    OnionProtocol is both a transport and a protocol.  As a protocol,
    it can run over any other ITransport.  As a transport, it
    implements stackable TLS.  That is, whatever application traffic
    is generated by the protocol running on top of OnionProtocol can
    be encapsulated in a TLS conversation.  Or, that TLS conversation
    can be encapsulated in another TLS conversation.  Or **that** TLS
    conversation can be encapsulated in yet *another* TLS
    conversation.

    Each layer of TLS can use different connection parameters, such as
    keys, ciphers, certificate requirements, etc.  At the remote end
    of this connection, each has to be decrypted separately, starting
    at the outermost and working in.  OnionProtocol can do this
    itself, of course, just as it can encrypt each layer starting with
    the innermost.
    """

    def __init__(self, *args, **kwargs):
        ProtocolWrapper.__init__(self, *args, **kwargs)
        # The application level protocol is the sentinel at the tail
        # of the linked list stack of protocol wrappers.  The stack
        # begins at this sentinel.
        self._tailProtocol = self._currentProtocol = self.wrappedProtocol


    def startTLS(self, contextFactory, client, bytes=None):
        """
        Add a layer of TLS, with SSL parameters defined by the given
        contextFactory.

        If *client* is True, this side of the connection will be an
        SSL client.  Otherwise it will be an SSL server.

        If extra bytes which may be (or almost certainly are) part of
        the SSL handshake were received by the protocol running on top
        of OnionProtocol, they must be passed here as the **bytes**
        parameter.
        """
        # The newest TLS session is spliced in between the previous
        # and the application protocol at the tail end of the list.
        tlsProtocol = TLSMemoryBIOProtocol(None, self._tailProtocol, False)
        tlsProtocol.factory = TLSMemoryBIOFactory(contextFactory, client, None)

        if self._currentProtocol is self._tailProtocol:
            # This is the first and thus outermost TLS session.  The
            # transport is the immutable sentinel that no startTLS or
            # stopTLS call will move within the linked list stack.
            # The wrappedProtocol will remain this outermost session
            # until it's terminated.
            self.wrappedProtocol = tlsProtocol
            nextTransport = PopOnDisconnectTransport(
                original=self.transport,
                pop=self._pop
            )
            # Store the proxied transport as the list's head sentinel
            # to enable an easy identity check in _pop.
            self._headTransport = nextTransport
        else:
            # This a later TLS session within the stack.  The previous
            # TLS session becomes its transport.
            nextTransport = PopOnDisconnectTransport(
                original=self._currentProtocol,
                pop=self._pop
            )

        # Splice the new TLS session into the linked list stack.
        # wrappedProtocol serves as the link, so the protocol at the
        # current position takes our new TLS session as its
        # wrappedProtocol.
        self._currentProtocol.wrappedProtocol = tlsProtocol
        # Move down one position in the linked list.
        self._currentProtocol = tlsProtocol
        # Expose the new, innermost TLS session as the transport to
        # the application protocol.
        self.transport = self._currentProtocol
        # Connect the new TLS session to the previous transport.  The
        # transport attribute also serves as the previous link.
        tlsProtocol.makeConnection(nextTransport)

        # Left over bytes are part of the latest handshake.  Pass them
        # on to the innermost TLS session.
        if bytes is not None:
            tlsProtocol.dataReceived(bytes)


    def stopTLS(self):
        self.transport.loseConnection()


    def _pop(self):
        pop = self._currentProtocol
        previous = pop.transport
        # If the previous link is the head sentinel, we've run out of
        # linked list.  Ensure that the application protocol, stored
        # as the tail sentinel, becomes the wrappedProtocol, and the
        # head sentinel, which is the underlying transport, becomes
        # the transport.
        if previous is self._headTransport:
            self._currentProtocol = self.wrappedProtocol = self._tailProtocol
            self.transport = previous
        else:
            # Splice out a protocol from the linked list stack.  The
            # previous transport is a PopOnDisconnectTransport proxy,
            # so first retrieve proxied object off its original
            # attribute.
            previousProtocol = previous.original
            # The previous protocol's next link becomes the popped
            # protocol's next link
            previousProtocol.wrappedProtocol = pop.wrappedProtocol
            # Move up one position in the linked list.
            self._currentProtocol = previousProtocol
            # Expose the new, innermost TLS session as the transport
            # to the application protocol.
            self.transport = self._currentProtocol



class OnionFactory(WrappingFactory):
    """
    A LWrappingFactory that overrides
    LWrappingFactory.registerProtocol and
    LWrappingFactory.unregisterProtocol.  These methods store in and
    remove from a dictionary LProtocolWrapper instances.  The
    Ctransport patching done as part of the linked-list management
    above causes the instances' hash to change, because the
    C__hash__ is proxied through to the wrapped transport.  They're
    not essential to this program, so the easiest solution is to make
    them do nothing.
    """
    protocol = OnionProtocol

    def registerProtocol(self, protocol):
        pass


    def unregisterProtocol(self, protocol):
        pass

（这也可以在GitHub上找到。）

第二个问题的解决方案在于PopOnDisconnectTransport。原始代码试图通过connectionLost 从堆栈中弹出一个 TLS 会话，但因为只有一个关闭的文件描述符会导致调用connectionLost，所以它无法删除未关闭底层套接字的已停止 TLS 会话。

在撰写本文时，TLSMemoryBIOProtocol 恰好在两个地方调用其传输的loseConnection：_shutdownTLS 和_tlsShutdownFinished。 _shutdownTLS 在主动关闭时调用（loseConnection、abortConnection、unregisterProducer 和 after loseConnection and all pending writes have been flushed），而在被动关闭时调用 _tlsShutdownFinished（handshake failures、empty reads、read errors 和 @ 987654348@)。这一切都意味着在loseConnection 期间，关闭连接的双方 都可以将停止的 TLS 会话从堆栈中弹出。 PopOnDisconnectTransport 这样做是幂等的，因为 loseConnection 通常是幂等的，TLSMemoryBIOProtocol 肯定希望它是幂等的。

将堆栈管理逻辑放入loseConnection 的缺点是它取决于TLSMemoryBIOProtocol 的实现细节。通用解决方案需要跨多个 Twisted 级别的新 API。

在此之前，我们会遇到Hyrum's Law 的另一个例子。

【参考方案2】：

如果您对两个层使用相同的 TLS 参数并且连接到同一主机，那么您可能对两个加密层使用相同的密钥对。尝试为嵌套层使用不同的密钥对，例如隧道到第三个主机/端口。即：localhost:30000（客户端）-> localhost:8080（使用密钥对 A 的 TLS 层 1）-> localhost:8081（使用密钥对 B 的 TLS 层 2）。

【讨论】：

为什么使用相同的密钥对会有问题？

我同意@Jumbogram 的观点，因为 TLS-in-TLS 在技术上应该可以工作，所以我突然想到，在双重加密的情况下，数学中可能存在一些问题。不过，我无法用直接的 RSA 重现失败。

@LeoAccend - 如果您无法重现故障，您在哪里使用什么代码？

【参考方案3】：

您可能需要在启动之前通知远程设备您希望启动一个环境并为第二层分配资源，如果该设备有能力的话。

【讨论】：

谢谢。但是，失败的示例都在 localhost 上运行。 “设备”是 bzr 分支lp:~exarkun/+junk/onion 中的代码。