IOCP 接收和发送
Posted
技术标签:
【中文标题】IOCP 接收和发送【英文标题】：IOCP recv AND send 【发布时间】：2014-08-20 20:05:14 【问题描述】：到目前为止，我发现的所有示例要么只是读取或写入，要么是 10000 行野兽，我什至不知道从哪里开始了解它们是如何工作的。
为了测试我的代码，我将浏览器指向我的服务器并发送了一个简单的 http 请求。结果令人困惑。
例如，在某一时刻 GetQueuedCompletionStatus 返回，WSARecv 说它读取了我发送的 http 响应的字节数，尽管这个响应应该（并且确实）最终到达客户端，并且 recvbuffer 甚至没有填充这些字节。
此外，我不知道在其他浏览器关闭连接后何时释放缓冲区，因为 GetQueuedCompletionStatus 在我调用 closesocket 后不断返回几次。
此外，一旦 GetQueuedCompletionStatus 返回，我不知道何时有要读取的数据或要写入的数据。我可以两个都试一下，看看哪个失败，但这似乎很粗鲁。
为了揭示我对 IOCP 的任何误解，我编写了一些伪代码来表达我认为我的代码的作用：
main
create server socket
create io completion port
while true
accept client socket
create completion port for client socket
create recv buffer and send buffer for client
call WSARecv once with 0 bytes for whatever reason
worker thread
while true
wait until GetQueuedCompletionStatus returns
do something if that failed, not quite sure what (free buffers?)
if no bytes were transferred, close socket
try to recv data
try to send data
实际代码：
#include <stdio.h>
#include <stdlib.h>
#include <winsock2.h>
#define BUFFER_SIZE 1024
typedef struct
WSAOVERLAPPED overlapped;
SOCKET socket;
WSABUF sendbuf;
WSABUF recvbuf;
char sendbuffer[BUFFER_SIZE];
char recvbuffer[BUFFER_SIZE];
client;
DWORD WINAPI worker_thread(HANDLE iocp)
DWORD flags = 0, n = 0;
ULONG unused;
client *c;
while (1)
int ret = GetQueuedCompletionStatus(iocp, &n, &unused, (LPOVERLAPPED*)&c, INFINITE);
printf("%3d triggered\n", c->socket);
if (ret == FALSE)
printf("%3d GetQueuedCompletionStatus error %i\n", c->socket, WSAGetLastError());
continue;
if (c->socket == INVALID_SOCKET)
printf("error: socket already closed\n");
continue;
if (n == 0)
printf("%3d disconnected\n", c->socket);
closesocket(c->socket);
c->socket = INVALID_SOCKET;
continue;
/* how do I know if there is data to read or data to write? */
WSARecv(c->socket, &(c->recvbuf), 1, &n, &flags, &(c->overlapped), NULL);
printf("%3d WSARecv %ld bytes\n", c->socket, n);
WSASend(c->socket, &(c->sendbuf), 1, &n, flags, &(c->overlapped), NULL);
printf("%3d WSASend %ld bytes\n", c->socket, n);
/* TODO handle partial sends */
c->sendbuf.len = 0;
return 0;
SOCKET make_server(int port)
int yes = 1;
struct sockaddr_in addr;
SOCKET sock;
WSADATA wsaData;
WSAStartup(MAKEWORD(2, 2), &wsaData);
addr.sin_family = AF_INET;
addr.sin_addr.s_addr = htonl(INADDR_ANY);
addr.sin_port = htons(port);
sock = WSASocket(AF_INET, SOCK_STREAM, 0, NULL, 0, WSA_FLAG_OVERLAPPED);
setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, (const char*)yes, sizeof(yes));
bind(sock, (struct sockaddr*)&addr, sizeof(addr));
listen(sock, SOMAXCONN);
return sock;
int main()
const char *text =
"HTTP/1.0 200 OK\r\n"
"Content-Length: 13\r\n"
"Content-Type: text/html\r\n"
"Connection: Close\r\n"
"\r\n"
"Hello, World!";
SOCKET server_socket = make_server(8080);
HANDLE iocp = CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, 0, 0);
CreateThread(NULL, 0, worker_thread, iocp, 0, NULL);
while (1)
DWORD flags = 0, n = 0;
client *c;
struct sockaddr_in addr;
int addrlen = sizeof(addr);
SOCKET client_socket = WSAAccept(server_socket, (struct sockaddr*)&addr, &addrlen, NULL, 0);
printf("%3d connected\n", client_socket);
CreateIoCompletionPort((HANDLE)client_socket, iocp, 0, 0);
c = (client*)calloc(1, sizeof(*c));
c->socket = client_socket;
c->sendbuf.len = strlen(text);
c->recvbuf.len = BUFFER_SIZE;
c->sendbuf.buf = c->sendbuffer;
c->recvbuf.buf = c->recvbuffer;
strcpy(c->sendbuf.buf, text);
/* for some reason I have to receive 0 bytes once */
WSARecv(c->socket, &(c->recvbuf), 1, &n, &flags, &(c->overlapped), NULL);
示例输出：
/* Browser makes two tcp connections on socket 124 and 128. */
124 connected
128 connected
/* GetQueuedCompletionStatus returned for socket 124. */
124 triggered
/* We received the browser's http request. */
124 WSARecv 375 bytes
/* Send http response to browser. */
124 WSASend 96 bytes
/* GetQueuedCompletionStatus returned again. */
124 triggered
/* This is wrong, we should not receive our response to the browser. */
/* Also we didn't even receive data here. */
/* recvbuffer still contains the http request. */
124 WSARecv 96 bytes
/* this is ok */
124 WSASend 0 bytes
124 triggered
124 disconnected
/* Why does GetQueuedCompletionStatus still return? the socket is closed! */
/* Also how can I tell when I can safely free the buffers */
/* if GetQueuedCompletionStatus keeps returning? */
-1 triggered
-1 GetQueuedCompletionStatus error 1236
-1 triggered
-1 GetQueuedCompletionStatus error 1236
/* same again for second http request */
128 triggered
128 WSARecv 375 bytes
128 WSASend 96 bytes
128 triggered
128 WSARecv 96 bytes
128 WSASend 0 bytes
128 triggered
128 disconnected
-1 triggered
-1 GetQueuedCompletionStatus error 1236
-1 triggered
-1 GetQueuedCompletionStatus error 1236
128 connected
128 triggered
128 WSARecv 375 bytes
128 WSASend 96 bytes
128 triggered
128 WSARecv 96 bytes
128 WSASend 0 bytes
128 triggered
128 disconnected
-1 triggered
-1 GetQueuedCompletionStatus error 1236
-1 triggered
-1 GetQueuedCompletionStatus error 1236
128 connected
128 triggered
128 WSARecv 289 bytes
128 WSASend 96 bytes
128 triggered
128 WSARecv 96 bytes
128 WSASend 0 bytes
128 triggered
128 disconnected
-1 triggered
-1 GetQueuedCompletionStatus error 1236
-1 triggered
-1 GetQueuedCompletionStatus error 1236
【问题讨论】：
我有一些示例 IOCP 代码，如果需要，您可以查看，请参见此处：serverframework.com/products---the-free-framework.html @Optimus1，如果您没有找到有用的链接，我很抱歉。 【参考方案1】：您的伪代码工作流程应该看起来更像这样：
main
create server socket
create io completion port
create worker thread
while not done
accept client socket
associate client socket with completion port
create recv, send, and work buffers for client
call WSARecv with >0 bytes to start filling recv buffer
if failed
close client socket and free associated buffers
terminate worker thread
close client sockets
close server socket
worker thread
while not terminated
call GetQueuedCompletionStatus
if failed
if failed because of IO error
close socket and free associated buffers
else if not timeout
handle error as needed
else if no bytes were transferred
close socket and free associated buffers
else if IO was WSARecv
move data from recv buffer to end of work buffer
while work buffer has a complete message
remove message from front of work buffer, process as needed
if output to send
if send buffer not empty
append output to end of send buffer, will send later
else
move output to send buffer
call WSASend
if failed
close socket and free associated buffers
call WSARecv with >0 bytes to start filling recv buffer
if failed
close socket and free associated buffers
else if IO was WSASend
remove reported number of bytes from front of send buffer
if send buffer not empty
call WSASend
if failed
close socket and free associated buffers
我将把它作为练习留给你，让你把它翻译成你的代码。
【讨论】：
恕我直言，您的故障处理过于简单。由于客户端关闭而导致的读取失败不需要您关闭发送端，除非您想...另外我假设“空闲关联缓冲区”仅与失败的操作使用的缓冲区有关... 这是一个简单示例项目的简单工作流程。在现实世界的复杂项目中，错误处理可能同样复杂。例如，如果入站读取失败并且出站发送已经在进行中，您可以立即停止读取并释放接收缓冲区，但延迟关闭套接字并释放发送缓冲区，直到活动发送首先完成/失败。 很公平，我只是想指出，否则简单示例中的简单错误处理往往最终成为生产代码中的真实内容......【参考方案2】：我的大部分信息来自逆向工程this code。 我必须警告您不要使用此代码，尽管它充满了错误和不良行为，它是为更早版本的 windows 制作的。
但是一般原则是相似的，所以这是一个很好的练习，看看你能从中学到多少（但无论你做什么，都不要在没有大量修改的情况下使用它）。
最好的代码部分在IOCPDlg.cpp 和IOCPDlg.h。
IOCP 和读取的一般原则是每个端口应该始终有 1 个读取请求排队。当然，除非您不想从中阅读。
在您的工作线程中使用已完成状态，您应该执行以下操作：
使用GetQueuedCompletionStatus和GetLastError()的返回码检查是否有错误：
处理错误，但过滤掉超时（读取超时本身并不是错误）：
if (!ReturnValue && GetErrorValue != ERROR_SEM_TIMEOUT )
/-- i free up the socket and associated buffers here --/
continue;
进程 IO。我使用以下来区分 IO（我将在底部添加它的代码）：
/-- get the base address of the struct holding lpOverlapped --/
pOverlapPlus = CONTAINING_RECORD(lpOverlapped, OVERLAPPEDPLUS, ol);
如果您刚刚处理的 IO 是 IORead 类型，则排队另一个读取。
不应在此线程中进行发送，应使用不同的线程/函数将发送 IO 请求添加到 IOCP。
释放 OVERLAPPEDPLUS 类，您需要为每个请求创建一个新类，因此您需要在每个请求出列后释放它。
OVERLAPPEDPLUS的代码，可以在CodeProject Article找到。
enum IOType
IOInitialize,
IORead,
IOWrite,
IOIdle
;
class OVERLAPPEDPLUS
public:
OVERLAPPED ol;
IOType ioType;
OVERLAPPEDPLUS(IOType _ioType)
ZeroMemory(this, sizeof(OVERLAPPEDPLUS));
ioType = _ioType;
;
【讨论】：
我也从代码中了解了iocp，你提到过。你能多说一下这段代码中的“错误和不良行为”吗？提前谢谢。 我记得的两个主要问题是WAIT_TIMEOUT 应该是 ERROR_SEM_TIMEOUT，缺少写信号量，以及会在剩余 0 字节时触发的写入，可以通过测试来修复在排队最终发送请求之前，缓冲区剩余 0 个字节。除了这 3 件主要的事情之外，还有一些小事情（我不记得了）与 MFC 相关，这应该没问题，因为任何新项目都不应该使用 MFC。【参考方案3】：
你应该有一个状态变量让你知道你是在读还是写（或者可能是其他 io 操作状态......你初始化那个状态变量以在你的主程序中读取（在你发出 wsarecv 之前），并设置每当您发送时要写入该状态变量，这样您就可以在工作线程中查询该状态变量以了解您是在读取还是发送
在您的工作线程中：
switch (state_var)
case IO_READ:
//process wsarecv
//process data
break;
case IO_SEND:
//process wsasend
//send more data
//if there are no more to send
//state_var = IO_READ;
//call wsarecv
break;
//process other io command you define
【讨论】：
以上是关于IOCP 接收和发送的主要内容，如果未能解决你的问题，请参考以下文章