io_uring 源码分析与使用指南<\/h1>

1. io_uring 概述<\/h2>
io_uring 是 Linux 5.1 引入的系统调用接口，用于异步执行系统调用。相比原有的 Native AIO，io_uring 提供了更高效的异步 I\/O 机制。<\/p>

核心概念<\/h3>

SQ (Submission Queue)<\/strong>: 提交任务的环形队列<\/li>
SQE (Submission Queue Entry)<\/strong>: 提交队列中的单个任务项<\/li>
CQ (Completion Queue)<\/strong>: 获取结果的环形队列<\/li>

CQE (Completion Queue Entry)<\/strong>: 完成队列中的单个结果项<\/li> <\/ul>
2. liburing 库使用<\/h2>
安装方法<\/h3>
# 自动安装<\/span> <\/span><\/span>sudo apt-get install liburing-dev <\/span><\/span> <\/span><\/span># 手动安装<\/span> <\/span><\/span>git clone https:\/\/git.kernel.dk\/liburing <\/span><\/span>cd liburing <\/span><\/span>.\/configure <\/span><\/span>make <\/span><\/span>sudo make install <\/span><\/span><\/code><\/pre>核心结构体<\/h3> struct<\/span> io_uring { <\/span><\/span> struct<\/span> io_uring_sq sq; \/\/ 提交队列 <\/span><\/span><\/span><\/span> struct<\/span> io_uring_cq cq; \/\/ 完成队列 <\/span><\/span><\/span><\/span> unsigned<\/span> flags; \/\/ 配置标志 <\/span><\/span><\/span><\/span> int<\/span> ring_fd; \/\/ 文件描述符 <\/span><\/span><\/span><\/span>}; <\/span><\/span><\/code><\/pre>初始化与销毁<\/h3> \/\/ 初始化 <\/span><\/span><\/span><\/span>int<\/span> io_uring_queue_init<\/span>(unsigned<\/span> entries, struct<\/span> io_uring *<\/span>ring, unsigned<\/span> flags); <\/span><\/span> <\/span><\/span>\/\/ 销毁 <\/span><\/span><\/span><\/span>void<\/span> io_uring_queue_exit<\/span>(struct<\/span> io_uring *<\/span>ring); <\/span><\/span><\/code><\/pre>参数说明：<\/p> entries<\/code>: 队列大小（必须是2的幂，如1024）<\/li> flags<\/code>: 配置标志（如IORING_SETUP_SQPOLL<\/code>启用内核轮询模式）<\/li> <\/ul> 提交队列操作<\/h3> \/\/ 获取一个空闲的SQE <\/span><\/span><\/span><\/span>struct<\/span> io_uring_sqe *<\/span>io_uring_get_sqe<\/span>(struct<\/span> io_uring *<\/span>ring); <\/span><\/span> <\/span><\/span>\/\/ 提交请求到内核 <\/span><\/span><\/span><\/span>int<\/span> io_uring_submit<\/span>(struct<\/span> io_uring *<\/span>ring); <\/span><\/span><\/code><\/pre>完成队列操作<\/h3> \/\/ 阻塞等待至少一个完成事件 <\/span><\/span><\/span><\/span>int<\/span> io_uring_wait_cqe<\/span>(struct<\/span> io_uring *<\/span>ring, struct<\/span> io_uring_cqe **<\/span>cqe_ptr); <\/span><\/span> <\/span><\/span>\/\/ 非阻塞检查完成事件 <\/span><\/span><\/span><\/span>int<\/span> io_uring_peek_cqe<\/span>(struct<\/span> io_uring *<\/span>ring, struct<\/span> io_uring_cqe **<\/span>cqe_ptr); <\/span><\/span> <\/span><\/span>\/\/ 标记CQE已处理 <\/span><\/span><\/span><\/span>void<\/span> io_uring_cqe_seen<\/span>(struct<\/span> io_uring *<\/span>ring, struct<\/span> io_uring_cqe *<\/span>cqe); <\/span><\/span><\/code><\/pre>准备I\/O请求的辅助函数<\/h3> \/\/ 文件操作 <\/span><\/span><\/span><\/span>void<\/span> io_uring_prep_readv<\/span>(struct<\/span> io_uring_sqe *<\/span>sqe, int<\/span> fd, const<\/span> struct<\/span> iovec *<\/span>iov, unsigned<\/span> nr_vecs, off_t offset); <\/span><\/span>void<\/span> io_uring_prep_writev<\/span>(struct<\/span> io_uring_sqe *<\/span>sqe, int<\/span> fd, const<\/span> struct<\/span> iovec *<\/span>iov, unsigned<\/span> nr_vecs, off_t offset); <\/span><\/span>void<\/span> io_uring_prep_openat<\/span>(struct<\/span> io_uring_sqe *<\/span>sqe, int<\/span> dfd, const<\/span> char<\/span> *<\/span>path, int<\/span> flags, mode_t mode); <\/span><\/span>void<\/span> io_uring_prep_close<\/span>(struct<\/span> io_uring_sqe *<\/span>sqe, int<\/span> fd); <\/span><\/span> <\/span><\/span>\/\/ 网络操作 <\/span><\/span><\/span><\/span>void<\/span> io_uring_prep_accept<\/span>(struct<\/span> io_uring_sqe *<\/span>sqe, int<\/span> sockfd, struct<\/span> sockaddr *<\/span>addr, socklen_t *<\/span>addrlen, int<\/span> flags); <\/span><\/span>void<\/span> io_uring_prep_send<\/span>(struct<\/span> io_uring_sqe *<\/span>sqe, int<\/span> sockfd, const<\/span> void<\/span> *<\/span>buf, size_t len, int<\/span> flags); <\/span><\/span>void<\/span> io_uring_prep_recv<\/span>(struct<\/span> io_uring_sqe *<\/span>sqe, int<\/span> sockfd, void<\/span> *<\/span>buf, size_t len, int<\/span> flags); <\/span><\/span> <\/span><\/span>\/\/ 超时与事件 <\/span><\/span><\/span><\/span>void<\/span> io_uring_prep_timeout<\/span>(struct<\/span> io_uring_sqe *<\/span>sqe, struct<\/span> __kernel_timespec *<\/span>ts, unsigned<\/span> count, unsigned<\/span> flags); <\/span><\/span>void<\/span> io_uring_prep_poll_add<\/span>(struct<\/span> io_uring_sqe *<\/span>sqe, int<\/span> fd, short<\/span> poll_mask); <\/span><\/span><\/code><\/pre>完整示例：异步读取文件<\/h3> #include<\/span> <liburing.h><\/span> <\/span><\/span><\/span>#include<\/span> <fcntl.h><\/span> <\/span><\/span><\/span>#include<\/span> <stdio.h><\/span> <\/span><\/span><\/span>#include<\/span> <unistd.h><\/span> <\/span><\/span><\/span><\/span> <\/span><\/span>#define BUF_SIZE 4096 <\/span><\/span><\/span><\/span> <\/span><\/span>int<\/span> main<\/span>() { <\/span><\/span> struct<\/span> io_uring ring; <\/span><\/span> char<\/span> buf[BUF_SIZE]; <\/span><\/span> <\/span><\/span> \/\/ 初始化io_uring <\/span><\/span><\/span><\/span> if<\/span> (io_uring_queue_init(32<\/span>, &<\/span>ring, 0<\/span>) <<\/span> 0<\/span>) { <\/span><\/span> perror("io_uring_queue_init"<\/span>); <\/span><\/span> return<\/span> 1<\/span>; <\/span><\/span> } <\/span><\/span> <\/span><\/span> \/\/ 打开文件 <\/span><\/span><\/span><\/span> int<\/span> fd =<\/span> open("test.txt"<\/span>, O_RDONLY); <\/span><\/span> if<\/span> (fd <<\/span> 0<\/span>) { <\/span><\/span> perror("open"<\/span>); <\/span><\/span> return<\/span> 1<\/span>; <\/span><\/span> } <\/span><\/span> <\/span><\/span> \/\/ 获取SQE <\/span><\/span><\/span><\/span> struct<\/span> io_uring_sqe *<\/span>sqe =<\/span> io_uring_get_sqe(&<\/span>ring); <\/span><\/span> if<\/span> (!<\/span>sqe) { <\/span><\/span> fprintf(stderr, "Failed to get SQE<\/span>\n<\/span>"<\/span>); <\/span><\/span> return<\/span> 1<\/span>; <\/span><\/span> } <\/span><\/span> <\/span><\/span> \/\/ 准备读请求 <\/span><\/span><\/span><\/span> struct<\/span> iovec iov =<\/span> { .iov_base =<\/span> buf, .iov_len =<\/span> BUF_SIZE }; <\/span><\/span> io_uring_prep_readv(sqe, fd, &<\/span>iov, 1<\/span>, 0<\/span>); <\/span><\/span> <\/span><\/span> \/\/ 提交请求 <\/span><\/span><\/span><\/span> io_uring_submit(&<\/span>ring); <\/span><\/span> <\/span><\/span> \/\/ 等待完成 <\/span><\/span><\/span><\/span> struct<\/span> io_uring_cqe *<\/span>cqe; <\/span><\/span> io_uring_wait_cqe(&<\/span>ring, &<\/span>cqe); <\/span><\/span> <\/span><\/span> \/\/ 处理结果 <\/span><\/span><\/span><\/span> if<\/span> (cqe-><\/span>res <<\/span> 0<\/span>) { <\/span><\/span> fprintf(stderr, "Async read failed: %s<\/span>\n<\/span>"<\/span>, strerror(-<\/span>cqe-><\/span>res)); <\/span><\/span> } else<\/span> { <\/span><\/span> printf("Read %d bytes<\/span>\n<\/span>"<\/span>, cqe-><\/span>res); <\/span><\/span> } <\/span><\/span> <\/span><\/span> \/\/ 标记CQE已处理 <\/span><\/span><\/span><\/span> io_uring_cqe_seen(&<\/span>ring, cqe); <\/span><\/span> <\/span><\/span> \/\/ 清理 <\/span><\/span><\/span><\/span> close(fd); <\/span><\/span> io_uring_queue_exit(&<\/span>ring); <\/span><\/span> <\/span><\/span> return<\/span> 0<\/span>; <\/span><\/span>} <\/span><\/span><\/code><\/pre>3. io_uring 系统调用分析<\/h2> 三个核心系统调用<\/h3> io_uring_setup()<\/strong>:<\/p> 返回文件描述符给用户操作<\/li> 注册io_uring的上下文(ctx)<\/li> 创建CQ和SQ队列<\/li> <\/ul> <\/li> io_uring_enter()<\/strong>:<\/p> 提交新的IO请求<\/li> 可选择等待IO完成请求<\/li> <\/ul> <\/li> io_uring_register()<\/strong>:<\/p> 注册\/注销\/更新缓冲区、文件、个性化等<\/li> <\/ul> <\/li> <\/ol> io_uring_setup 系统调用<\/h3> SYSCALL_DEFINE2(io_uring_setup, u32, entries, struct<\/span> io_uring_params __user *<\/span>, params) <\/span><\/span><\/code><\/pre>工作流程：<\/p> 将用户空间参数拷贝到内核空间<\/li> 检查保留字段是否为0<\/li> 检查flags的合法性<\/li> 调用io_uring_create()<\/code>创建io_uring上下文，并返回文件描述符<\/li> <\/ol> io_uring_create 函数<\/h4> static<\/span> struct<\/span> file *<\/span>io_uring_create(unsigned<\/span> entries, struct<\/span> io_uring_params *<\/span>p) <\/span><\/span><\/code><\/pre>工作流程：<\/p> 检查和调整SQ\/CQ环的大小默认CQsize为SQ的两倍<\/li> 如果设置IORING_SETUP_CQSIZE<\/code>，可自定义CQsize（需大于SQsize）<\/li> <\/ul> <\/li> 分配并初始化上下文(ctx)空间（结构体io_ring_ctx<\/code>）<\/li> 分配和初始化SQ\/CQ环<\/li> 获取文件描述符并安装<\/li> <\/ol> 关键函数：<\/p> io_ring_ctx_alloc()<\/code>: 分配上下文空间，初始化列表头、旋转锁等<\/li> io_allocate_scq_urings()<\/code>: 分配与用户空间共享的rings本体<\/li> io_sq_offload_create()<\/code>: 初始化工作队列<\/li> io_sq_offload_start()<\/code>: 唤醒工作线程<\/li> io_uring_get_file()<\/code>: 获取文件描述符<\/li> <\/ul> io_ring_ctx_alloc 分配上下文空间<\/h4> 使用GFP_KERNEL<\/code>标志进行常规分配。<\/p> io_allocate_scq_urings 分配SQ\/CQ环<\/h4> 获取rings大小<\/li> 调用io_mem_alloc()<\/code>，最终调用__get_free_pages()<\/code>直接分配页<\/li> io_rings<\/code>结构体通过mmap()与用户进程共享<\/li> <\/ol> io_sq_offload_create 初始化工作队列<\/h4> 检查ctx->flags<\/code>是否开启SQ轮询模式<\/li> 检查权限（需要SYS_ADMIN<\/code>或SYS_NICE<\/code>）<\/li> 将上下文的SQ数据链添加到sqd的新列表<\/li> 设置线程超时（默认1秒）<\/li> 设置线程是否需要指定CPU<\/li> 分配task_struct<\/code>内存并初始化<\/li> 调用io_init_wq_offload()<\/code>初始化工作队列<\/li> <\/ol> io_uring_get_file 获取文件结构体<\/h4> 获取文件结构体<\/li> 调用io_uring_install_fd()<\/code>安装文件描述符<\/li> 使用sock_create_kern()<\/code>创建内核UNIX套接字<\/li> 通过anon_inode_getfile()<\/code>创建匿名inode并返回ctx的文件结构体<\/li> <\/ol> io_uring_enter 系统调用<\/h3> SYSCALL_DEFINE6(io_uring_enter, unsigned<\/span> int<\/span>, fd, u32, to_submit, u32, min_complete, u32, flags, const<\/span> sigset_t __user *<\/span>, sig, size_t, sigsz) <\/span><\/span><\/code><\/pre>工作流程：<\/p> 检查flags、文件描述符<\/li> 根据ctx判断是否需要SQ轮询，否则采用to_submit<\/code>数量提交<\/li> 设置IORING_ENTER_GETEVENTS<\/code>则处理完成事件<\/li> 如果启用了IOPOLL<\/code>但未启用SQPOLL<\/code>模式，调用io_iopoll_check<\/code>进行轮询检查<\/li> 否则调用io_cqring_wait<\/code>等待完成事件<\/li> 释放资源后返回SQE提交数量<\/li> <\/ol> 关键函数：<\/p> io_sqpoll_wait_sq()<\/code>: 将当前进程添加到ctx->sqo_sq_wait<\/code>队列并挂起<\/li> io_submit_sqes()<\/code>: 提交SQES到io_uring<\/li> io_iopoll_check()<\/code>: 轮询检查是否有事件完成<\/li> io_cqring_wait()<\/code>: 等待完成队列中有足够的事件<\/li> <\/ul> io_sqpoll_wait_sq 等待SQ有空间<\/h4> 通过io_sqring_full()<\/code>判断SQring是否为满状态<\/li> 如果满，将当前进程添加到ctx->sqo_sq_wait<\/code>等待队列<\/li> 进行进程调度，直到SQring有空闲、收到信号或SQ线程死亡<\/li> <\/ol> io_submit_sqes 提交SQES<\/h4> 检查CQ溢出风险<\/li> 确定提交事件数量（用户提交数量、SQ事件数量、SQ队列长度取最小）<\/li> 增加当前任务的inflight计数和usage计数<\/li> 循环处理每个SQE：通过上下文获取SQE<\/li> 初始化io请求(req)<\/li> 调用io_submit_sqe()<\/code>提交单个SQE请求<\/li> <\/ul> <\/li> <\/ol> io_iopoll_check 轮询事件完成<\/h4> 循环调用io_iopoll_getevents()<\/code><\/li> 对nr_events<\/code>进行累加<\/li> 每8次启动一次任务，尝试完成I\/O请求<\/li> 有事件完成或需要进程调度时退出<\/li> <\/ol> io_cqring_wait 等待CQ满足任务数<\/h4> 调用io_run_task_work()<\/code>处理已有事件<\/li> 设置信号掩码、超时时间、超时计数<\/li> 挂起当前进程加入等待队列<\/li> 调用io_run_task_work_sig()<\/code>执行待处理任务并检查信号<\/li> 根据CQ队列状态返回（不为空返回0，否则返回错误码）<\/li> <\/ol> io_uring_register 系统调用<\/h3> SYSCALL_DEFINE4(io_uring_register, unsigned<\/span> int<\/span>, fd, unsigned<\/span> int<\/span>, opcode, void<\/span> __user *<\/span>, arg, unsigned<\/span> int<\/span>, nr_args) <\/span><\/span><\/code><\/pre>工作流程：<\/p> 判断文件描述符是否符合io_uring<\/li> 检查上下文是否存在（通过引用计数）<\/li> 检查opcode合法性<\/li> 根据opcode执行相应操作<\/li> <\/ol> 常见opcode：<\/p> IORING_REGISTER_BUFFERS<\/code>: 注册缓冲区<\/li> IORING_UNREGISTER_BUFFERS<\/code>: 注销缓冲区<\/li> IORING_REGISTER_FILES<\/code>: 注册文件<\/li> IORING_UNREGISTER_FILES<\/code>: 注销文件<\/li> IORING_REGISTER_FILES_UPDATE<\/code>: 更新已注册文件<\/li> IORING_REGISTER_EVENTFD(_ASYNC)<\/code>: 注册事件fd<\/li> IORING_UNREGISTER_EVENTFD<\/code>: 注销事件fd<\/li> <\/ul> io_sqe_buffer_register 注册用户缓冲区<\/h4> 检查参数合法性<\/li> 判断用户缓冲区是否存在<\/li> 检查大小合法性<\/li> 为ctx->user_bufs<\/code>分配缓冲区数组（GFP_KERNEL<\/code>）<\/li> 遍历数组为每个元素分配空间<\/li> 从用户空间复制IO向量信息到内核态<\/li> 检查基地址、长度（最大1GB）<\/li> 分配vmas<\/code>和pages<\/code>空间<\/li> 使用kvmalloc_array()<\/code>为每个缓冲区分配空间<\/li> 调用pin_user_pages()<\/code>固定用户页面到内存<\/li> 检查每个vma<\/code>（不支持非huge page的文件内存）<\/li> 初始化bio_vec<\/code>和当前缓冲区<\/li> <\/ol> io_sqe_buffer_unregister 注销用户缓冲区<\/h4> 检查用户缓冲区是否存在<\/li> 解除用户映射的bio_vec<\/code>向量<\/li> 释放imu->bvec<\/code><\/li> 释放ctx->user_bufs<\/code>并置零<\/li> <\/ol> io_sqe_files_register 注册文件<\/h4> 遍历用户传入的所有文件<\/li> 使用fget()<\/code>获取file<\/code>结构体（增加文件引用计数）<\/li> 检查不支持注册io_uring本身<\/li> 调用io_sqe_files_scm()<\/code>设置SCM_RIGHTS<\/code><\/li> 分配并设置引用节点后返回<\/li> <\/ol> 4. 高级功能<\/h2> 注册资源（减少开销）<\/h3> 文件描述符注册<\/strong>:<\/p> 通过IORING_REGISTER_FILES<\/code>注册<\/li> 使用IOSQE_FIXED_FILE<\/code>标志避免内核检查开销<\/li> <\/ul> <\/li> 缓冲区注册<\/strong>:<\/p> 通过IORING_REGISTER_BUFFERS<\/code>注册<\/li> 内核预注册缓冲区，提升零拷贝性能<\/li> <\/ul> <\/li> <\/ol> 内核线程与轮询模式<\/h3> IORING_SETUP_SQPOLL<\/code>: 启用内核轮询模式<\/li> 默认线程超时为1秒<\/li> 可设置线程绑定特定CPU<\/li> <\/ul> 飞行计数 (inflight)<\/h3> 在io_submit_sqes()<\/code>中增加计数<\/li> 用于跟踪正在处理中的I\/O请求<\/li> 与SCM_RIGHTS<\/code>机制相关，影响资源管理<\/li> <\/ul> 5. 性能优化建议<\/h2> 合理设置SQ和CQ的大小（通常为2的幂）<\/li> 对于高并发场景，考虑启用IORING_SETUP_SQPOLL<\/code><\/li> 预注册常用文件和缓冲区减少开销<\/li> 批量提交请求减少系统调用次数<\/li> 使用io_uring_peek_cqe()<\/code>非阻塞检查减少等待时间<\/li> 及时调用io_uring_cqe_seen()<\/code>释放资源<\/li> <\/ol> 6. 安全注意事项<\/h2> 确保正确处理错误返回码<\/li> 避免缓冲区溢出（最大1GB限制）<\/li> 注意资源泄漏（文件描述符、内存等）<\/li> 合理设置权限（SQPOLL需要SYS_ADMIN<\/code>或SYS_NICE<\/code>）<\/li> 考虑使用IORING_REGISTER_RESTRICTIONS<\/code>限制操作类型<\/li> <\/ol> 7. 总结<\/h2> io_uring 提供了高效的异步 I\/O 机制，通过共享内存环形队列减少系统调用开销，支持多种操作类型（文件、网络、超时等）。合理使用注册资源和高级功能可以进一步提升性能。理解其内部实现机制有助于开发更高效的应用程序和排查潜在问题。<\/p>

io_uring 源码分析与使用指南<\/h1>

1. io_uring 概述<\/h2> io_uring 是 Linux 5.1 引入的系统调用接口，用于异步执行系统调用。相比原有的 Native AIO，io_uring 提供了更高效的异步 I\/O 机制。<\/p>

2. liburing 库使用<\/h2>

3. io_uring 系统调用分析<\/h2>

io_ring_ctx_alloc 分配上下文空间<\/h4> 使用GFP_KERNEL<\/code>标志进行常规分配。<\/p>

4. 高级功能<\/h2>

1. io_uring 概述<\/h2>
io_uring 是 Linux 5.1 引入的系统调用接口，用于异步执行系统调用。相比原有的 Native AIO，io_uring 提供了更高效的异步 I\/O 机制。<\/p>

io_ring_ctx_alloc 分配上下文空间<\/h4>
使用`GFP_KERNEL<\/code>标志进行常规分配。<\/p>`