Redis 7.0 Multi Part AOF的设计和实现

Posted 阿里技术

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了Redis 7.0 Multi Part AOF的设计和实现相关的知识,希望对你有一定的参考价值。


25 2022 09* 681725 2022 09* 25 2022 09* 25 2022 09* 213525 2022 09* *s, len) * not one already. */ (!server.aof_stop_sending_diff && aeGetFileEvents(server.el,server.aof_pipe_write_data_to_child) == aeCreateFileEvent(server.el, server.aof_pipe_write_data_to_child, AE_WRITABLE, aofChildWriteDiffData, *filename) * We can\'t read forever (the server may receive data from clients * faster than it is able to send data to the child), so we try to read * some more data in a loop as soon as there is a good chance more data * will come. If it looks like we are wasting time, we abort (this * happens after 20 ms without new data). */ nodata = start = mstime(); && nodata < (aeWait(server.aof_pipe_read_data_from_parent, AE_READABLE, nodata++; nodata = timeouts. */ aofReadDiffFromParent(); exitcode, bysignal) (aofRewriteBufferWrite(newfd) == serverLog(LL_WARNING, close(newfd); cleanup; aof_pipe_write_data_to_child;aof_pipe_read_data_from_parent;aof_pipe_write_ack_to_parent;aof_pipe_read_ack_from_child;aof_pipe_write_ack_to_child;aof_pipe_read_ack_from_parent;

sds file_name; file_seq; aof_file_type file_type; aofInfo *base_aof_info; *incr_aof_list; *history_aof_list; `base_aof_info` and `incr_aof_list` will be moved to this list. We will delete these AOF files when AOFRW finish. */ curr_base_file_seq; curr_incr_file_seq; dirty; disk, we need to persist it immediately. */ aofManifest *aof_manifest; 1 1 2 1 1 2 BASE_FILE_SUFFIX INCR_FILE_SUFFIX RDB_FORMAT_SUFFIX AOF_FORMAT_SUFFIX MANIFEST_NAME_SUFFIX * from an old redis version. We will use enter upgrade mode in three situations. * * 1. If the \'server.aof_dirname\' directory not exist * 2. If the \'server.aof_dirname\' directory exists but the manifest file is missing * 3. If the \'server.aof_dirname\' directory exists and the manifest file it contains * has only one base AOF record, and the file name of this base AOF is \'server.aof_filename\', * and the \'server.aof_filename\' file not exist in \'server.aof_dirname\' directory * */ (fileExist(server.aof_filename)) (!dirExists(server.aof_dirname) || (am->base_aof_info == && listLength(am->incr_aof_list) == (am->base_aof_info != && listLength(am->incr_aof_list) == && !strcmp(am->base_aof_info->file_name, server.aof_filename) && !aofFileExist(server.aof_filename))) aofUpgradePrepare(am);


(am->base_aof_info) aofInfoFree(am->base_aof_info); aofInfo *ai = aofInfoCreate(); ai->file_name = sdsnew(server.aof_filename); ai->file_seq = ai->file_type = AOF_FILE_TYPE_BASE; am->base_aof_info = ai; am->curr_base_file_seq = am->dirty = (persistAofManifest(am) != C_OK) sds aof_filepath = makePath(server.aof_dirname, server.aof_filename); (rename(server.aof_filename, aof_filepath) == sdsfree(aof_filepath); * advance, it will be set to `server.loading_total_bytes`. */ total_size = getBaseAndIncrAppendOnlyFilesSize(am); startLoading(total_size, RDBFLAGS_AOF_PREAMBLE, (am->base_aof_info) aof_name = (char*)am->base_aof_info->file_name; updateLoadingFileName(aof_name); loadSingleAppendOnlyFile(aof_name); (listLength(am->incr_aof_list)) listNode *ln; listIter li; listRewind(am->incr_aof_list, &li); ((ln = listNext(&li)) != aofInfo *ai = (aofInfo*)ln->value; aof_name = (char*)ai->file_name; updateLoadingFileName(aof_name); loadSingleAppendOnlyFile(aof_name); server.aof_current_size = total_size; server.aof_rewrite_base_size = server.aof_current_size; server.aof_fsync_offset = server.aof_current_size; stopLoading(); 1 1 1 1 2 2 1 1 2 exitcode, bysignal) ( temp_am = aofManifestDup(server.aof_manifest); * as the HISTORY type. */ new_base_filename = getNewBaseFileNameAndMarkPreAsHistory(temp_am); (rename(tmpfile, new_base_filename) == aofManifestFree(temp_am); cleanup; * to AOF_FILE_TYPE_HIST, and move them to the \'history_aof_list\'. */ markRewrittenIncrAofAsHistory(temp_am); (persistAofManifest(temp_am) == C_ERR) bg_unlink(new_base_filename); aofManifestFree(temp_am); cleanup; aofManifestFreeAndUpdate(temp_am); * deletion failure will not cause any problems. */ aofDelHistoryFiles();(ftruncate(server.aof_fd, server.aof_last_incr_size) == (server.aof_state == AOF_ON && !hasActiveChildProcess() && server.aof_rewrite_perc && server.aof_current_size > server.aof_rewrite_min_size && !aofRewriteLimited()) = server.aof_rewrite_base_size ? server.aof_rewrite_base_size : growth = (server.aof_current_size* (growth >= server.aof_rewrite_perc) rewriteAppendOnlyFileBackground();

AOFRW限流机制的引入,还可以有效的避免AOFRW高频重试带来的CPU和fork开销。Redis中很多的RT抖动都和fork有关系。

五  总结


MP-AOF的引入,成功的解决了之前AOFRW存在的内存和CPU开销对Redis实例甚至业务访问带来的不利影响。同时,在解决这些问题的过程中,我们也遇到了很多未曾预料的挑战,这些挑战主要来自于Redis庞大的使用群体、多样化的使用场景,因此我们必须考虑用户在各种场景下使用MP-AOF可能遇到的问题。如兼容性、易用性以及对Redis代码尽可能的减少侵入性等。这都是Redis社区功能演进的重中之重。

同时,MP-AOF的引入也为Redis的数据持久化带来了更多的想象空间。如在开启aof-use-rdb-preamble时,BASE AOF本质是一个RDB文件,因此我们在进行全量备份的时候无需在单独执行一次BGSAVE操作。直接备份BASE AOF即可。MP-AOF支持关闭自动清理HISTORY AOF的能力,因此那些历史的AOF有机会得以保留,并且目前Redis已经支持在AOF中加入timestamp annotation,因此基于这些我们甚至可以实现一个简单的PITR能力( point-in-time recovery)。

MP-AOF的设计原型来自于Tair for redis企业版[2]的binlog实现,这是一套在阿里云Tair服务上久经验证的核心功能,在这个核心功能上阿里云Tair成功构建了全球多活、PITR等企业级能力,使用户的更多业务场景需求得到满足。今天我们将这个核心能力贡献给Redis社区,希望社区用户也能享受这些企业级特性,并通过这些企业级特性更好的优化,创造自己的业务代码。有关MP-AOF的更多细节,请移步参考相关PR(#9788),那里有更多的原始设计和完整代码。

[1]http://mysql.taobao.org/monthly/2018/12/06/
[2]https://help.aliyun.com/document_detail/145956.html



搜索与推荐技术实战训练营
点击阅读原文查看详情

redis的安装

下载redis

下载linux版的redis

安装redis

在目录下解压redis

tar -zxvf redis-6.0.9.tar.gz

进入解压后的文件,输入make

安装过程有可能出现错误

输入命令
安装gcc:yum install gcc-c++
如果还有问题就输入一下命令升级gcc

yum -y install centos-release-scl

yum -y install devtoolset-9-gcc devtoolset-9-gcc-c++ devtoolset-9-binutils

scl enable devtoolset-9 bash

设置永久升级
echo "source /opt/rh/devtoolset-9/enable" >>/etc/profile

再次运行 make
如果报错显示Jemalloc/jemalloc.h: 没有那个文件或目录
输入make distclean后再make即可

如果还是不行,就尝试重新解压redis压缩包

make完成后输入make install

查看 /usr/local/bin

会有redis的启动脚本

如果不输入make install 仍然可以运行但是没有脚本方便

配置redis后台运行

进入redis.conf文件,修改daemonize 为yes

测试redist

启动redis

redis-server /usr/redis/redis-6.0.9/redis.conf

输入:./redis-server /usr/redis/redis-6.0.9/redis.conf

查看redis服务端口号

ps -ef|grep redis

进入redis的命令行窗口

redis-cli -p 6379
如果没make install的话同理./redis-^Ci -p 6379

退出当前的redis服务

输入SHUTDOWN,关闭服务

以上是关于Redis 7.0 Multi Part AOF的设计和实现的主要内容,如果未能解决你的问题,请参考以下文章

redis

Redis事务相关源码探究

Redis事务相关源码探究

iX redis

Redis6.0 高级

Redis6.0 高级