关于SQL Server 数据库归档的一些思考和改进
Posted 东山絮柳仔
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了关于SQL Server 数据库归档的一些思考和改进相关的知识，希望对你有一定的参考价值。
一.需求背景
SQL Server开源的归档工具不多，DBA一般都是通过计划任务来触发执行，执行的脚本多是SP或者是SSIS包。SSIS包的性能稍好一些，但是维护更新成本高些。所以更常见的是通过SP脚本来实现归档操作。
当数据库规模较小时，可以方便的直接在数据库上进行脚本的编写部署。但是随着数据库越来越多，管理维护成本就会越来越大，越来越不方便。现在我们实行的方式是通过中央管理器来管理众多的数据库备份（这是在拥有专门的备份程序前的一个过渡方案）。我们将归档基础配置信息、归档运行历史记录、异常报错等数据统一维护在中央数据库上。如此,可以方便统一的查看、管理和维护。
二.主要架构
三.主要关联表
2.1 归档基础配置表
表字段含义，请耐心查看字段说明。
CREATE TABLE [dbo].[DBData_ArchiveConfig]( [ID] [int] IDENTITY(1,1) NOT NULL, [IP] [varchar](50) NULL, [DBName] [varchar](50) NULL, [DataTable] [varchar](50) NULL, [TargetIP] [varchar](50) NULL, [TargetDB] [varchar](50) NULL, [TargetTable] [varchar](50) NULL, [Prerequisite] [varchar](300) NULL, [DelMaxQTY] [int] NULL, [IsCheckOrderID] [int] NULL, [SP_Name] [int] NULL, [StartTime] [datetime] NULL, [EndTime] [datetime] NULL ) ON [PRIMARY] GO EXEC sys.sp_addextendedproperty @name=N\'MS_Description\', @value=N\'Server IP(数据位于中央管理器中，所以归档数据库库所在的IP要维护，可维修虚拟的IP)\' , @level0type=N\'SCHEMA\',@level0name=N\'dbo\', @level1type=N\'TABLE\',@level1name=N\'DBData_ArchiveConfig\', @level2type=N\'COLUMN\',@level2name=N\'IP\' GO EXEC sys.sp_addextendedproperty @name=N\'MS_Description\', @value=N\'要归档的数据库\' , @level0type=N\'SCHEMA\',@level0name=N\'dbo\', @level1type=N\'TABLE\',@level1name=N\'DBData_ArchiveConfig\', @level2type=N\'COLUMN\',@level2name=N\'DBName\' GO EXEC sys.sp_addextendedproperty @name=N\'MS_Description\', @value=N\'要归档的表\' , @level0type=N\'SCHEMA\',@level0name=N\'dbo\', @level1type=N\'TABLE\',@level1name=N\'DBData_ArchiveConfig\', @level2type=N\'COLUMN\',@level2name=N\'DataTable\' GO EXEC sys.sp_addextendedproperty @name=N\'MS_Description\', @value=N\'备份指向的IP\' , @level0type=N\'SCHEMA\',@level0name=N\'dbo\', @level1type=N\'TABLE\',@level1name=N\'DBData_ArchiveConfig\', @level2type=N\'COLUMN\',@level2name=N\'TargetIP\' GO EXEC sys.sp_addextendedproperty @name=N\'MS_Description\', @value=N\'备份指向的数据库\' , @level0type=N\'SCHEMA\',@level0name=N\'dbo\', @level1type=N\'TABLE\',@level1name=N\'DBData_ArchiveConfig\', @level2type=N\'COLUMN\',@level2name=N\'TargetDB\' GO EXEC sys.sp_addextendedproperty @name=N\'MS_Description\', @value=N\'备份指向的表\' , @level0type=N\'SCHEMA\',@level0name=N\'dbo\', @level1type=N\'TABLE\',@level1name=N\'DBData_ArchiveConfig\', @level2type=N\'COLUMN\',@level2name=N\'TargetTable\' GO EXEC sys.sp_addextendedproperty @name=N\'MS_Description\', @value=N\'归档条件\' , @level0type=N\'SCHEMA\',@level0name=N\'dbo\', @level1type=N\'TABLE\',@level1name=N\'DBData_ArchiveConfig\', @level2type=N\'COLUMN\',@level2name=N\'Prerequisite\' GO EXEC sys.sp_addextendedproperty @name=N\'MS_Description\', @value=N\'循环中一次归档删除的数据量\' , @level0type=N\'SCHEMA\',@level0name=N\'dbo\', @level1type=N\'TABLE\',@level1name=N\'DBData_ArchiveConfig\', @level2type=N\'COLUMN\',@level2name=N\'DelMaxQTY\' GO EXEC sys.sp_addextendedproperty @name=N\'MS_Description\', @value=N\'此为 备用字段，考虑可能有些表，会和其他表关联\' , @level0type=N\'SCHEMA\',@level0name=N\'dbo\', @level1type=N\'TABLE\',@level1name=N\'DBData_ArchiveConfig\', @level2type=N\'COLUMN\',@level2name=N\'IsCheckOrderID\' GO EXEC sys.sp_addextendedproperty @name=N\'MS_Description\', @value=N\'为提高并发度，一个DB对应的归档SP可能是多个，通过此列，进行分组。\' , @level0type=N\'SCHEMA\',@level0name=N\'dbo\', @level1type=N\'TABLE\',@level1name=N\'DBData_ArchiveConfig\', @level2type=N\'COLUMN\',@level2name=N\'SP_Name\' GO EXEC sys.sp_addextendedproperty @name=N\'MS_Description\', @value=N\'此为拓展字段，原计划根据 开始时间、结束时间，每天可以多个时间段内执行\' , @level0type=N\'SCHEMA\',@level0name=N\'dbo\', @level1type=N\'TABLE\',@level1name=N\'DBData_ArchiveConfig\', @level2type=N\'COLUMN\',@level2name=N\'StartTime\' GO EXEC sys.sp_addextendedproperty @name=N\'MS_Description\', @value=N\'此为拓展字段，原计划根据 开始时间、结束时间，每天可以多个时间段内执行\' , @level0type=N\'SCHEMA\',@level0name=N\'dbo\', @level1type=N\'TABLE\',@level1name=N\'DBData_ArchiveConfig\', @level2type=N\'COLUMN\',@level2name=N\'EndTime\' GO
2.2 归档运行的Log表
CREATE TABLE [dbo].[DBData_ArchiveLog]( [ID] [int] IDENTITY(1,1) NOT NULL, [IP] [varchar](30) NULL, [DBName] [varchar](30) NULL, [DataTable] [varchar](80) NULL, [BakQTY] [varchar](30) NULL, [BakStartDate] [datetime] NULL, [BakEndDate] [datetime] NULL ) ON [PRIMARY] GO
2.3 异常错误信息表
执行的过程中会外包一层 try...catch,将操作过程中的错误信息保存在表 DBData_ArchiveErrLog。表结构如下：
CREATE TABLE [dbo].[DBData_ArchiveErrLog]( [ID] [int] IDENTITY(1,1) NOT NULL, [IP] [varchar](30) NULL, [DBName] [varchar](60) NULL, [DataTable] [varchar](80) NULL, [TargetIP] [varchar](30) NULL, [TargetDB] [varchar](60) NULL, [TargetTable] [varchar](80) NULL, [Errormsg] [nvarchar](max) NULL, [TransDateTime] [varchar](30) NULL ) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY] GO
四. 存储过程相应的主要代码
SET ANSI_NULLS ON GO SET QUOTED_IDENTIFIER ON GO -- ============================================= -- Author: <Author,,Name> -- Create date: <Create Date,,> -- Description: <Description,,> -- ============================================= CREATE PROCEDURE [dbo].[SP_XXXXX_DataArchive] AS SET NOCOUNT ON; DECLARE @sql1 VARCHAR(MAX) DECLARE @sql VARCHAR(MAX) DECLARE @sql2 VARCHAR(MAX) DECLARE @IP VARCHAR(MAX) DECLARE @DBName VARCHAR(MAX) DECLARE @DataTable VARCHAR(MAX) DECLARE @TargetIP VARCHAR(MAX) DECLARE @TargetDB VARCHAR(MAX) DECLARE @TargetTable VARCHAR(MAX) DECLARE @Prerequisite VARCHAR(MAX) DECLARE @DelMaxQTY INT DECLARE @StartTime DATETIME DECLARE @EndTime DATETIME DECLARE @qty INT DECLARE @ISCHECKORDERID INT ----Carson 2018-12-17 备份数据的时间往往比删除的时间长3倍,因此,如果考虑将备份的操作转移到辅助库,将会对线上的操作影响降至更低 DECLARE @BakDateIP VARCHAR(30) set @BakDateIP=\'[XXX.XXX.XXX.XXX].\'-------后面一定要有一个点 --------------------------------------------------归档操作--------------------------------- DECLARE DBName CURSOR FOR SELECT IP , DBName , DataTable , TargetIP , TargetDB , TargetTable , Prerequisite , DelMaxQTY , ISCHECKORDERID , StartTime , EndTime FROM [中央管理器].[中央管理数据库].[dbo].[DBData_ArchiveConfig] WHERE DataTable <> \'\' AND TargetTable <> \'\' AND DBNAME = \'XXXXXXXXX\' and SP_Name=\'?????\' OPEN DBName FETCH NEXT FROM DBName INTO @IP, @DBName, @DataTable, @TargetIP, @TargetDB, @TargetTable, @Prerequisite, @DelMaxQTY, @ISCHECKORDERID, @StartTime, @EndTime WHILE ( @@fetch_status = 0 ) BEGIN DECLARE @datetime DATETIME IF @ISCHECKORDERID <> \'1\' AND @DataTable <> \'\' BEGIN SET @datetime = CONVERT(VARCHAR(10), GETDATE() - 30, 120) SET @sql = \'Insert into [\' + @TargetIP + \'].\' + @TargetDB + \'.\' + \'dbo.\' + @TargetTable + \' select * FROM \' + @BakDateIP + @DBName + \'.\' + \'dbo.\' + @DataTable + \' with(nolock) where \' + @Prerequisite + \'\' SET @sql1 = \'DECLARE @icount INTEGER SELECT @icount = COUNT(1) FROM \' + @BakDateIP + @DBName + \'.\' + \'dbo.\' + @DataTable + \' where \' + @Prerequisite + \' insert into [中央管理器].[中央管理数据库].dbo.DBData_ArchiveLog (IP, DBName, DataTable, BakQTY, BakStartDate, BakEndDate) select \'\'\' + @IP + \'\'\',\'\'\' + @DBName + \'\'\',\'\'\' + @DataTable + \'\'\',@icount,getdate(),null WHILE @icount > 0 BEGIN DELETE TOP (\' + CAST(@DelMaxQTY AS VARCHAR(10)) + \') FROM \' + @DBName + \'.\' + \'dbo.\' + @DataTable + \' where \' + @Prerequisite + \' SET @icount = @icount -(\' + CAST(@DelMaxQTY AS VARCHAR(10)) + \') WAITFOR DELAY \'\'00:00:01\'\' END \' BEGIN TRY EXEC (@sql) EXEC (@sql1) END TRY SQL Server数据归档的解决方案