sql自动审核工具-inception

Posted 2020-10-01

[inception使用规范及说明文档](http://mysql-inception.github.io/inception-document/usage/)
[代码仓库](https://github.com/smile-java/inception)

inception介绍

inception是去哪网团队开发的一个集审核、执行、备份及生成回滚语句于一身的MySQL自动化运维工具，

可以集成进mysql自动化运维平台来实现sql的自动审核。
开发语言：C/C++ ，在mysql源码的基础上改造的

限制

目前只支持通过C/C++接口、Python接口对inception的访问

inception充当的角色

inception对于自动化应用程序（简称rid）来说是服务器，对于数据库server来说是客户端。当通过自动化平台提交语句（DML/DDL）后，

执行过程可以概括为（如果rid是Python程序）

1. 开发人员提交待审核的sql到rid
2. rid将要访问的数据库的连接串封装到sql语句块的头部，然后调用inception
3. inception对sql进行语法和语义的检查以及按照参数文件中指定的审核项进行审核
4. 审核通过后执行语句
5. 通过解析binlog生成回滚语句保存到参数文件指定的备份库中

流程图如下：

inception安装

可以单独部署到一台主机上，并在此主机上创建备份库

cd /usr/local/src
git clone https://github.com/smile-java/inception
cd inception
# 调用脚本编译安装；指定新生成的文件到目录debug
sh inception_build.sh debug

 编译报错信息

安装inception
    sh inception_build.sh debug    
    CMake Error at cmake/bison.cmake:78 (MESSAGE):
      Bison (GNU parser generator) is required to build MySQL.Please install
       bison.
     
     -- Configuring incomplete, errors occurred!
See also "/data0/sql/inception/debug/CMakeFiles/CMakeOutput.log".
See also "/data0/sql/inception/debug/CMakeFiles/CMakeError.log".
make: *** No rule to make target `install‘.  Stop.

是缺少依赖包导致，解决是安装bison包，然后将debug删除重新编译安装即可

启动inception服务

/data0/sql/inception/debug/sql/Inception --defaults-file=/data0/sql/inception/debug/inc.cnf

注意： 因为Inception支持OSC执行的功能，是通过调用pt-online-schema-change工具来做的，但如果Inception后台启动（&）的话，可能会导致pt-online-schema-change在执行完成之后，长时间不返回，进而导致Inception卡死的问题，这个问题后面会解决，但现阶段请尽量不要使用后台启动的方式，或者可以使用nohup Inception启动命令 &的方式来启动。

inception参数说明

有关审核时参照的规范相关的参数：http://mysql-inception.github.io/inception-document/variables/

有关inception服务器连接的参数

• port
• socket=/自己目录，请自行修改/inc.socket

有关备份库的参数

• inception_remote_backup_host //远程备份库的host
• inception_remote_backup_port //远程备份库的port
• inception_remote_system_user //远程备份库的一个用户
• inception_remote_system_password //上面用户的密码

有关支持OSC相关的参数：http://mysql-inception.github.io/inception-document/osc/

使用案例

调用inception的Python模板

#!/usr/bin/python
#-\\*-coding: utf-8-\\*-
import MySQLdb
sql=‘/*--user=admin;--password=123123;--host=127.0.0.1;--execute=1;--port=3309;*/inception_magic_start;use test;query #语句块inception_magic_commit;‘
try:
    conn=MySQLdb.connect(host=‘127.0.0.1‘,user=‘root‘,passwd=‘‘,db=‘‘,port=123123)
    cur=conn.cursor()
    ret=cur.execute(sql)
    result=cur.fetchall()
    num_fields = len(cur.description)
    field_names = [i[0] for i in cur.description]
    print field_names
    for row in result:
        print row[0], "|",row[1],"|",row[2],"|",row[3],"|",row[4],"|",
        row[5],"|",row[6],"|",row[7],"|",row[8],"|",row[9],"|",row[10]
    cur.close()
    conn.close()
except MySQLdb.Error,e:
     print "Mysql Error %d: %s" % (e.args[0], e.args[1])

DDL操作

如果要调用OSC执行，需要开启参数inception_osc_bin_dir，次参数是会话级别的，每次在提交DDL时可以选择是否通过OSC执行

#登录到inception
mysql -uroot -h127.0.0.1 -p123123
# 使用OSC执行ddl
inception set session inception_osc_bin_dir=‘/usr/local/bin‘

query为：create table inctest(id int);

执行结果输出

 

[‘ID‘, ‘stage‘, ‘errlevel‘, ‘stagestatus‘, ‘errormessage‘, ‘SQL‘, ‘Affected_rows‘, ‘sequence‘, ‘backup_dbname‘, ‘execute_time‘, ‘sqlsha1‘]
1 | CHECKED | 0 | Audit completed | None | use test | 0 | ‘0_0_0‘ | None | 0 | 
2 | CHECKED | 1 | Audit completed | Set engine to innodb for table ‘inctest‘.
Set charset to one of ‘utf8mb4,utf8‘ for table ‘inctest‘.
Set comments for table ‘inctest‘.
Column ‘id‘ in table ‘inctest‘ have no comments.
Column ‘id‘ in table ‘inctest‘ is not allowed to been nullable.
Set Default value for column ‘id‘ in table ‘inctest‘
Set a primary key for table ‘inctest‘. | create table inctest(id int) | 0 | ‘0_0_1‘ | 127_0_0_1_3309_test | 0 | 

errormessage列显示不符合规范的地方，检查的具体项有

• 表必须要有主键，主键为自增，且自增值为1，初始自增值不能大于1
• 表必须有comment、存储引擎必须执行为innodb、表字符集必须是inception参数中配置的其中一个
• 新增的列必须为非空且指定默认值

按规范更改query为：create table inctest(id int unsigned NOT NULL AUTO_INCREMENT comment "id",primary key(id)) ENGINE=InnoDB DEFAULT CHARSET=utf8 comment="test1";

输出为

[‘ID‘, ‘stage‘, ‘errlevel‘, ‘stagestatus‘, ‘errormessage‘, ‘SQL‘, ‘Affected_rows‘, ‘sequence‘, ‘backup_dbname‘, ‘execute_time‘, ‘sqlsha1‘]
1 | RERUN | 0 | Execute Successfully | None | use test | 0 | ‘1502183472_6520_0‘ | None | 0.000 | 
2 | EXECUTED | 0 | Execute Successfully
Backup successfully | None | create table inctest(id int unsigned NOT NULL AUTO_INCREMENT comment "id",primary key(id)) ENGINE=InnoDB DEFAULT CHARSET=utf8 comment="test1" | 0 | ‘1502183472_6520_1‘ | 127_0_0_1_3309_test | 0.110 | 

如果sqlsha1列输出非0，则说明使用到了OSC

问题：execute_time列对应的无值

在备份实例中会生成以mysql服务器IP、端口、库名命名的库，此处为127_0_0_1_3309_test

09:19:23[127_0_0_1_3309_test](;)> show tables;
+------------------------------------+
| Tables_in_127_0_0_1_3309_test      |
+------------------------------------+
| $_$inception_backup_information$_$ |
| inctest                            |
+------------------------------------+
3 rows in set (0.00 sec)

09:19:26[127_0_0_1_3309_test](;)> select * from $_$inception_backup_information$_$;
+-------------------+-------------------+------------------+------------------+----------------+-----------------------------------------------------------------------------------------------------------------------------------------------+-----------+--------+-----------+------+---------------------+-------------+
| opid_time         | start_binlog_file | start_binlog_pos | end_binlog_file  | end_binlog_pos | sql_statement                                                                                                                                 | host      | dbname | tablename | port | time                | type        |
+-------------------+-------------------+------------------+------------------+----------------+-----------------------------------------------------------------------------------------------------------------------------------------------+-----------+--------+-----------+------+---------------------+-------------+
| 1502183472_6520_1 |                   |                0 |                  |              0 | create table inctest(id int unsigned NOT NULL AUTO_INCREMENT comment "id",primary key(id)) ENGINE=InnoDB DEFAULT CHARSET=utf8 comment="test1" | 127.0.0.1 | test   | inctest   | 3309 | 2017-08-08 17:11:12 | CREATETABLE |
+-------------------+-------------------+------------------+------------------+----------------+-----------------------------------------------------------------------------------------------------------------------------------------------+-----------+--------+-----------+------+---------------------+-------------+
8 rows in set (0.00 sec)

09:19:59[127_0_0_1_3309_test](;)> select * from inctest;
+----+------------------------------+-------------------+
| id | rollback_statement           | opid_time         |
+----+------------------------------+-------------------+
|  2 | DROP TABLE `test`.`inctest`; | 1502183472_6520_1 |
+----+------------------------------+-------------------+
2 rows in set (0.00 sec)

表$_$inception_backup_information$_$记录的是inception的操作日志

回滚sql存储在和原操作表同名的表中，opid_time是执行语句的唯一序列号，如果知道执行sql的序列号，想要获得对应的回滚sql，可以执行

select rollback_statement from 127_0_0_1_3309_test.inctest where opid_time =‘1502183472_6520_1’;