[ pymysql ] 三种插入方式的速度对比

Posted ponponon

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了[ pymysql ] 三种插入方式的速度对比相关的知识，希望对你有一定的参考价值。

  • 方案一：使用 executemany 一次 + commit 一次
  • 方案二：使用 execute 10000 次 + commit 一次
  • 方案三：使用 （execute + commit） 都 10000 次

生成伪数据

fake: Faker = Faker(locale=\'zh_CN\')

start = time.time()
tables = []
for i in range(10000):
    row = (fake.name(), fake.text())
    tables.append(row)
    # print(row)

end = time.time()
print(f\'生成数据完成！ pay time is {end - start} s\')

connection: Connection = pymysql.connect(
    user=\'root\',
    password=\'yourpassword\',
    host=\'192.168.31.203\',
    port=3306,
    database=\'test_001\'
)

方案一

print(\'开始方案一......\')

with connection:
    # 创建数据表
    with connection.cursor() as cursor:
        cursor: Cursor
        sql = """
        create table if not exists `tweet`(
            `id` bigint NOT NULL AUTO_INCREMENT,
            `username` varchar(255) NOT NULL,
            `content` varchar(255) NOT NULL,
            PRIMARY KEY (`id`)
        )
        """
        cursor.execute(sql)
    start = time.time()

    # 插入数据
    with connection.cursor() as cursor:
        cursor: Cursor

        sql = \'insert into tweet (`username`,`content`) values (%s,%s) \'
        cursor.executemany(sql, tables)

        connection.commit()

    end = time.time()
    print(f\'方案一：插入数据完成！ pay time is {end - start} s\')

    # 删除数据表
    with connection.cursor() as cursor:
        cursor: Cursor
        sql = \'drop table tweet;\'
        cursor.execute(sql)

方案二

print(\'开始方案二......\')
connection: Connection = pymysql.connect(
    user=\'root\',
    password=\'yourpassword\',
    host=\'192.168.31.203\',
    port=3306,
    database=\'test_001\'
)
with connection:
    # 创建数据表
    with connection.cursor() as cursor:
        cursor: Cursor
        sql = """
        create table if not exists `tweet`(
            `id` bigint NOT NULL AUTO_INCREMENT,
            `username` varchar(255) NOT NULL,
            `content` varchar(255) NOT NULL,
            PRIMARY KEY (`id`)
        )
        """
        cursor.execute(sql)
    start = time.time()

    # 插入数据
    with connection.cursor() as cursor:
        cursor: Cursor

        for row in tables:
            sql = \'insert into tweet (`username`,`content`) values (%s,%s) \'
            cursor.execute(sql, row)

        connection.commit()

    end = time.time()
    print(f\'方案二：插入数据完成！ pay time is {end - start} s\')

    # 删除数据表
    with connection.cursor() as cursor:
        cursor: Cursor
        sql = \'drop table tweet;\'
        cursor.execute(sql)

方案三

print(\'开始方案三......\')
connection: Connection = pymysql.connect(
    user=\'root\',
    password=\'yourpassword\',
    host=\'192.168.31.203\',
    port=3306,
    database=\'test_001\'
)
with connection:
    # 创建数据表
    with connection.cursor() as cursor:
        cursor: Cursor
        sql = """
        create table if not exists `tweet`(
            `id` bigint NOT NULL AUTO_INCREMENT,
            `username` varchar(255) NOT NULL,
            `content` varchar(255) NOT NULL,
            PRIMARY KEY (`id`)
        )
        """
        cursor.execute(sql)
    start = time.time()

    # 插入数据
    with connection.cursor() as cursor:
        cursor: Cursor

        for row in tables:
            sql = \'insert into tweet (`username`,`content`) values (%s,%s) \'
            cursor.execute(sql, row)
            connection.commit()

    end = time.time()
    print(f\'方案三：插入数据完成！ pay time is {end - start} s\')

    # 删除数据表
    with connection.cursor() as cursor:
        cursor: Cursor
        sql = \'drop table tweet;\'
        cursor.execute(sql)

运行结果

生成数据完成！ pay time is 1.2480072975158691 s
开始方案一......
方案一：插入数据完成！ pay time is 2.1816399097442627 s
开始方案二......
方案二：插入数据完成！ pay time is 27.0793399810791 s
开始方案三......
方案三：插入数据完成！ pay time is 247.8990249633789 s

可以看到

速度：方案一 > 方案二 > 方案三

完整代码

import pymysql
from pymysql.connections import Connection
from pymysql.cursors import Cursor
from faker import Faker
import time

fake: Faker = Faker(locale=\'zh_CN\')

start = time.time()
tables = []
for i in range(10000):
    row = (fake.name(), fake.text())
    tables.append(row)
    # print(row)

end = time.time()
print(f\'生成数据完成！ pay time is {end - start} s\')

connection: Connection = pymysql.connect(
    user=\'root\',
    password=\'yourpassword\',
    host=\'192.168.31.203\',
    port=3306,
    database=\'test_001\'
)

print(\'开始方案一......\')

with connection:
    # 创建数据表
    with connection.cursor() as cursor:
        cursor: Cursor
        sql = """
        create table if not exists `tweet`(
            `id` bigint NOT NULL AUTO_INCREMENT,
            `username` varchar(255) NOT NULL,
            `content` varchar(255) NOT NULL,
            PRIMARY KEY (`id`)
        )
        """
        cursor.execute(sql)
    start = time.time()

    # 插入数据
    with connection.cursor() as cursor:
        cursor: Cursor

        sql = \'insert into tweet (`username`,`content`) values (%s,%s) \'
        cursor.executemany(sql, tables)

        connection.commit()

    end = time.time()
    print(f\'方案一：插入数据完成！ pay time is {end - start} s\')

    # 删除数据表
    with connection.cursor() as cursor:
        cursor: Cursor
        sql = \'drop table tweet;\'
        cursor.execute(sql)

# --------------------------------------------------------------------

print(\'开始方案二......\')
connection: Connection = pymysql.connect(
    user=\'root\',
    password=\'yourpassword\',
    host=\'192.168.31.203\',
    port=3306,
    database=\'test_001\'
)
with connection:
    # 创建数据表
    with connection.cursor() as cursor:
        cursor: Cursor
        sql = """
        create table if not exists `tweet`(
            `id` bigint NOT NULL AUTO_INCREMENT,
            `username` varchar(255) NOT NULL,
            `content` varchar(255) NOT NULL,
            PRIMARY KEY (`id`)
        )
        """
        cursor.execute(sql)
    start = time.time()

    # 插入数据
    with connection.cursor() as cursor:
        cursor: Cursor

        for row in tables:
            sql = \'insert into tweet (`username`,`content`) values (%s,%s) \'
            cursor.execute(sql, row)

        connection.commit()

    end = time.time()
    print(f\'方案二：插入数据完成！ pay time is {end - start} s\')

    # 删除数据表
    with connection.cursor() as cursor:
        cursor: Cursor
        sql = \'drop table tweet;\'
        cursor.execute(sql)

# --------------------------------------------------------------------

print(\'开始方案三......\')
connection: Connection = pymysql.connect(
    user=\'root\',
    password=\'yourpassword\',
    host=\'192.168.31.203\',
    port=3306,
    database=\'test_001\'
)
with connection:
    # 创建数据表
    with connection.cursor() as cursor:
        cursor: Cursor
        sql = """
        create table if not exists `tweet`(
            `id` bigint NOT NULL AUTO_INCREMENT,
            `username` varchar(255) NOT NULL,
            `content` varchar(255) NOT NULL,
            PRIMARY KEY (`id`)
        )
        """
        cursor.execute(sql)
    start = time.time()

    # 插入数据
    with connection.cursor() as cursor:
        cursor: Cursor

        for row in tables:
            sql = \'insert into tweet (`username`,`content`) values (%s,%s) \'
            cursor.execute(sql, row)
            connection.commit()

    end = time.time()
    print(f\'方案三：插入数据完成！ pay time is {end - start} s\')

    # 删除数据表
    with connection.cursor() as cursor:
        cursor: Cursor
        sql = \'drop table tweet;\'
        cursor.execute(sql)

以上是关于[ pymysql ] 三种插入方式的速度对比的主要内容，如果未能解决你的问题，请参考以下文章

Python之PyMySQL的使用

moviepy处理视频帧和遍历的方式处理视频帧速度对比。

PHP大批量插入数据库的3种方法和速度对比

mysqlclient和PyMySQL对比

MySQL高级之PyMySQL的使用

常见的三种注解注入方式对比