python内存数据库pydblite

Posted 2020-08-08

 

Pure-Python engine

最近由于项目开发中发现python informixDB模块对多线程的支持非常不好，当开启两个线程同时连接informix数据库的时候，数据库会报错，显示SQL process正在进行当中，根据python 多线程的机制我们怀疑是连接数据库时，informix将两个线程的cursor识别为同一个，故数据库报错。通过python中multiprocess模块讲所有关于数据库的操作全部改为多进程。

但是这就带来另外一个问题，在之前多线程的情况下，项目中维护着一个Queue，里面存储着若干已经创建好的informix tenant pool 的instance信息，用于客户快速获取可用数据库资源。但是有了多进程的存在，每次客户取一个instance，主进程都需要从Queue中取take一个出来，与此同时take()操作会触发创建一个tenant instance，里边包含一个进程将instance信息再次存储到Queue中。这里会涉及到Queue的进程间通讯的问题。需要将Queue改为multiprocess.queue才能避免数据丢失。这里我想尝试一下用内存数据库来试着简化进程通信的步骤。

• import class Base from module PyDbLite : from PyDbLite import Base
• create a database instance, passing it a path in the file system : db = Base(‘dummy‘)
• for a new database, define the field names : db.create(‘name‘,‘age‘,‘size‘) 
  You don‘t have to define the field types. PyDbLite will accept any value that can be serialized by the cPickle module : strings, Unicode strings, integers, floats, dates and datetimes (instances of the date and datetime classes in the datetime module), user-defined classes, etc
• if the base exists, open it : db.open()
• you can pass a parameter "mode" to the create() method, to specify what you want to do if the base already exists in the file system
  • mode = "open" : db.create(‘name‘,‘age‘,‘size‘,mode="open") opens the database and ignores the field definition
  • mode = "override" : db.create(‘name‘,‘age‘,‘size‘,mode="override") erases the existing base and creates a new one with the field definition
  • if mode is not specified and the base already exists, an IOError is raised
• insert a new record
  • by keywords : db.insert(name=‘homer‘,age=23,size=1.84) 
    If some fields are missing, they are initialized with the value None
  • by positional arguments : db.insert(‘homer‘,23,1.84) 
    The arguments must be provided in the same order as in the create() method
• save the changes on disk : db.commit() 
  If you don‘t commit the changes, the insertion, deletion and update operations will not be saved on disk. To return to the previous version, just open() it again (this is equivalent to rollback in transactional databases)
• besides the fields passed to the create() method, an internal field called __id__ is added. It is a integer which is guaranteed to be unique and unchanged for each record in the base, so that it can be used as the record identifier
• another internal field called __version__ is also managed by the database engine. It is a integer which is set to 0 when the record is created, then incremented by 1 each time the record is updated. This is used to detect concurrency control, for instance in a web application where 2 users select the same record and want to update it at the same time
• the selection of records uses Python list comprehension syntax : 
  recs = [ r for r in db if 30 > r[‘age‘] >= 18 and r[‘size‘] < 2 ] 
  returns the records in the base where the age is between 18 and 30, and size is below 2 meters. The record is a dictionary, where the key is the field name and value is the field value
• Python generator expression syntax can also be used : 
  for r in (r for r in db if r[‘name‘] in (‘homer‘,‘marge‘) ):
      do_something_with(r) 
  iterates on the records where the name is one of ‘homer‘ or ‘marge‘
• to iterate on all the records : 
  for r in db:
      do_something_with(r)
• a record can be accessed by its identifier : record = db[rec_id] returns the record such that record[‘__id__‘] == rec_id
• finally, a shortcut can be used for simple selections : db(key1=val1,key2=val2) returns the list of records where the keys take the given value. It is equivalent to [ r for r in db if r["key1"]==val1 and r["key2"]==val2], but much more concise
• to speed up selections, an index can be created on a field : db.create_index(‘age‘) 
  When an index is created, the database instance has an attribute (here _age : note the heading underscore, to avoid name conflicts with internal names). This attribute is a dictionary-like object, where keys are the values taken by the field, and values are the records whose field values are egal to the key : 
  records = db._age[23] returns the list of records with age == 23 
  If no record has this value, lookup by this value returns an empty list 
  The index supports iteration on the field values, and the keys() method returns all existing values for the field
• number of records in the base : len(db)
• to delete a record : db.delete(record) or, if you know the record identifier : del db[rec_id]
• to delete a list of records : db.delete(list_of_records) 
  list_of_records can be any iterable (list, tuple, set, etc) yielding records
• to update a record : db.update(record,age=24)
• to add a new field to an existing base and specify a default value : db.add_field(‘new_field‘[,default=v]). If no default is provided, the field value is None
• to drop an existing field : db.drop_field(‘name‘)
• to get the list of fields : db.fields

import pydblite
# 使用内存数据库
pydb = pydblite.Base("address")
# 创建a,b,c三个字段
pydb.create(‘a‘, ‘b‘, ‘c‘)
# 为字段a,b创建索引
pydb.create_index(‘a‘, ‘b‘)
# 插入一条数据

pydb.insert(a=0, b=0, c=1)
pydb.insert(a=1, b=0, c=1)
pydb.insert(a=1, b=0, c=1)
pydb.insert(a=1, b=0, c=1)
pydb.update(records=pydb[1],a=2,c="li")
pydb.delete(pydb[0])
# 查询符合特定要求的数据
results = pydb(a=2)
for i in  results:
    print results[i]