Gorm 源码分析

Posted 2021-04-23 明朗万物

This browser does not support music or audio playback. Please play it in Weixin or another browser.

gorm 是 golang web 服务开发者常用的工具，这款 orm 工具极大简化了 DB 相关操作。我个人用 gorm 比较多，基于“知其然知其所以然”的原则，我对 gorm1.0 版本的源码进行了深入阅读，这里记录一下我的学习成果。

一、应用层使用流程

1.初始化：调用 open 得到一个 gorm.DB 对象，表示对当前 DB 链接的抽象，并发安全

2.使用 gorm 提高的链式 api 组装查询逻辑，如

sql. Where("id in (?)", ids).Or("title like (?)", "%"+keyword + "%")

3.调用 gorm 的 Find 接口执行查询，这一步中 gorm 实际进行了多个步骤

prepareSQL() // 拼接查询逻辑，组合成查询语句sql.DB.Query() // 调用底层golang sql api 执行查询

二、核心数据结构

gorm.DB

对 gorm 的 DB 链接进行抽象，该结构的对象可复用，因为基于该结构的后续操作都会先对该对象进行 clone。gorm.DB 对 golang 原生的 sql 包的 DB 数据结构进行了封装。

type DB struct {   Value             interface{}  // 存储查询结果的对象   Error             error        // 查询过程产生的错误信息 RowsAffected int64   callbacks         *Callback    // 回调对象，后续提到   db                sqlCommon    // golang 原生 sql 包的 DB 结构   parent            *DB   search            *search      // 查询条件 logMode int logger logger dialect Dialect singularTable bool source string values map[string]interface{} joinTableHandlers map[string]JoinTableHandler   blockGlobalUpdate bool Ctx context.Context isTestRequest bool shouldSkipTest bool}

gorm.Callback

gorm.DB 对象的 callbacks 保存在一个 gorm.Callback 结构中，全局变量 DefaultCallback 在包初始化时完成初始化，注册多个 callback 。查询时根据查询命令的类别（CURD）执行相应列表中的注册函数。

type Callback struct {   creates    []*func(scope *Scope) // create 操作相关的回调，后面的 field 同理 updates []*func(scope *Scope) deletes []*func(scope *Scope) queries []*func(scope *Scope) rowQueries []*func(scope *Scope) processors []*CallbackProcessor}

gorm.Scope

抽象出一次操作（查询），每个操作都依附于一个链接，链式查询的 api 就作用于该结构上。

// Scope contain current operation's information when you perform any operation on the databasetype Scope struct {   Search          *search   // 一个 SQL 语句的查询条件存在这里 Value interface{} SQL string SQLVars []interface{} db *DB instanceID string primaryKeyField *Field skipLeft bool fields *[]*Field selectAttrs *[]string kind string // 读，写和 unknown}

gorm.search

将查询条件分类，上游 api 的操作结果都在这个结构中存储。执行查询时先将 search 中的所有字段拼接成一个 SQL 语句再查询，拼接 SQL 的函数为 prepareSQL。

type search struct { db *DB whereConditions []map[string]interface{} orConditions []map[string]interface{} notConditions []map[string]interface{} havingConditions []map[string]interface{} joinConditions []map[string]interface{} initAttrs []interface{} assignAttrs []interface{} selects map[string]interface{} omits []string orders []interface{} preload []searchPreload offset interface{} limit interface{} group string tableName string raw bool Unscoped bool ignoreOrderQuery bool}

sql.DB

golang 原生 sql 包中的基础数据结构。主要抽象 db 的连接池，封装查询，对外暴露查询方法 QueryContext 和 Query 用于非事务查询。

// DB is a database handle representing a pool of zero or more// underlying connections. It's safe for concurrent use by multiple// goroutines.//// The sql package creates and frees connections automatically; it// also maintains a free pool of idle connections. If the database has// a concept of per-connection state, such state can be reliably observed// within a transaction (Tx) or connection (Conn). Once DB.Begin is called, the// returned Tx is bound to a single connection. Once Commit or// Rollback is called on the transaction, that transaction's// connection is returned to DB's idle connection pool. The pool size// can be controlled with SetMaxIdleConns.type DB struct { // Atomic access only. At top of struct to prevent mis-alignment // on 32-bit platforms. Of type time.Duration. waitDuration int64 // Total time waited for new connections.
 connector driver.Connector // numClosed is an atomic counter which represents a total number of // closed connections. Stmt.openStmt checks it before cleaning closed // connections in Stmt.css. numClosed uint64
 mu sync.Mutex // protects following fields freeConn []*driverConn connRequests map[uint64]chan connRequest nextRequest uint64 // Next key to use in connRequests. numOpen int // number of opened and pending open connections // Used to signal the need for new connections // a goroutine running connectionOpener() reads on this chan and // maybeOpenNewConnections sends on the chan (one send per needed connection) // It is closed during db.Close(). The close tells the connectionOpener // goroutine to exit. openerCh chan struct{} closed bool dep map[finalCloser]depSet lastPut map[*driverConn]string // stacktrace of last conn's put; debug only maxIdleCount int // zero means defaultMaxIdleConns; negative means 0 maxOpen int // <= 0 means unlimited maxLifetime time.Duration // maximum amount of time a connection may be reused maxIdleTime time.Duration // maximum amount of time a connection may be idle before being closed cleanerCh chan struct{} waitCount int64 // Total number of connections waited for. maxIdleClosed int64 // Total number of connections closed due to idle count. maxIdleTimeClosed int64 // Total number of connections closed due to idle time. maxLifetimeClosed int64 // Total number of connections closed due to max connection lifetime limit.
 stop func() // stop cancels the connection opener and the session resetter.}
// QueryContext executes a query that returns rows, typically a SELECT.// The args are for any placeholder parameters in the query.func (db *DB) QueryContext(ctx context.Context, query string, args ...interface{}) (*Rows, error) { var rows *Rows var err error for i := 0; i < maxBadConnRetries; i++ { rows, err = db.query(ctx, query, args, cachedOrNewConn) if err != driver.ErrBadConn { break } } if err == driver.ErrBadConn { return db.query(ctx, query, args, alwaysNewConn) } return rows, err}
// Query executes a query that returns rows, typically a SELECT.// The args are for any placeholder parameters in the query.func (db *DB) Query(query string, args ...interface{}) (*Rows, error) { return db.QueryContext(context.Background(), query, args...)}

sql.Tx

对 sql.DB 加了锁，封装事务相关功能。向外暴露方法 Exec 和 ExecContext，gorm 在执行事务型查询时依赖这两个方法。

// Tx is an in-progress database transaction.//// A transaction must end with a call to Commit or Rollback.//// After a call to Commit or Rollback, all operations on the// transaction fail with ErrTxDone.//// The statements prepared for a transaction by calling// the transaction's Prepare or Stmt methods are closed// by the call to Commit or Rollback.type Tx struct { db *DB
 // closemu prevents the transaction from closing while there // is an active query. It is held for read during queries // and exclusively during close. closemu sync.RWMutex
 // dc is owned exclusively until Commit or Rollback, at which point // it's returned with putConn. dc *driverConn txi driver.Tx
 // releaseConn is called once the Tx is closed to release // any held driverConn back to the pool. releaseConn func(error)
 // done transitions from 0 to 1 exactly once, on Commit // or Rollback. once done, all operations fail with // ErrTxDone. // Use atomic operations on value when checking value. done int32
 // keepConnOnRollback is true if the driver knows // how to reset the connection's session and if need be discard // the connection. keepConnOnRollback bool
 // All Stmts prepared for this transaction. These will be closed after the // transaction has been committed or rolled back. stmts struct { sync.Mutex v []*Stmt }
 // cancel is called after done transitions from 0 to 1. cancel func()
 // ctx lives for the life of the transaction. ctx context.Context}
// ExecContext executes a query that doesn't return rows.// For example: an INSERT and UPDATE.func (tx *Tx) ExecContext(ctx context.Context, query string, args ...interface{}) (Result, error) { dc, release, err := tx.grabConn(ctx) if err != nil { return nil, err } return tx.db.execDC(ctx, dc, release, query, args)}
// Exec executes a query that doesn't return rows.// For example: an INSERT and UPDATE.func (tx *Tx) Exec(query string, args ...interface{}) (Result, error) { return tx.ExecContext(context.Background(), query, args...)}

三、设计理念

1. 链式语法组合任意查询

2. 将已经存储的分散查询拼接成 SQL查询语句

3. 调用 golang 提供的 driver 查询接口进行查询

四、一些问题

主要是对复杂查询不友好，如用 and 连接多组用 or 连接的查询： 

(id = 2 or keyword like '%xx%' and (name in ('aa', 'bb'))) and (title like '%xxx%' or content like '%xxxx%')

目前难以实现，只能手写查询语句。手写查询语句的话，gorm 的作用就要大打折扣了。

五、Gorm 2.0

Gorm 目前已经发布 2.0 稳定版，官方文档也更新到了 2.0。上一节中提到的问题在 2.0 中有更好的解决方案，后面可以跟进。