database/sql query 超时设置

Posted 2022-12-12 Q博士

参考文章:Query

问题描述

接上文，mysql的同步访问问题解决后，继续压测，发现访问mysql的耗时逐渐增大，影响模块整体处理能力。我设置了readTimeout，没有任何反应，请求耗时还是会超过readTimeout所设置的值。因为：

• readTimeout只能限制连接数据读取时间，如果程序发生在获取连接前等待时间过长，无法通过这个参数设置，且readTimeout*3才是真正的读取超时时间，因为会重试3次。
• 当设置的MaxOpenConn消耗殆尽时，再次进入的请求会夯死在database/sql内部，直到有空闲连接可操作。

显然Query方法是无法满足我们的需求。

解决方法

翻源码，发现了一个新方法QueryContext,通过传入context，该context有超时时间，那么就达到了该查询请求有了超时设置，nice。

// conn returns a newly-opened or cached *driverConn.
func (db *DB) conn(ctx context.Context, strategy connReuseStrategy) (*driverConn, error) 
    db.mu.Lock()
    if db.closed 
        db.mu.Unlock()
        return nil, errDBClosed
    
    // Check if the context is expired.
    select 
    default:
    case <-ctx.Done():
        db.mu.Unlock()
        return nil, ctx.Err()
    
    lifetime := db.maxLifetime

    // Prefer a free connection, if possible.
    numFree := len(db.freeConn)
    if strategy == cachedOrNewConn && numFree > 0 
        conn := db.freeConn[0]
        copy(db.freeConn, db.freeConn[1:])
        db.freeConn = db.freeConn[:numFree-1]
        conn.inUse = true
        db.mu.Unlock()
        if conn.expired(lifetime) 
            conn.Close()
            return nil, driver.ErrBadConn
        
        // Lock around reading lastErr to ensure the session resetter finished.
        conn.Lock()
        err := conn.lastErr
        conn.Unlock()
        if err == driver.ErrBadConn 
            conn.Close()
            return nil, driver.ErrBadConn
        
        return conn, nil
    

    // Out of free connections or we were asked not to use one. If we're not
    // allowed to open any more connections, make a request and wait.
    if db.maxOpen > 0 && db.numOpen >= db.maxOpen 
        // Make the connRequest channel. It's buffered so that the
        // connectionOpener doesn't block while waiting for the req to be read.
        req := make(chan connRequest, 1)
        reqKey := db.nextRequestKeyLocked()
        db.connRequests[reqKey] = req
        db.waitCount++
        db.mu.Unlock()

        waitStart := time.Now()

        // Timeout the connection request with the context.
        select 
        case <-ctx.Done():
            // Remove the connection request and ensure no value has been sent
            // on it after removing.
            db.mu.Lock()
            delete(db.connRequests, reqKey)
            db.mu.Unlock()

            atomic.AddInt64(&db.waitDuration, int64(time.Since(waitStart)))

            select 
            default:
            case ret, ok := <-req:
                if ok && ret.conn != nil 
                    db.putConn(ret.conn, ret.err, false)
                
            
            return nil, ctx.Err()
        case ret, ok := <-req:
            atomic.AddInt64(&db.waitDuration, int64(time.Since(waitStart)))

            if !ok 
                return nil, errDBClosed
            
            if ret.err == nil && ret.conn.expired(lifetime) 
                ret.conn.Close()
                return nil, driver.ErrBadConn
            
            if ret.conn == nil 
                return nil, ret.err
            
            // Lock around reading lastErr to ensure the session resetter finished.
            ret.conn.Lock()
            err := ret.conn.lastErr
            ret.conn.Unlock()
            if err == driver.ErrBadConn 
                ret.conn.Close()
                return nil, driver.ErrBadConn
            
            return ret.conn, ret.err
        
    

    db.numOpen++ // optimistically
    db.mu.Unlock()
    ci, err := db.connector.Connect(ctx)
    if err != nil 
        db.mu.Lock()
        db.numOpen-- // correct for earlier optimism
        db.maybeOpenNewConnections()
        db.mu.Unlock()
        return nil, err
    
    db.mu.Lock()
    dc := &driverConn
        db:        db,
        createdAt: nowFunc(),
        ci:        ci,
        inUse:     true,
    
    db.addDepLocked(dc, dc)
    db.mu.Unlock()
    return dc, nil

使用该访问进行query查询

sqlCtx, _ := context.WithTimeout(context.Background(), time.Duration(p.queryTimeout)*time.Millisecond)
     //defer cancel()
 rows, err := p.orderClient.QueryContext(sqlCtx, sqlStr)

压测数据

时间单位都是ms

thread_num	mysql_query_timeout	max_conn_num	max_idle_conn	qps	mysql_err_num	mysql_err_max_time	mysql_succ_num	mysql_succ_max_time
1000	1000	0	50	4000	10	1500	6280	317
1000	1000	50	50	3500	135	1020	5370	833
1000	2000	50	50	3300	39	2020	5040	1172
400	2000	50	50	2800	3	2015	4620	640
400	1000	50	50	2800	9	1020	4646	484

数据显示SetMaxOpenConns为0的情况下，设置context是没有效果的。但是>0后，该timeout设置有效，原因也很好理解，因为maxOpenConn=0时，请求不会等待空闲连接。所以也不会有ctx.Done()逻辑存在。