聊聊storagetapper的pipe

Posted 2021-09-27 codecraft

序

本文主要研究一下storagetapper的pipe

Pipe

storagetapper/pipe/pipe.go

type Pipe interface {
    NewConsumer(topic string) (Consumer, error)
    NewProducer(topic string) (Producer, error)
    Type() string
    Config() *config.PipeConfig
    Close() error
}

Pipe接口定义了NewConsumer、NewProducer、Type、Config、Close方法

Consumer

storagetapper/pipe/pipe.go

type Consumer interface {
    Close() error
    //CloseOnFailure doesn\'t save offsets
    CloseOnFailure() error
    Message() chan interface{}
    Error() chan error
    FetchNext() (interface{}, error)
    //Allows to explicitly persists current consumer position
    SaveOffset() error

    //SetFormat allow to tell consumer the format of the file when there is no
    //header
    SetFormat(format string)
}

Consumer接口定义了Close、CloseOnFailure、Message、Error、FetchNext、SaveOffset、SetFormat方法

Producer

storagetapper/pipe/pipe.go

type Producer interface {
    Push(data interface{}) error
    PushK(key string, data interface{}) error
    PushSchema(key string, data []byte) error
    //PushBatch queues the messages instead of sending immediately
    PushBatch(key string, data interface{}) error
    //PushCommit writes out all the messages queued by PushBatch
    PushBatchCommit() error
    Close() error
    CloseOnFailure() error

    SetFormat(format string)

    PartitionKey(source string, key string) string
}

Producer接口定义了Push、PushK、PushSchema、PushBatch、PushBatchCommit、Close、CloseOnFailure、SetFormat、PartitionKey

Create

storagetapper/pipe/pipe.go

func Create(pipeType string, cfg *config.PipeConfig, db *sql.DB) (Pipe, error) {

    init := Pipes[strings.ToLower(pipeType)]
    if init == nil {
        return nil, fmt.Errorf("unsupported pipe: %s", strings.ToLower(pipeType))
    }

    pipe, err := init(cfg, db)
    if err != nil {
        return nil, err
    }

    return pipe, nil
}

type constructor func(cfg *config.PipeConfig, db *sql.DB) (Pipe, error)

//Pipes is the list of registered pipes
//Plugins insert their constructors into this map
var Pipes map[string]constructor

//registerPlugin should be called from plugin\'s init
func registerPlugin(name string, init constructor) {
    if Pipes == nil {
        Pipes = make(map[string]constructor)
    }
    Pipes[name] = init
}

Create方法根据pipeType、PipeConfig、db来创建pipe

小结

storagetapper的Pipe接口定义了NewConsumer、NewProducer、Type、Config、Close方法，其Create方法根据pipeType、PipeConfig、db来创建pipe。

doc

• storagetapper