Golang ETCD包的安装使用

Posted HaimaBlog

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Golang ETCD包的安装使用相关的知识，希望对你有一定的参考价值。

目录

视频地址

https://www.bilibili.com/video/BV19J41147uT?p=7

课件资料

https://www.liwenzhou.com/posts/Go/go_etcd/

ETCD

日志收集项目

为什么要自己写不用ELK?

ELK: 部署的时候麻烦每一个filebeat都需要配置一个配置文件

使用etcd来管理被收集的日志项。

项目的架构

上节课项目进度

  1. kafka：消息队列
  2. tailf：从文件里读日志
  3. go-ini：解析配置文件

今日内容

etcd

etcd介绍

详见群文件：etcd.pdf

http://www.5lmh.com/数据库操作/go操作etcd/操作etcd.html

安装etcd

详见：https://docs.qq.com/doc/DTndrQXdXYUxUU09O

Ubuntu(linux系统)安装etcd

方法一：
下载：etcd-v3.3.18-linux-amd64
解压文件即可使用，无需安装

方法二：
https://github.com/etcd-io/etcd/releases/tag/v3.4.14

新建 install_etcd.sh

ETCD_VER=v3.4.14

# choose either URL
GOOGLE_URL=https://storage.googleapis.com/etcd
GITHUB_URL=https://github.com/etcd-io/etcd/releases/download
DOWNLOAD_URL=${GITHUB_URL}

rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
rm -rf /tmp/etcd-download-test && mkdir -p /tmp/etcd-download-test

curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
#curl -L https://github.com/etcd-io/etcd/releases/download/v3.4.14/etcd-v3.4.14-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz

tar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /tmp/etcd-download-test --strip-components=1
#rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
mv /tmp/etcd-download-test /opt/etcd-v3.4.14-linux-amd64
cd /opt/etcd-v3.4.14-linux-amd64
/opt/etcd-v3.4.14-linux-amd64/etcd --version
/opt/etcd-v3.4.14-linux-amd64/etcdctl version



# start a local etcd server
/opt/etcd-v3.4.14-linux-amd64/etcd

# write,read to etcd
/opt/etcd-v3.4.14-linux-amd64/etcdctl --endpoints=localhost:2379 put foo bar
/opt/etcd-v3.4.14-linux-amd64/etcdctl --endpoints=localhost:2379 get foo

#加入系统环境变量
ln -s /opt/etcd-v3.4.14-linux-amd64/etcdctl /usr/bin/etcdctl

命令行操作etcd

注意一定要使用V3版本的API

启动服务端

在命令行中

进入解压出来的目录中/opt/etcd-v3.3.18-linux-amd64

root@haima-PC:/opt/etcd-v3.3.18-linux-amd64# su root  //切换到root账号
root@haima-PC:/opt/etcd-v3.3.18-linux-amd64# ./etcd   //启动服务端
2020-05-19 07:14:33.832992 I | etcdmain: etcd Version: 3.3.18
2020-05-19 07:14:33.833045 I | etcdmain: Git SHA: 3c8740a79
2020-05-19 07:14:33.833054 I | etcdmain: Go Version: go1.12.9
2020-05-19 07:14:33.833063 I | etcdmain: Go OS/Arch: linux/amd64
2020-05-19 07:14:33.833072 I | etcdmain: setting maximum number of CPUs to 4, total number of available CPUs is 4
2020-05-19 07:14:33.833085 W | etcdmain: no data-dir provided, using default data-dir ./default.etcd
2020-05-19 07:14:33.833443 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2020-05-19 07:14:33.836644 I | embed: listening for peers on http://localhost:2380
2020-05-19 07:14:33.836736 I | embed: listening for client requests on localhost:2379
2020-05-19 07:14:33.849127 I | etcdserver: name = default
2020-05-19 07:14:33.849150 I | etcdserver: data dir = default.etcd
2020-05-19 07:14:33.849165 I | etcdserver: member dir = default.etcd/member
2020-05-19 07:14:33.849174 I | etcdserver: heartbeat = 100ms
2020-05-19 07:14:33.849184 I | etcdserver: election = 1000ms
2020-05-19 07:14:33.849194 I | etcdserver: snapshot count = 100000
2020-05-19 07:14:33.849228 I | etcdserver: advertise client URLs = http://localhost:2379
2020-05-19 07:14:33.851066 I | etcdserver: restarting member 8e9e05c52164694d in cluster cdf818194e3a8c32 at commit index 8
2020-05-19 07:14:33.851102 I | raft: 8e9e05c52164694d became follower at term 2
2020-05-19 07:14:33.851117 I | raft: newRaft 8e9e05c52164694d [peers: [], term: 2, commit: 8, applied: 0, lastindex: 8, lastterm: 2]
2020-05-19 07:14:33.867515 W | auth: simple token is not cryptographically signed
2020-05-19 07:14:33.871849 I | etcdserver: starting server... [version: 3.3.18, cluster version: to_be_decided]
2020-05-19 07:14:33.878161 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
2020-05-19 07:14:33.878282 N | etcdserver/membership: set the initial cluster version to 3.3
2020-05-19 07:14:33.878340 I | etcdserver/api: enabled capabilities for version 3.3
2020-05-19 07:14:35.352464 I | raft: 8e9e05c52164694d is starting a new election at term 2
2020-05-19 07:14:35.352492 I | raft: 8e9e05c52164694d became candidate at term 3
2020-05-19 07:14:35.352510 I | raft: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 3
2020-05-19 07:14:35.352526 I | raft: 8e9e05c52164694d became leader at term 3
2020-05-19 07:14:35.352541 I | raft: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 3
2020-05-19 07:14:35.352898 I | etcdserver: published {Name:default ClientURLs:[http://localhost:2379]} to cluster cdf818194e3a8c32
2020-05-19 07:14:35.352917 I | embed: ready to serve client requests
2020-05-19 07:14:35.353073 E | etcdmain: forgot to set Type=notify in systemd service file?
2020-05-19 07:14:35.353526 N | embed: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!

启动成功后，

编写后台启动服务角本

start.sh

#!/bin/bash
echo "删除etcd进程"
killall etcd
echo "启动etcd进程"
nohup ./etcd >> etcd.log 2>&1 & # 日志文件输出到/tmp/etcd.log目录

ps -aux | grep etcd

后台运行了，可以关闭黑窗口

root@haima-PC:/opt/etcd-v3.4.14-linux-amd64# ./start.sh 
删除etcd进程
etcd: no process found
启动etcd进程
root      5746  0.0  0.1 10612352 18648 pts/2  Sl+  12:22   0:00 ./etcd
root      5748  0.0  0.0  14540   932 pts/2    S+   12:22   0:00 grep etcd

编写停止服务角本

stop.sh

#!/bin/bash
echo "删除etcd进程"
killall etcd
ps -aux | grep etcd

停止etcd服务

root@haima-PC:/opt/etcd-v3.4.14-linux-amd64# ./stop.sh 
删除etcd进程
root      7233  0.0  0.0  14540   964 pts/2    S+   12:23   0:00 grep etcd

客户端操作

再开启一个命令行窗口

先设置环境变量：

Windows执行：

SET ETCDCTL_API=3

Mac或者Linux 执行

export ETCDCTL_API=3

如果不设置上面的环境变量，会报下面的错误：

root@haima-PC:/opt/etcd-v3.3.18-linux-amd64# ./etcdctl --endpoints=http://127.0.0.1:2379 put name lisi
No help topic for \'put\'

命令行操作：使用自带的etcdctl 命令操作
进入解压出来的/opt/etcd-v3.3.18-linux-amd64目录下

PUT：

./etcdctl --endpoints=http://127.0.0.1:2379 put key value

GET：

./etcdctl --endpoints=http://127.0.0.1:2379 GET key

DEL:

./etcdctl --endpoints=http://127.0.0.1:2379 DEL key

haima@haima-PC:/opt/etcd-v3.3.18-linux-amd64$ su
密码：
root@haima-PC:/opt/etcd-v3.3.18-linux-amd64# ./etcdctl --endpoints=http://127.0.0.1:2379 put name lisi
No help topic for \'put\'
root@haima-PC:/opt/etcd-v3.3.18-linux-amd64# export ETCDCTL_API=3
root@haima-PC:/opt/etcd-v3.3.18-linux-amd64# ./etcdctl --endpoints=http://127.0.0.1:2379 put name lisi
OK
root@haima-PC:/opt/etcd-v3.3.18-linux-amd64# ./etcdctl --endpoints=http://127.0.0.1:2379 get name
name
lisi

go操作etcd

go mod设置代理

SET GOPROXY=https://goproxy.cn // Windows
export GOPROXY=https://goproxy.cn // Mac&Linux

PUT/GET

新建 etcd_put/main.go

package main

import (
	"context"
	"fmt"
	"go.etcd.io/etcd/clientv3"
	"time"
)

// etcd client put/get demo
// use etcd/clientv3
func main() {
	cli, err := clientv3.New(clientv3.Config{
		Endpoints:   []string{"127.0.0.1:2379"},
		DialTimeout: 5 * time.Second,
	})
	if err != nil {
		// handle error!
		fmt.Printf("connect to etcd failed, err:%v\\n", err)
		return
	}
	fmt.Println("connect to etcd success")

	defer cli.Close()

	// put
	ctx, cancel := context.WithTimeout(context.Background(), time.Second)
	//value := `[{"path":"c:/tmp/nginx.log","topic":"web.log"},{"path":"d:/xxx/redis.log","topic":"redis.log"}]`
	value := `[{"path":"c:/tmp/nginx.log","topic":"web.log"},{"path":"d:/xxx/redis.log","topic":"redis.log"},{"path":"d:/xxx/mysql.log","topic":"mysql.log"}]`
	_, err = cli.Put(ctx, "/logagent/collect_config", value)
	//_, err = cli.Put(ctx, "baodelu", "dsb")
	cancel()
	if err != nil {
		fmt.Printf("put to etcd failed, err:%v\\n", err)
		return
	}

	// get
	ctx, cancel = context.WithTimeout(context.Background(), time.Second)
	resp, err := cli.Get(ctx, "/logagent/collect_config")
	cancel()
	if err != nil {
		fmt.Printf("get from etcd failed, err:%v\\n", err)
		return
	}
	for _, ev := range resp.Kvs {
		fmt.Printf("%s:%s\\n", ev.Key, ev.Value)
	}
}

先运行服务端，
再启动go代码
首次运行go mod报错

go mod init
go mod tidy

go: haimait/learn/etcd/etcd_put imports
        go.etcd.io/etcd/clientv3 tested by
        go.etcd.io/etcd/clientv3.test imports
        github.com/coreos/etcd/auth imports
        github.com/coreos/etcd/mvcc/backend imports
        github.com/coreos/bbolt: github.com/coreos/bbolt@v1.3.4: parsing go.mod:
        module declares its path as: go.etcd.io/bbolt
                but was required as: github.com/coreos/bbolt

我的go版本是（go version 可以查看版本）

go version go1.14.2 linux/amd64

报错原因：
go.mod中更改下etcd的版本号，估计是etcd版本用的"google.golang.org/grpc/resolver"版本比较新;

解决方法：
在go.mod里写上

module haimait/learn/etcd/etcd_watch

go 1.14
require (
	github.com/coreos/bbolt v1.3.3 // indirect
	github.com/coreos/etcd v3.3.21+incompatible // indirect
        go.etcd.io/etcd v3.3.13+incompatible    
)

再次执行

haima@haima-PC:/media/haima/34E401CC64DD0E28/site/go/src/haimait/learn/etcd/etcd_put$ go mod tidy
go: finding module for package google.golang.org/grpc/keepalive
go: finding module for package golang.org/x/crypto/bcrypt
go: finding module for package google.golang.org/grpc/grpclog
go: finding module for package google.golang.org/grpc/health/grpc_health_v1
go: finding module for package google.golang.org/grpc/metadata
go: finding module for package google.golang.org/grpc/credentials
go: finding module for package google.golang.org/grpc/codes
go: finding module for package github.com/gogo/protobuf/gogoproto
go: finding module for package google.golang.org/grpc/peer
go: finding module for package github.com/dgrijalva/jwt-go
go: finding module for package github.com/coreos/pkg/capnslog
go: finding module for package github.com/prometheus/client_golang/prometheus/promhttp
go: finding module for package google.golang.org/genproto/googleapis/api/annotations
go: finding module for package google.golang.org/grpc/connectivity
go: finding module for package github.com/golang/protobuf/proto
go: finding module for package google.golang.org/grpc/resolver/dns
go: finding module for package google.golang.org/grpc/resolver
go: finding module for package github.com/json-iterator/go
go: finding module for package go.uber.org/zap/zapcore
go: finding module for package go.uber.org/zap
go: finding module for package golang.org/x/time/rate
go: finding module for package golang.org/x/net/trace
go: finding module for package github.com/xiang90/probing
go: finding module for package github.com/grpc-ecosystem/go-grpc-prometheus
go: finding module for package golang.org/x/net/context
go: finding module for package github.com/grpc-ecosystem/grpc-gateway/utilities
go: finding module for package github.com/coreos/go-semver/semver
go: finding module for package github.com/jonboulle/clockwork
go: finding module for package github.com/modern-go/reflect2
go: finding module for package sigs.k8s.io/yaml
go: finding module for package google.golang.org/grpc/resolver/passthrough
go: finding module for package github.com/soheilhy/cmux
go: finding module for package google.golang.org/grpc/balancer
go: finding module for package github.com/coreos/go-systemd/journal
go: finding module for package google.golang.org/grpc
go: finding module for package github.com/google/btree
go: finding module for package google.golang.org/grpc/health
go: finding module for package github.com/tmc/grpc-websocket-proxy/wsproxy
go: finding module for package github.com/dustin/go-humanize
go: finding module for package github.com/grpc-ecosystem/go-grpc-middleware
go: finding module for package github.com/prometheus/client_golang/prometheus
go: finding module for package github.com/prometheus/client_model/go
go: finding module for package github.com/gogo/protobuf/proto
go: finding module for package google.golang.org/grpc/status
go: finding module for package github.com/grpc-ecosystem/grpc-gateway/runtime
go: finding module for package golang.org/x/sys/unix
go: finding module for package go.etcd.io/bbolt
go: finding module for package google.golang.org/grpc/naming
go: finding module for package github.com/google/uuid
go: finding module for package github.com/golang/groupcache/lru
go: found google.golang.org/grpc in google.golang.org/grpc v1.29.1
go: found github.com/gogo/protobuf/gogoproto in github.com/gogo/protobuf v1.3.1
go: found github.com/golang/protobuf/proto in github.com/golang/protobuf v1.4.2
go: found google.golang.org/genproto/googleapis/api/annotations in google.golang.org/genproto v0.0.0-20200515170657-fc4c6c6a6587
go: found github.com/grpc-ecosystem/go-grpc-prometheus in github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
go: found github.com/prometheus/client_golang/prometheus/promhttp in github.com/prometheus/client_golang v1.6.0
go: found golang.org/x/crypto/bcrypt in golang.org/x/crypto v0.0.0-20200510223506-06a226fb4e37
go: found github.com/coreos/pkg/capnslog in github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f
go: found github.com/dgrijalva/jwt-go in github.com/dgrijalva/jwt-go v3.2.0+incompatible
go: found github.com/google/uuid in github.com/google/uuid v1.1.1
go: found go.uber.org/zap in go.uber.org/zap v1.15.0
go: found github.com/soheilhy/cmux in github.com/soheilhy/cmux v0.1.4
go: found github.com/coreos/go-systemd/journal in github.com/coreos/go-systemd v0.0.0-20191104093116-d3cd4ed1dbcf
go: found github.com/coreos/go-semver/semver in github.com/coreos/go-semver v0.3.0
go: found github.com/json-iterator/go in github.com/json-iterator/go v1.1.9
go: found github.com/modern-go/reflect2 in github.com/modern-go/reflect2 v1.0.1
go: found github.com/grpc-ecosystem/grpc-gateway/runtime in github.com/grpc-ecosystem/grpc-gateway v1.14.5
go: found github.com/tmc/grpc-websocket-proxy/wsproxy in github.com/tmc/grpc-websocket-proxy v0.0.0-20200427203606-3cfed13b9966
go: found golang.org/x/net/trace in golang.org/x/net v0.0.0-20200513185701-a91f0712d120
go: found sigs.k8s.io/yaml in sigs.k8s.io/yaml v1.2.0
go: found github.com/jonboulle/clockwork in github.com/jonboulle/clockwork v0.1.0
go: found github.com/dustin/go-humanize in github.com/dustin/go-humanize v1.0.0
go: found github.com/grpc-ecosystem/go-grpc-middleware in github.com/grpc-ecosystem/go-grpc-middleware v1.2.0
go: found golang.org/x/time/rate in golang.org/x/time v0.0.0-20200416051211-89c76fbcd5d1
go: found github.com/xiang90/probing in github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2
go: found golang.org/x/sys/unix in golang.org/x/sys v0.0.0-20200515095857-1151b9dac4a9
go: found github.com/google/btree in github.com/google/btree v1.0.0
go: found github.com/golang/groupcache/lru in github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e
go: found go.etcd.io/bbolt in go.etcd.io/bbolt v1.3.4
go: found github.com/prometheus/client_model/go in github.com/prometheus/client_model v0.2.0
go: finding module for package github.com/gorilla/websocket
go: found github.com/gorilla/websocket in github.com/gorilla/websocket v1.4.2

haima@haima-PC:/media/haima/34E401CC64DD0E28/site/go/src/haimait/learn/etcd/etcd_put$ go run main.go
connect to etcd success
/logagent/collect_config:[{"path":"c:/tmp/nginx.log","topic":"web.log"},{"path":"d:/xxx/redis.log","topic":"redis.log"},{"path":"d:/xxx/mysql.log","topic":"mysql.log"}]

如果还是解决不了，请参考下面连接

https://my.oschina.net/u/2321997/blog/4258724/print

watch

非常重要
新建 etcd_watch/main.go

package main

import (
	"context"
	"fmt"
	"go.etcd.io/etcd/clientv3"
	"time"
)
// etcd watch


func main() {
	cli, err := clientv3.New(clientv3.Config{
		Endpoints: []string{"127.0.0.1:2379"},
		DialTimeout: 5 * time.Second,
	})
	if err != nil {
		// handle error!
		fmt.Printf("connect to etcd failed, err:%v\\n", err)
		return
	}
	fmt.Println("connect to etcd success")

	defer cli.Close()
	// watch
	// 派一个哨兵 一直监视着 /logagent/collect_config 这个key的变化（新增、修改、删除）
	ch := cli.Watch(context.Background(), "/logagent/collect_config")
	// 从通道尝试取值(监视的信息)
	for wresp := range ch{
		for _, evt := range wresp.Events{
			fmt.Printf("Type:%v key:%v value:%v\\n", evt.Type, string(evt.Kv.Key), string(evt.Kv.Value))
		}
	}
}

运行代码

haima@haima-PC:/media/haima/34E401CC64DD0E28/site/go/src/haimait/learn/etcd/etcd_watch$ go run main.go
connect to etcd success
Type:PUT key:/logagent/collect_config value:[{"path":"c:/tmp/nginx1111.log","topic":"web.log"},{"path":"d:/xxx/redis.log","topic":"redis.log"},{"path":"d:/xxx/mysql.log","topic":"mysql.log"}]

可以看到，当key /logagent/collect_config 的值变化时，会自动监听到

我的代码目录

.
├── etcd_put
│   ├── go.mod
│   ├── go.sum
│   └── main.go
└── etcd_watch
├── go.mod
├── go.sum
└── main.go

使用etcd优化日志收集项目

本周任务

  1. Raft协议

    1. 选举
    2. 日志复制机制
    3. 异常处理（脑裂）
    4. zookeeper的zad协议的区别

以上是关于Golang ETCD包的安装使用的主要内容，如果未能解决你的问题，请参考以下文章

golang基于etcd做分布式任务锁

etcd 租约Watch功能分布式锁的golang实践

浅入深出ETCD之集群部署与golang客户端使用

golang对etcd的简单操作

golang goroutine例子[golang并发代码片段]

手撸golang etcd raft协议之8