区块链HyperLedger Besu EthSigner集群服务
Posted kida_yuan
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了区块链HyperLedger Besu EthSigner集群服务相关的知识，希望对你有一定的参考价值。
说完了Besu客户端，Tessera隐私交易，是时候也说一下“三剑客”中的最后一位EthSigner。关于EthSigner在《【区块链】HyperLedger Besu Vault密钥服务》一文中稍微有提及过，它主要用作私有链（或联盟链）的区块签署使用的。
之前都没有跟大家详细说一下区块链组件之间是怎么交互的。为了方便说明，我引用官网上用一个隐私交易的交易过程进行说明，如下图：
从Dapp触发一个隐私交易请求（调用eea_sendTransaction接口）给EthSigner，EthSigner会根据Dapp传送的密钥信息找到链上用户签署密钥。接下来EthSigner会通过调用eea_sendRawTransaction接口将交易数据提交Besu上链，Besu会先判断这个交易是普通交易还是隐私交易，若是隐私交易的情况下会转发给Tessera进行隐私交易存证（对双方交易内容包括账号，数据进行隐私处理）后再回传给Besu进行上链。与此同时EthSigner生成的密钥会对上链区块进行数字封存，至此完成了隐私交易的数据上链。至于节点之间同步的我就不再多说了，看过前几篇文章的同学应该都清楚了。
在这里再补充一点信息，如下图：
如上图所示，EthSigner并不是必须的区块链组件。若Dapp足够优秀可以直接取代EthSigner，只不过目前“三剑客”组合仍然是官方推荐的实施方案而已。
说了这么多，下面我们就来看一下关于EthSigner的部署吧。由于我已经废弃了Vault方案，因此这里将直接采用本地密钥存储的方案实行。本地签署密钥需要用到nodejs生成，因此要先安装npm和cnpm（这是因为签署密钥需要通过web3.js生成）。
# 安装npm
apt-get install npm
# 安装cnpm
npm config set registry https://registry.npm.taobao.org
npm install -g cnpm --registry=https://registry.npm.taobao.org
在安装cnpm后执行cnpm install web3的时候有机会会报错：
root@node203:~# cnpm install web3
internal/modules/cjs/loader.js:638
throw err;
^
Error: Cannot find module 'fs/promises'
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
at Function.Module._load (internal/modules/cjs/loader.js:562:25)
at Module.require (internal/modules/cjs/loader.js:692:17)
at require (internal/modules/cjs/helpers.js:25:18)
at Object.<anonymous> (/usr/local/lib/node_modules/cnpm/node_modules/npminstall/bin/install.js:10:12)
at Module._compile (internal/modules/cjs/loader.js:778:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
at Module.load (internal/modules/cjs/loader.js:653:32)
at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
at Function.Module._load (internal/modules/cjs/loader.js:585:3)
这是因为安装的cnpm版本过高引起的，这个时候需要手动降级。先删除原来的版本
root@node203:~# npm uninstall -g cnpm
removed 550 packages in 2.009s
然后重新安装一个低版本即可（我这里安装7.1.0版本）
root@node203:~# npm install cnpm@7.1.0 -g
npm WARN deprecated uuid@3.4.0: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details.
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated har-validator@5.1.5: this library is no longer supported
/usr/local/bin/cnpm -> /usr/local/lib/node_modules/cnpm/bin/cnpm
npm WARN notsup Unsupported engine for open@8.4.0: wanted: "node":">=12" (current: "node":"10.19.0","npm":"6.14.4")
npm WARN notsup Not compatible with your version of node/npm: open@8.4.0
+ cnpm@7.1.0
added 845 packages from 995 contributors in 17.895s
之后通过cnpm install web3即可完成安装
root@node203:/home/yzh/Documents/blockchain/ethsigner/shell/nodejs# cnpm install web3
✔ Installed 1 packages
✔ Linked 340 latest versions
[1/8] scripts.postinstall web3@1.7.4 › web3-shh@1.7.4 run "echo \\"WARNING: the web3-shh api will be deprecated in the next version\\"", root: "/home/yzh/Documents/blockchain/ethsigner/shell/nodejs/node_modules/_web3-shh@1.7.4@web3-shh"
WARNING: the web3-shh api will be deprecated in the next version
[1/8] scripts.postinstall web3@1.7.4 › web3-shh@1.7.4 finished in 7ms
[2/8] scripts.install web3@1.7.4 › web3-core@1.7.4 › web3-core-requestmanager@1.7.4 › web3-providers-ws@1.7.4 › websocket@1.0.34 › bufferutil@^4.0.1 run "node-gyp-build", root: "/home/yzh/Documents/blockchain/ethsigner/shell/nodejs/node_modules/_bufferutil@4.0.6@bufferutil"
[2/8] scripts.install web3@1.7.4 › web3-core@1.7.4 › web3-core-requestmanager@1.7.4 › web3-providers-ws@1.7.4 › websocket@1.0.34 › bufferutil@^4.0.1 finished in 744ms
[3/8] scripts.install web3@1.7.4 › web3-core@1.7.4 › web3-core-requestmanager@1.7.4 › web3-providers-ws@1.7.4 › websocket@1.0.34 › utf-8-validate@^5.0.2 run "node-gyp-build", root: "/home/yzh/Documents/blockchain/ethsigner/shell/nodejs/node_modules/_utf-8-validate@5.0.9@utf-8-validate"
[3/8] scripts.install web3@1.7.4 › web3-core@1.7.4 › web3-core-requestmanager@1.7.4 › web3-providers-ws@1.7.4 › websocket@1.0.34 › utf-8-validate@^5.0.2 finished in 741ms
[4/8] scripts.postinstall web3@1.7.4 › web3-bzz@1.7.4 run "echo \\"WARNING: the web3-bzz api will be deprecated in the next version\\"", root: "/home/yzh/Documents/blockchain/ethsigner/shell/nodejs/node_modules/_web3-bzz@1.7.4@web3-bzz"
WARNING: the web3-bzz api will be deprecated in the next version
[4/8] scripts.postinstall web3@1.7.4 › web3-bzz@1.7.4 finished in 5ms
[5/8] scripts.install web3@1.7.4 › web3-utils@1.7.4 › ethereumjs-util@7.1.5 › ethereum-cryptography@0.1.3 › keccak@^3.0.0 run "node-gyp-build || exit 0", root: "/home/yzh/Documents/blockchain/ethsigner/shell/nodejs/node_modules/_keccak@3.0.2@keccak"
[5/8] scripts.install web3@1.7.4 › web3-utils@1.7.4 › ethereumjs-util@7.1.5 › ethereum-cryptography@0.1.3 › keccak@^3.0.0 finished in 750ms
[6/8] scripts.install web3@1.7.4 › web3-utils@1.7.4 › ethereumjs-util@7.1.5 › ethereum-cryptography@0.1.3 › secp256k1@^4.0.1 run "node-gyp-build || exit 0", root: "/home/yzh/Documents/blockchain/ethsigner/shell/nodejs/node_modules/_secp256k1@4.0.3@secp256k1"
[6/8] scripts.install web3@1.7.4 › web3-utils@1.7.4 › ethereumjs-util@7.1.5 › ethereum-cryptography@0.1.3 › secp256k1@^4.0.1 finished in 740ms
[7/8] scripts.postinstall web3@1.7.4 › web3-core@1.7.4 › web3-core-requestmanager@1.7.4 › web3-providers-ws@1.7.4 › websocket@1.0.34 › es5-ext@^0.10.50 run " node -e \\"tryrequire('./_postinstall')catch(e)\\" || exit 0", root: "/home/yzh/Documents/blockchain/ethsigner/shell/nodejs/node_modules/_es5-ext@0.10.61@es5-ext"
[7/8] scripts.postinstall web3@1.7.4 › web3-core@1.7.4 › web3-core-requestmanager@1.7.4 › web3-providers-ws@1.7.4 › websocket@1.0.34 › es5-ext@^0.10.50 finished in 376ms
[8/8] scripts.postinstall web3@latest run "echo \\"WARNING: the web3-shh and web3-bzz api will be deprecated in the next version\\"", root: "/home/yzh/Documents/blockchain/ethsigner/shell/nodejs/node_modules/_web3@1.7.4@web3"
WARNING: the web3-shh and web3-bzz api will be deprecated in the next version
[8/8] scripts.postinstall web3@latest finished in 5ms
✔ Run 8 scripts
anti semver web3@1.7.4 › web3-core@1.7.4 › @types/bn.js@5.1.0 › @types/node@* delcares @types/node@*(resolved as 18.0.3) but using ancestor(web3-core)'s dependency @types/node@^12.12.6(resolved as 12.20.55)
deprecate web3@1.7.4 › web3-bzz@1.7.4 › swarm-js@0.1.40 › mkdirp-promise@^5.0.1 This package is broken and no longer maintained. 'mkdirp' itself supports promises now, please switch to that.
deprecate web3@1.7.4 › web3-eth@1.7.4 › web3-eth-accounts@1.7.4 › uuid@3.3.2 Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details.
deprecate web3@1.7.4 › web3-eth@1.7.4 › web3-eth-ens@1.7.4 › content-hash@2.5.2 › multicodec@^0.5.5 This module has been superseded by the multiformats module
deprecate web3@1.7.4 › web3-bzz@1.7.4 › swarm-js@0.1.40 › eth-lib@0.1.29 › servify@0.1.12 › request@^2.79.0 request has been deprecated, see https://github.com/request/request/issues/3142
deprecate web3@1.7.4 › web3-eth@1.7.4 › web3-eth-ens@1.7.4 › content-hash@2.5.2 › multihashes@0.4.21 › multibase@^0.7.0 This module has been superseded by the multiformats module
deprecate web3@1.7.4 › web3-bzz@1.7.4 › swarm-js@0.1.40 › eth-lib@0.1.29 › servify@0.1.12 › request@2.88.2 › har-validator@~5.1.3 this library is no longer supported
deprecate web3@1.7.4 › web3-bzz@1.7.4 › swarm-js@0.1.40 › eth-lib@0.1.29 › servify@0.1.12 › request@2.88.2 › uuid@^3.3.2 Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details.
deprecate web3@1.7.4 › web3-eth@1.7.4 › web3-eth-ens@1.7.4 › content-hash@2.5.2 › cids@^0.7.1 This module has been superseded by the multiformats module
deprecate web3@1.7.4 › web3-eth@1.7.4 › web3-eth-ens@1.7.4 › content-hash@2.5.2 › cids@0.7.5 › multicodec@^1.0.0 This module has been superseded by the multiformats module
deprecate web3@1.7.4 › web3-eth@1.7.4 › web3-eth-ens@1.7.4 › content-hash@2.5.2 › cids@0.7.5 › multibase@~0.6.0 This module has been superseded by the multiformats module
✔ All packages installed (362 packages installed from npm registry, used 6s(network 3s), speed 0B/s, json 0(0B), tarball 0B, manifests cache hit 340, etag hit 0 / miss 0)
再之后就就要安装jq插件。安装jq插件的目的是解析http请求返回使用的，譬如：启动区块链服务后我需要通过curl访问restful接口并解析返回json字符串，这个时候通过jq插件就能够轻松解析出结果，就不需要一层一层去反解返回的json字符串了（因为自己是开发出身，所以不经意会以开发的思路解决问题，如果有更好的解决方式jq插件其实可以不安装）。
root@node203:/home/yzh/Documents/blockchain/ethsigner/shell# apt-get install jq
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
libjq1 libonig5
The following NEW packages will be installed:
jq libjq1 libonig5
0 upgraded, 3 newly installed, 0 to remove and 153 not upgraded.
Need to get 313 kB of archives.
After this operation, 1,062 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://mirrors.ustc.edu.cn/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB]
Get:2 http://mirrors.ustc.edu.cn/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB]
Get:3 http://mirrors.ustc.edu.cn/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB]
Fetched 313 kB in 0s (812 kB/s)
Selecting previously unselected package libonig5:amd64.
(Reading database ... 328030 files and directories currently installed.)
Preparing to unpack .../libonig5_6.9.4-1_amd64.deb ...
Unpacking libonig5:amd64 (6.9.4-1) ...
Selecting previously unselected package libjq1:amd64.
Preparing to unpack .../libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ...
Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ...
Selecting previously unselected package jq.
Preparing to unpack .../jq_1.6-1ubuntu0.20.04.1_amd64.deb ...
Unpacking jq (1.6-1ubuntu0.20.04.1) ...
Setting up libonig5:amd64 (6.9.4-1) ...
Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ...
Setting up jq (1.6-1ubuntu0.20.04.1) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.7) ...
最后就要准备好一个密钥创建脚本（create_keyfile.js），如下：
var args = process.argv.splice(2);
const Web3 = require('web3');
//Web3初始化(指向JSON-RPC节点)
const web3 = new Web3(new Web3.providers.HttpProvider('http://127.0.0.1:8545'));
var V3KeyStore = web3.eth.accounts.encrypt(args[0], args[1]);
console.log(JSON.stringify(V3KeyStore));
process.exit();
至此全部前期准备都已经做好，接下来可以通过脚本创建用户了。事不宜迟我们可以先写个脚本（ethsigner_run.sh）：
# 节点安装路径
deploy_path=/home/yzh/Documents/blockchain/docker/ethsigner
# nodejs生成用户密钥文件路径
createkey_path=/home/yzh/Documents/blockchain/config/ethsigner/nodejs/create_keyfile.js
# ethsigner docker镜像名称
docker_image=consensys/ethsigner:21.10.0
# 签署用户（虚拟账号）生成个数
signer_counter=2000
# 区块链id（可以在创世块配置文件中获取）
chain_id=12345
# 自定义校验节点名称
custom_name=node202_signer
# 校验节点静态ip
ethsigner_ip=192.18.0.212
# 签署节点ip
signer_node_ip=192.18.0.202
# 签署用户（虚拟账号）存档路径
has_users=/home/yzh/Documents/blockchain/config/ethsigner/users
# 分多少个线程并发
thread_num=100
if [ ! -d $deploy_path ]; then
# 1. 判断是否存在部署目录和是否已经存在用户了
if [ ! -d $has_users ]; then
# 临时文件名称
tmpfifo="tmpfifo"
mkfifo $tmpfifo
exec 6<>$tmpfifo
rm -f $tmpfifo
for ((i = 1; i <= $thread_num; i++)); do
echo
done >&6
# 2. 若不存在则创建部署目录和用户目录
mkdir -p $deploy_path && mkdir -p $deploy_path/signer_address
mkdir -p $has_users && chmod 777 -R $has_users
# 3. 将密码信息转换成数组并通过循环读取每一个密码信息
for ((i = 1; i <= $signer_counter; i++)); do
read -u6
echo "Currently $i accounts have been generated"
# 4. 生成签署账号64位的私钥
hex_address=$(openssl rand -hex 32)
# 5. 然后通过执行createkey.js生成密钥信息
keyfile=$(node $createkey_path "0x"$hex_address $i"_BCA")
# 6. 通过解析返回的json获取到签署节点的加密密文
address_val=$(echo $keyfile | jq '.address')
address_val=$(echo $address_val | sed 's/\\"//g')
# 7. 根据上面收集的内容生成keyFile、passwordFile和配置用的toml文件
keyFilePath=$deploy_path/$address_val/keyFile
passwordFilePath=$deploy_path/$address_val/passwordFile
tomlFilePath=$deploy_path/signer_address/$address_val".toml"
# 8. 将信息按照要求写入到指定的文件里面，这里需要注意的配置文件中关于路径的写法一定要按照镜像中机器的写法不然是没有找到对应路径的
mkdir -p $deploy_path/$address_val
touch $keyFilePath && touch $passwordFilePath && touch $tomlFilePath
chmod 777 -R $deploy_path
# 9. 将密码写入到passwordFile
echo -e $i"_BCA" >>$passwordFilePath
# 10. 将生成的key内容写入keyFile
echo -e "$keyfile" >>$keyFilePath
# 11. 将权限配置文件写入toml配置
echo -e "[metadata]" >>$tomlFilePath
echo -e "createdAt = $(date "+%Y-%m-%d %H:%M:%S")" >>$tomlFilePath
echo -e "description = \\"$hex_address account configuration\\" \\n" >>$tomlFilePath
echo -e "[signing] \\ntype = \\"file-based-signer\\"" >>$tomlFilePath
echo -e "key-file = \\"/var/lib/ethsigner/$address_val/keyFile\\"" >>$tomlFilePath
echo -e "password-file = \\"/var/lib/ethsigner/$address_val/passwordFile\\"" >>$tomlFilePath
echo "" >&6
&
done
wait
exec 6>&-
echo "All accounts are generated"
# 12. 将配置信息进行备份
find $deploy_path -name "*" -exec cp -rf $has_users \\;
else
mkdir -p $deploy_path
find $has_users -name "*" -exec cp -rf $deploy_path \\;
fi
# 13. 使用docker来启动eth_signer
docker run --name $custom_name --network=besu_swarm --ip $ethsigner_ip -p 8945:8545 --mount type=bind,source=$deploy_path,target=/var/lib/ethsigner -d $docker_image --chain-id=$chain_id --downstream-http-host=$signer_node_ip --downstream-http-port=8545 --http-listen-host=0.0.0.0 multikey-signer --directory=/var/lib/ethsigner/signer_address
fi
以上是通过shell模拟多线程执行的脚本用来创建多个区块链签署密钥，脚本上带有注释应该能够看得明白的，但是脚本中关于配置文件的读取其实需要按照一定的规则进行放置，这里由于涉及到不可描述的原因这里就没有提供，并且为了使分享内容不牵扯到知识产权的纠纷问题，脚本内容我也是有做调整。所以脚本不能直接使用，这里只是给各位一个实现思路。
直接执行脚本即可生成本地密钥，如下：
root@node203:/home/yzh/Documents/blockchain/config/ethsigner/shell# ./ethsigner_run.sh
Currently 1 accounts have been generated
Currently 2 accounts have been generated
Currently 3 accounts have been generated
Currently 4 accounts have been generated
Currently 6 accounts have been generated
Currently 5 accounts have been generated
Currently 7 accounts have been generated
Currently 8 accounts have been generated
Currently 9 accounts have been generated
Currently 10 accounts have been generated
Currently 11 accounts have been generated
Currently 13 accounts have been generated
Currently 15 accounts have been generated
Currently 16 accounts have been generated
Currently 21 accounts have been generated
Currently 17 accounts have been generated
Currently 18 accounts have been generated
Currently 14 accounts have been generated
Currently 19 accounts have been generated
Currently 25 accounts have been generated
Currently 23 accounts have been generated
Currently 28 accounts have been generated
Currently 35 accounts have been generated
Currently 12 accounts have been generated
Currently 22 accounts have been generated
Currently 30 accounts have been generated
Currently 29 accounts have been generated
……
由于生成过程采用了快速分片模拟多线程的情况，因此看到的输出不是顺序输出
Currently 1995 accounts have been generated
Currently 1994 accounts have been generated
Currently 1996 accounts have been generated
Currently 1997 accounts have been generated
Currently 1998 accounts have been generated
Currently 1999 accounts have been generated
Currently 2000 accounts have been generated
All accounts are generated
5d17812000e480798ef96264a88df76681d92c3f9db8f0baef81035929a1b589
看到这样输出后证明脚本已经执行完成，这个时候可以通过docker logs -f查看服务的输出情况
root@node203:/home/yzh/Documents/blockchain/config/ethsigner/shell# docker logs -f 5d17812000e4
Setting logging level to INFO
2022-07-07 10:05:08.875+00:00 | main | INFO | SignerSubCommand | Version = ethsigner/v21.10.0/linux-x86_64/-na-openjdk64bitservervm-java-11
2022-07-07 10:05:09.438+00:00 | main | INFO | Runner | Server is up, and listening on 8545
看到这样的输出证明启动已经成功。
由于签署密钥还要提供给应用程序（Dapp）使用用于与系统账号绑定，我这边是选择将密钥同步到Redis中由应用程序直接跟Redis对接获取密钥信息，于是我又写了一个shell脚本用于将密钥同步到Redis，如下：
scan_path=/home/yzh/Documents/blockchain/config/ethsigner/users
# redis微服务ip地址和端口
redis_mic_ip=192.168.100.138
redis_mic_port=6379
# 当前节点编码
node_code=YZH
declare -i count=0
# 1. 初始化setup变量为后面拼接字符串做准备
setup=""
# 2. 遍历指定的文件目录获取到所有后缀为toml文件
for file in $(ls $scan_path)
do
if [ "$file##*." = "toml" ]; then
param="$file%%.*"
# 3. 开始拼接字符串
setup="$setup $param"
# 4. 由于redis对于传入参数的长度有限制，因此需要分批进行这里面将按照设定进行计数
count=$(($count + 1))
fi
# 5. 每当循环了100次的时候就保存一次到redis中
if (( $count == 100 )); then
docker run --rm --name redis-cli -it goodsmileduck/redis-cli redis-cli -h $redis_mic_ip -p $redis_mic_port RPUSH ETH_ACCOUNTS"_"$node_code $setup
count=0
setup=""
fi
done
这个脚本其实说白了就是将磁盘中的文件读取然后通过redis-cli镜像写入到目标redis里面而已。效果如下：
root@node203:/home/yzh/Documents/blockchain/config/ethsigner/shell# ./setup_account_to_redis.sh
(integer) 100
(integer) 200
(integer) 300
(integer) 400
(integer) 500
(integer) 600
(integer) 700
(integer) 800
(integer) 900
(integer) 1000
(integer) 1100
(integer) 1200
(integer) 1300
(integer) 1400
(integer) 1500
(integer) 1600
(integer) 1700
(integer) 1800
(integer) 1900
(integer) 2000
执行脚本后就能够在redis客户端中看到数据存储的情况，如下图：
至此，Besu区块链的基础组件部署就演示完毕了，接下来就需要看看实际效果了。
以上是关于区块链HyperLedger Besu EthSigner集群服务的主要内容，如果未能解决你的问题，请参考以下文章