swift对象存储环境搭建
Posted 万笑佛
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了swift对象存储环境搭建相关的知识，希望对你有一定的参考价值。
关于swift对象存储，国内的资料太少了，网上能找到的可用的寥寥无几，大都是只言片语，官网个人觉得写得也不是特别好，不是很好理解，本人不才
打算把这些年使用swift对象存储的经验慢慢整理分享给大家，本篇先从基础环境搭建开始，最后会写到java对swift对象存储的使用。所有分享出来的都是
我亲身实验过的，有问题欢迎大家留言。关于swift对象存储是什么网上有写的比较好的文章我就不再复述了，如下开始搭建一个实验环境。
三台服务器的ip
192.168.0.101
192.168.0.102
192.168.0.103
注：三台服务器每台服务器使用一块硬盘，搭建一个对象存储集群。如果是生产环境可能需要更多的服务器和磁盘
一、配置yum源
[root@kangvcar ~]# cd /etc/yum.repos.d/
[root@kangvcar ~]# cat /etc/redhat-release //查看系统的版本
CentOS Linux release 7.2.1511 (Core)
[root@kangvcar ~]# wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
--2017-06-20 06:43:08-- http://mirrors.aliyun.com/repo/Centos-7.repo
Resolving mirrors.aliyun.com (mirrors.aliyun.com)... 112.124.140.210, 115.28.122.210
Connecting to mirrors.aliyun.com (mirrors.aliyun.com)|112.124.140.210|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2573 (2.5K) [application/octet-stream]
Saving to: ‘/etc/yum.repos.d/CentOS-Base.repo’
100%[=======================================================================================================>] 2,573 --.-K/s in 0s
2017-06-20 06:43:08 (118 MB/s) - ‘/etc/yum.repos.d/CentOS-Base.repo’ saved [2573/2573]
[root@kangvcar ~]# yum clean all
Loaded plugins: fastestmirror
Cleaning repos: base extras updates
Cleaning up everything
Cleaning up list of fastest mirrors
[root@kangvcar ~]# yum makecache //把yum源缓存到本地，加快软件的搜索好安装速度
[root@kangvcar ~]# yum list //总共列出了9954个包
二、安装各种依赖
sudo yum install curl gcc memcached rsync sqlite xfsprogs git-core \\
libffi-devel xinetd liberasurecode-devel \\
python-setuptools \\
python-coverage python-devel python-nose \\
pyxattr python-eventlet \\
python-greenlet python-paste-deploy \\
python-netifaces python-pip python-dns \\
python-mock
三、查看磁盘类型是不是xfs 是就不用管，不是需要重新分区格式化成xfs
[root@node01 opt]# parted -l
--大磁盘分区请参照如下链接地址
http://www.blogjava.net/haha1903/archive/2011/12/21/366942.html
四、下载swift-client、swift
git clone https://github.com/openstack/python-swiftclient.git
git checkout 2.4.0
git clone https://github.com/openstack/swift.git
git checkout 2.4.0
五、创建目录
mkdir -p /opt/soft
mkdir -p /opt/swift-disk/sdb1
将上一步下载的两个文件放到/opt/soft目录下
六、安装
cd /opt/soft/python-swiftclient
[root@node01 opt]# python setup.py develop;
cd /opt/soft/swift
[root@node01 swift]# pip install -r requirements.txt;
[root@node01 swift]# pip install --upgrade pip
[root@node01 swift]# sudo python setup.py develop;
七、配置 /etc/rc.local 当前也要执行
sudo mkdir -p /var/run/swift
sudo mkdir -p /var/cache/swift
八、配置 rsync
vi /etc/rsyncd.conf
# /etc/rsyncd: configuration file for rsync daemon mode
uid = root
gid = root
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
address = 0.0.0.0
[account]
max connections = 25
path = /opt/swift-disk/
read only = false
lock file = /var/lock/account.lock
[container]
max connections = 25
path = /opt/swift-disk/
read only = false
lock file = /var/lock/container.lock
[object]
max connections = 25
path = /opt/swift-disk/
read only = false
lock file = /var/lock/object.lock
九、启动 rsync 并设置开机启动
systemctl start rsyncd.service
systemctl enable rsyncd.service
十、启动 memcached 并设置开机启动
systemctl start memcached.service
systemctl enable memcached.service
十一、配置swift相关的配置文件
(1)创建 swift 配置文件 /etc/swift/swift.conf
vi /etc/swift/swift.conf
[swift-hash]
#echo `od -t x8 -N 8 -A n </dev/random` 生成随机数填写
swift_hash_path_suffix= EvjucvFN5aX1oAdjlQiw5+HV0JUqSSWwzoiem+fLAZE=
(2)配置 account 服务 /etc/swift/account-server.conf
vi /etc/swift/account-server.conf
[DEFAULT]
devices = /opt/swift-disk
mount_check = false
bind_ip = 0.0.0.0
bind_port = 6002
workers = 4
user = root
log_facility = LOG_LOCAL4
[pipeline:main]
pipeline = account-server
[app:account-server]
use = egg:swift#account
[account-replicator]
[account-auditor]
[account-reaper]
(3)配置 container 服务 /etc/swift/container-server.conf
vi /etc/swift/container-server.conf
[DEFAULT]
devices = /opt/swift-disk
mount_check = false
bind_ip = 0.0.0.0
bind_port = 6001
workers = 4
user = root
log_facility = LOG_LOCAL3
[pipeline:main]
pipeline = container-server
[app:container-server]
use = egg:swift#container
[container-replicator]
[container-updater]
[container-auditor]
[container-sync]
(4)配置 object 服务 /etc/swift/object-server.conf
vi /etc/swift/object-server.conf
[DEFAULT]
devices = /opt/swift-disk
mount_check = false
bind_ip = 0.0.0.0
bind_port = 6000
workers = 4
user = root
log_facility = LOG_LOCAL2
[pipeline:main]
pipeline = object-server
[app:object-server]
use = egg:swift#object
[object-replicator]
[object-updater]
[object-auditor]
(5)配置 proxy 服务 /etc/swift/proxy-server.conf
vi /etc/swift/proxy-server.conf
[DEFAULT]
bind_port = 8080
user = root
workers = 8
log_facility = LOG_LOCAL1
[pipeline:main]
pipeline = healthcheck cache tempauth proxy-logging proxy-server
[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true
account_autocreate = true
[filter:tempauth]
use = egg:swift#tempauth
user_admin_admin = admin .admin .reseller_admin
user_test_tester = testing .admin
user_test2_tester2 = testing2 .admin
user_test_tester3 = testing3
reseller_prefix = AUTH
token_life = 86400
# account和token的命名前缀，注意此处不可以加"_"。
# 例如X-Storage-Ur可能l为http://127.0.0.1:8080/v1/AUTH_test
# 例如X-Auth-Token为AUTH_tk440e9bd9a9cb46d6be07a5b6a585f7d1# token的有效期，单位：秒。
[filter:healthcheck]
use = egg:swift#healthcheck
[filter:cache]
use = egg:swift#memcache
# 这里可以是多个memcache server，proxy会自动当作一个集群来使用# 例如 memcache_servers = 192.168.2.129:11211,192.168.2.130:11211,192.168.2.131
memcache_servers = 192.168.0.101:11211,192.168.0.102:11211,192.168.0.103:11211
[filter:proxy-logging]
use = egg:swift#proxy_logging
十二、创建 ring 文件
cd /etc/swift
swift-ring-builder account.builder create 18 3 1
swift-ring-builder container.builder create 18 3 1
swift-ring-builder object.builder create 18 3 1
注：需要事先创建好3个 ring，18 表示 partition 数为2的18次方，3表示 replication 数为3，1 表示分区数据的最短迁移间隔时间为1小时。（官网说明里如果移除设备的话不在这个限制内）
十三、向ring 中加入设备 后面的100是权重建议使用磁盘的实际大小，如果磁盘上100G权重就设置100
swift-ring-builder account.builder add z1-192.168.0.101:6002/sdb1 100
swift-ring-builder container.builder add z1-192.168.0.101:6001/sdb1 100
swift-ring-builder object.builder add z1-192.168.0.101:6000/sdb1 100
swift-ring-builder account.builder add z2-192.168.0.102:6002/sdb1 100
swift-ring-builder container.builder add z2-192.168.0.102:6001/sdb1 100
swift-ring-builder object.builder add z2-192.168.0.102:6000/sdb1 100
swift-ring-builder account.builder add z3-192.168.0.103:6002/sdb1 100
swift-ring-builder container.builder add z3-192.168.0.103:6001/sdb1 100
swift-ring-builder object.builder add z3-192.168.0.103:6000/sdb1 100
注：z1 和 z2 表示 zone1 和 zone2（zone 这个概念是虚拟的，可以将一个 device 划定到一个 zone，在分配 partition 的时候会考虑到这个因素，尽量划分到不同的 zone 中）。
sdb1 为 swift 所使用的存储空间。
100 代表设备的权重，也是在分配 partition 的时候会考虑的因素。
十四、确认ring 的内容是否正确:
swift-ring-builder account.builder
swift-ring-builder container.builder
swift-ring-builder object.builder
十五、平衡(Rebalance) ring
swift-ring-builder account.builder rebalance
swift-ring-builder container.builder rebalance
swift-ring-builder object.builder rebalance
一键启动 swift 的所有服务。
#!/bin/bash
swift-init proxy start
swift-init account-server start
swift-init account-replicator start
swift-init account-auditor start
swift-init container-server start
swift-init container-replicator start
swift-init container-updater start
swift-init container-auditor start
swift-init object-server start
swift-init object-replicator start
swift-init object-updater start
swift-init object-auditor start
关闭服务
#!/bin/bash
swift-init proxy stop
swift-init account-server stop
swift-init account-replicator stop
swift-init account-auditor stop
swift-init container-server stop
swift-init container-replicator stop
swift-init container-updater stop
swift-init container-auditor stop
swift-init object-server stop
swift-init object-replicator stop
swift-init object-updater stop
swift-init object-auditor stop
十六、常用语句
1、直接使用 swift 客户端程序进行操作(帮助文档[root@node01]# swift --help)
（1）查看账户信息（账户是/etc/swift/proxy-server.conf里配置的）
swift -A http://127.0.0.1:8080/auth/v1.0 -U admin:admin -K admin stat
（2）创建 container
swift -A http://127.0.0.1:8080/auth/v1.0 -U admin:admin -K admin post container1
（3）查看 test 用户的 container 列表
swift -A http://127.0.0.1:8080/auth/v1.0 -U admin:admin -K admin list
（4）查看容器中的内容
swift -A http://127.0.0.1:8080/auth/v1.0 -U admin:admin -K admin list container1
（5）上传Object（文件），上传 3.txt 文件到 container1 容器中
swift -A http://127.0.0.1:8080/auth/v1.0 -U admin:admin -K admin upload container1 3.txt
上传其他路径的文件
swift -A http://127.0.0.1:8080/auth/v1.0 -U admin:admin -K admin upload container1 /opt/4.txt --object-name 4.txt
（6）下载Object（文件）
swift -A http://127.0.0.1:8080/auth/v1.0 -U admin:admin -K admin download container1 3.txt
将文件下载到其他路径
swift -A http://127.0.0.1:8080/auth/v1.0 -U admin:admin -K admin download container1 3.txt --output /opt/3.txt
下载该用户下所有文件
[root@node01 swift-disk]# swift -A http://127.0.0.1:8080/auth/v1.0 -U admin:admin -K admin download --all
（7）删除文件
swift -A http://127.0.0.1:8080/auth/v1.0 -U admin:admin -K admin delete container1 3.txt
（8）查看集群的基本信息（List cluster capabilities）
swift -A http://127.0.0.1:8080/auth/v1.0 -U admin:admin -K admin capabilities
(9)查看文件存储的物理位置（查看其它用法 swift-get-nodes --help）
查看账户信息在哪里：swift-get-nodes -a /etc/swift/account.ring.gz AUTH_admin
查看容器信息在哪里：swift-get-nodes -a /etc/swift/container.ring.gz AUTH_admin container1
查看文件信息在哪里：swift-get-nodes -a /etc/swift/object.ring.gz AUTH_admin container1 1.txt
2、RESTful API（账户是/etc/swift/proxy-server.conf里配置的）
所有的操作都需要先获取一个 auth-token，之后的所有操作都需要在 header 中附加上 X-Auth-Token 字段的信息进行权限认证。有一定时效性，过期后需要再次获取。
（1）获取 X-Storage-Url 和 X-Auth-Token
curl http://127.0.0.1:8080/auth/v1.0 -v -H \'X-Storage-User: admin:admin\' -H \'X-Storage-Pass: admin\'
或：curl http://192.168.0.102:8080/auth/v1.0 -v -H \'X-Storage-User: admin:admin\' -H \'X-Storage-Pass: admin\'
* About to connect() to 127.0.0.1 port 8080 (#0)
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET /auth/v1.0 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 127.0.0.1:8080
> Accept: */*
> X-Storage-User: admin:admin
> X-Storage-Pass: admin
< HTTP/1.1 200 OK
< X-Storage-Url: http://127.0.0.1:8080/v1/AUTH_admin
< X-Auth-Token: AUTH_tkf44c48c200e34f14850f6a8e3bc63f36
< Content-Type: text/html; charset=UTF-8
< X-Storage-Token: AUTH_tkf44c48c200e34f14850f6a8e3bc63f36
< X-Trans-Id: tx703a6ebfbfca46a49f605-005c1878e0
< Content-Length: 0
< Date: Tue, 18 Dec 2018 04:34:40 GMT
<
* Connection #0 to host 127.0.0.1 left intact
（2）查看账户信息
curl http://127.0.0.1:8080/v1/AUTH_admin -v -H \'X-Auth-Token: AUTH_tkf44c48c200e34f14850f6a8e3bc63f36\'
* About to connect() to 127.0.0.1 port 8080 (#0)
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET /v1/AUTH_admin HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 127.0.0.1:8080
> Accept: */*
> X-Auth-Token: AUTH_tkf44c48c200e34f14850f6a8e3bc63f36
>
< HTTP/1.1 200 OK
< Content-Length: 11
< X-Account-Object-Count: 2
< X-Account-Storage-Policy-Policy-0-Bytes-Used: 11
< X-Account-Storage-Policy-Policy-0-Container-Count: 1
< X-Timestamp: 1545104875.82331
< X-Account-Storage-Policy-Policy-0-Object-Count: 2
< X-Account-Bytes-Used: 11
< X-Account-Container-Count: 1
< Content-Type: text/plain; charset=utf-8
< Accept-Ranges: bytes
< X-Trans-Id: txea708b3ecf3047099ea6f-005c187945
< Date: Tue, 18 Dec 2018 04:36:21 GMT
<
container1
（3） 创建 container（container2为创建的容器名称）
curl http://127.0.0.1:8080/v1/AUTH_admin/container2 -X PUT -H "X-Auth_Token: AUTH_tkf44c48c200e34f14850f6a8e3bc63f36"
(4)各种操作总结(参考网页https://blog.csdn.net/ztejiagn/article/details/8905782)
例如：
<1>获取AUTH_admin账户下的容器列表
curl http://127.0.0.1:8080/v1/AUTH_admin/ -X GET -H "X-Auth_Token: AUTH_tkf44c48c200e34f14850f6a8e3bc63f36"
<2>获取AUTH_admin账户下的容器container1的对象列表
curl http://127.0.0.1:8080/v1/AUTH_admin/container1 -X GET -H "X-Auth_Token: AUTH_tkf44c48c200e34f14850f6a8e3bc63f36"
<3>创建、更新或拷贝对象
curl http://127.0.0.1:8080/v1/AUTH_admin/container1/1.txt -X PUT -T 1.txt -H "X-Auth_Token: AUTH_tkf44c48c200e34f14850f6a8e3bc63f36"
其他路径下的文件
curl http://127.0.0.1:8080/v1/AUTH_admin/container1/2.txt -X PUT -T /opt/node/2.txt -H "X-Auth_Token: AUTH_tkf44c48c200e34f14850f6a8e3bc63f36"
<4>获取对象内容和元数据
curl http://127.0.0.1:8080/v1/AUTH_admin/container1/2.txt -X GET -H "X-Auth_Token: AUTH_tkf44c48c200e34f14850f6a8e3bc63f36"
<5>删除对象
curl http://127.0.0.1:8080/v1/AUTH_admin/container1/2.txt -X DELETE -H "X-Auth_Token: AUTH_tkf44c48c200e34f14850f6a8e3bc63f36"
其他有用
CentOS7使用firewalld打开关闭防火墙与端口
1、firewalld的基本使用
启动： systemctl start firewalld
关闭： systemctl stop firewalld
查看状态： systemctl status firewalld
开机禁用 ： systemctl disable firewalld
开机启用 ： systemctl enable firewalld
2、日志在
swift日志
/var/log/messages
同步日志
/var/log/rsyncd.log
以上是关于swift对象存储环境搭建的主要内容，如果未能解决你的问题，请参考以下文章