010 Ceph RGW对象存储

Posted zyxnhr

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了010 Ceph RGW对象存储相关的知识，希望对你有一定的参考价值。

一、对象存储

1.1 介绍

通过对象存储，将数据存储为对象，每个对象除了包含数据，还包含数据自身的元数据

对象通过Object ID来检索，无法通过普通文件系统操作来直接访问对象，只能通过API来访问，或者第三方客户端（实际上也是对API的封装）

对象存储中的对象不整理到目录树中，而是存储在扁平的命名空间中，Amazon S3将这个扁平命名空间称为bucket。而swift则将其称为容器

无论是bucket还是容器，都不能嵌套

bucket需要被授权才能访问到，一个帐户可以对多个bucket授权，而权限可以不同

对象存储的优点：易扩展、快速检索

1.2 Rados网关介绍

RADOS网关也称为Ceph对象网关、RADOSGW、RGW，是一种服务，使客户端能够利用标准对象存储API来访问Ceph集群。它支持S3和Swift API

rgw运行于librados之上，事实上就是一个称之为Civetweb的web服务器来响应api请求

客户端使用标准api与rgw通信，而rgw则使用librados与ceph集群通信

rgw客户端通过s3或者swift api使用rgw用户进行身份验证。然后rgw网关代表用户利用cephx与ceph存储进行身份验证

二、RADOS网关部署

2.1 配置radosgw

[[email protected] ~]#  ceph auth get-or-create client.rgw.ceph5  mon ‘allow rwx‘ osd ‘allow rwx‘ -o /etc/ceph/backup.client.rgw.ceph5.keyring --cluster backup

[[email protected] ~]# vim /etc/ceph/backup.conf

fsid = 51dda18c-7545-4edb-8ba9-27330ead81a7
mon_initial_members = ceph5
mon_host = 172.25.250.14

auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

public_network = 172.25.250.0/24
cluster_network = 172.25.250.0/24

[mgr]
mgr modules = dashboard

[client.rgw.ceph5]
host = ceph5
keyring = /etc/ceph/backup.client.rgw.ceph5.keyring
rgw_frontends = civetweb port=80

[[email protected] ~]# systemctl restart [email protected]

[[email protected] ~]# ps -ef|grep rados

root     13828     1  0 18:07 ?        00:00:00 /usr/bin/radosgw -f --cluster backup --name client.rgw.ceph5 --setuser ceph --setgroup ceph

[[email protected] ~]# netstat -ntlp|grep 80

tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      13828/radosgw

[[email protected] ~]# ceph osd pool ls

技术图片

[[email protected] ~]# ceph -s

技术图片

[[email protected] ~]# ceph osd pool application enable rbd rbd

技术图片

[[email protected] ~]# ceph osd pool application enable rbdmirror rbd

技术图片

[[email protected] ~]# ceph -s

技术图片

[[email protected] ~]#  cat /usr/lib/systemd/system/[email protected]

[Unit]
Description=Ceph rados gateway
After=network-online.target local-fs.target time-sync.target
Wants=network-online.target local-fs.target time-sync.target
PartOf=ceph-radosgw.target

[Service]
LimitNOFILE=1048576
LimitNPROC=1048576
EnvironmentFile=-/etc/sysconfig/ceph
Environment=CLUSTER=ceph
ExecStart=/usr/bin/radosgw -f --cluster ${CLUSTER} --name client.%i --setuser ceph --setgroup ceph
PrivateDevices=yes
ProtectHome=true
ProtectSystem=full
PrivateTmp=true
TasksMax=infinity
Restart=on-failure
StartLimitInterval=30s
StartLimitBurst=5

[Install]
WantedBy=ceph-radosgw.target

 2.2 定义监听的线程数

[[email protected] ~]# vim /etc/ceph/backup.conf

[client.rgw.ceph5]
host = ceph5
keyring = /etc/ceph/backup.client.rgw.ceph5.keyring
rgw_frontends = civetweb port=80 num_threads=100
log = /var/log/ceph/$cluster.$name.log

[[email protected] ~]# systemctl restart [email protected]

[[email protected] ~]# ps -ef|grep rados

ceph 15553 1 1 20:26 ? 00:00:00 /usr/bin/radosgw -f --cluster backup --name client.rgw.ceph5 --setuser ceph --setgroup ceph

2.3 访问rados网关

[[email protected] ~]# curl http://ceph5

<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>

三、S3对象存储

3.1 S3简介

S3由Amazon于2006年推出，全称为Simple Storage Service

S3定义了对象存储，是对象存储事实上的标准，从某种意义上说，S3就是对象存储，对象存储就是S3

S3是对象存储市场的霸主，后续的对象存储都是对S3的模仿

3.2 用户以及权限设置

创建radosgw的用户

[[email protected] ~]#  radosgw-admin user create --uid joy --display-name ‘Joy Ning‘

技术图片 
{
    "user_id": "joy",
    "display_name": "Joy Ning",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [],
    "keys": [
        {
            "user": "joy",
            "access_key": "X0CVIF04TAJVTN9D29UL",
            "secret_key": "vMmPqPap0FC0IRC5J3t9AIPgXNoiw1H9TOWELd5B"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw"
}
"Joy Ning"

修改信息

 [[email protected] ~]# radosgw-admin user modify --uid joy --display-name ‘joy Ningrui‘  --max_buckets 2000

 技术图片

 禁用suspend

[[email protected] ~]# radosgw-admin user suspend --uid joy

 技术图片

 启用

[[email protected] ~]# radosgw-admin user enable --uid joy

技术图片

列出用户

[[email protected] ~]# radosgw-admin user list

 技术图片

 删除用户

[[email protected] ~]# radosgw-admin user rm --uid joy

[[email protected] ~]# radosgw-admin user list

 [[email protected] ~]#  radosgw-admin user create --uid joy --display-name ‘Joy Ning‘

{
    "user_id": "joy",
    "display_name": "Joy Ning",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [],
    "keys": [
        {
            "user": "joy",
            "access_key": "5XCV68WUQJFFJPVM3UHK",
            "secret_key": "xhaA2YB1CA3xH54xLbmwPcglqjDyuFez36F8XGuG"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw"
}

 [[email protected] ~]# radosgw-admin key create --uid joy --display-name ‘Joy Ning‘ --key-type=s3  --gen-access-key --gen-secret

 技术图片

 删除key

[[email protected] ~]# radosgw-admin key rm --uid joy --display-name ‘Joy Ning‘ --key-type=s3 --access-key HPT1SBAXCXW46ZACKPY0

 技术图片

3.3 设置配额

基于用户的配额

[[email protected] ~]# radosgw-admin quota set --quota-scope=user --uid=joy --max-size 1

[[email protected] ~]# radosgw-admin user info --uid joy

技术图片 

开启配额

[[email protected] ~]# radosgw-admin quota enable --quota-scope=user --uid joy

[[email protected] ~]# radosgw-admin user info --uid joy

技术图片 

[[email protected] ~]# radosgw-admin quota set --quota-scope=bucket --uid=joy --max-size 1

[[email protected] ~]# radosgw-admin quota enable --quota-scope=bucket --uid=joy

[[email protected] ~]# radosgw-admin user info --uid joy

技术图片

注：如果两个都进行配置，则那个先到，使用哪一个

关闭配额

可以disable

[[email protected] ~]# radosgw-admin quota disable  --quota-scope=bucket --uid=joy

也可以参数设为1

[[email protected] ~]# radosgw-admin quota set --quota-scope=user --uid joy --max-size -1

[[email protected] ~]# radosgw-admin user info --uid joy

技术图片

3.4 统计数据

统计所有

[[email protected] ~]# radosgw-admin usage show --uid joy

技术图片

[[email protected] ~]# radosgw-admin usage show --uid joy --start-date 2019-03-19 21:00:00 --end-date 2019-03-19 22:00:00

技术图片

3.5 利用rados网关来访问s3对象

[[email protected] ~]#  vim /etc/ceph/backup.conf

[[email protected] ~]# systemctl restart [email protected]
[[email protected] ~]# ps -ef|grep rados

ceph     18072     1  2 21:52 ?        00:00:00 /usr/bin/radosgw -f --cluster backup --name client.rgw.ceph5 --setuser ceph --setgroup ceph

四 验证配置

4.1 配置s3cmd

[[email protected] ceph]# yum -y install s3cmd

[[email protected] ceph]# s3cmd --configure

[email protected]‘s password: 
Permission denied, please try again.
[email protected]‘s password: 
hosts                                                                                                                100%  786     1.6MB/s   00:00    
[[email protected] ceph]# s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: 5XCV68WUQJFFJPVM3UHK
Secret Key: xhaA2YB1CA3xH54xLbmwPcglqjDyuFez36F8XGuG
Default Region [US]: 

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: redhat
Path to GPG program [/usr/bin/gpg]: 

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]: no

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can‘t connect to S3 directly
HTTP Proxy server name: ceph5.lab.example.com
HTTP Proxy server port [3128]: 80

New settings:
  Access Key: 5XCV68WUQJFFJPVM3UHK
  Secret Key: xhaA2YB1CA3xH54xLbmwPcglqjDyuFez36F8XGuG
  Default Region: US
  Encryption password: redhat
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: False
  HTTP Proxy server name: ceph5.lab.example.com
  HTTP Proxy server port: 80

Test access with supplied credentials? [Y/n] n

Save settings? [y/N] y
Configuration saved to ‘/root/.s3cfg‘

[[email protected] ceph]# vim /root/.s3cfg

host_base = ceph5
host_bucket = %(bucket)s.ceph5.lab.example.com
cloudfront_host = cloudfront.amazonaws.com
website_endpoint = http://%(bucket)s.ceph5.lab.example.com/

4.2 创建bucket

[[email protected] ceph]# s3cmd mb s3://test

技术图片

4.3 传送数据

[[email protected] ceph]# echo 11111 >/tmp/demoobject

[[email protected] ceph]# s3cmd put --acl-public /tmp/demoobject s3://test/demoobject

技术图片

[[email protected] ceph]# vim /etc/hosts

172.25.250.10  ceph1    ceph1.lab.example.com servera
172.25.250.11  ceph2    ceph2.lab.example.com serverb
172.25.250.12  ceph3    ceph3.lab.example.com serverc
172.25.250.13  ceph4    ceph4.lab.example.com serverd
172.25.250.14  ceph5    ceph5.lab.example.com servere  test.ceph5.lab.example.com

4.4 访问bucket

[[email protected] ceph]# curl http://test.ceph5.lab.example.com/demoobject
技术图片

4.5 查看bucket

到服务端看

[[email protected] ~]# radosgw-admin bucket list

技术图片

[[email protected] ~]# radosgw-admin bucket stats --bucket=test

{
    "bucket": "test",
    "zonegroup": "e80133e1-a513-44f5-ba90-e25b6c987b26",
    "placement_rule": "default-placement",
    "explicit_placement": {
        "data_pool": "",
        "data_extra_pool": "",
        "index_pool": ""
    },
    "id": "1b85c5b1-19d2-48a1-bb45-3ac75895aeed.4235.1",
    "marker": "1b85c5b1-19d2-48a1-bb45-3ac75895aeed.4235.1",
    "index_type": "Normal",
    "owner": "joy",
    "ver": "0#3",
    "master_ver": "0#0",
    "mtime": "2019-03-19 22:02:50.726716",
    "max_marker": "0#",
    "usage": {
        "rgw.main": {
            "size": 6,
            "size_actual": 4096,
            "size_utilized": 6,
            "size_kb": 1,
            "size_kb_actual": 4,
            "size_kb_utilized": 1,
            "num_objects": 1
        }
    },
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    }
}

[[email protected] ~]# radosgw-admin bucket check --bucket=test

技术图片

删除

[[email protected] ~]# radosgw-admin bucket rm --bucket=test

[[email protected] ceph]# s3cmd put --acl-public  /etc/ceph/ceph.conf  s3://test/ceph

upload: ‘/etc/ceph/ceph.conf‘ -> ‘s3://test/ceph‘  [1 of 1]
 589 of 589   100% in    0s    20.96 kB/s  done
Public URL of the object is: http://test.ceph5/ceph

 [[email protected] ceph]# curl http://test.ceph5.lab.example.com/ceph

# Please do not change this file directly since it is managed by Ansible and will be overwritten

[global]
fsid = 35a91e48-8244-4e96-a7ee-980ab989d20d



mon initial members = ceph2,ceph3,ceph4
mon host = 172.25.250.11,172.25.250.12,172.25.250.13

public network = 172.25.250.0/24
cluster network = 172.25.250.0/24

auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

[osd]
osd mkfs type = xfs
osd mkfs options xfs = -f -i size=2048
osd mount options xfs = noatime,largeio,inode64,swalloc
osd journal size = 5120

[mon]
mon_allow_pool_delete = true

[[email protected] ceph]# s3cmd get s3://test/demoobject ./demoobject

download: ‘s3://test/demoobject‘ -> ‘./demoobject‘ [1 of 1]
6 of 6 100% in 0s 1346.20 B/s done

[[email protected] ceph]# cat ./demoobject

技术图片

4.6 查看底层数据

[[email protected] ~]# ceph osd pool ls
rbd
rbdmirror
.rgw.root
default.rgw.control
default.rgw.meta
default.rgw.log
default.rgw.buckets.index
default.rgw.buckets.data
[[email protected] ~]#  rados -p  default.rgw.buckets.index ls --cluster backup
.dir.1b85c5b1-19d2-48a1-bb45-3ac75895aeed.4235.1
[[email protected] ~]#  rados -p  default.rgw.buckets.data ls
error opening pool default.rgw.buckets.data: (2) No such file or directory
[[email protected] ~]#  rados -p  default.rgw.buckets.data ls --cluster backup
1b85c5b1-19d2-48a1-bb45-3ac75895aeed.4235.1_demoobject
1b85c5b1-19d2-48a1-bb45-3ac75895aeed.4235.1_ceph

实验完成

 

博主声明：本文的内容来源主要来自誉天教育晏威老师，由本人实验完成操作验证，需要的博友请联系誉天教育（http://www.yutianedu.com/），获得官方同意或者晏老师（https://www.cnblogs.com/breezey/）本人同意即可转载，谢谢！

 

以上是关于010 Ceph RGW对象存储的主要内容，如果未能解决你的问题，请参考以下文章

第⑦讲：Ceph集群RGW对象存储核心概念及部署使用

对象存储网关RADOS Gateway(RGW)

Ceph使用系列之——Ceph RGW使用

Ceph对象存储RGW对接企业级网盘OwnCloud三步走

Ceph存储的三种模式部署 MDS文件存储 RBD块存储 RGW对象存储

ceph rgw lifecycle（ 对象存储 对象生命周期）