Ceph object Gateway 之 S3 API

Posted 2023-03-17

s3配置

创建用户

radosgw-admin user create --uid cjid --display-name cj

返回：

"user": "cjid",
"access_key": "77SH7KHBF310F4NTIA3A",
"secret_key": "THxUb5kcWBObET3LwkR0ObVjgzzzf28czqp161Nj"

安装s3cmd

yum install s3cmd

修改s3cmd 配置

~/.s3cfg

[default]
access_key = 77SH7KHBF310F4NTIA3A
secret_key = THxUb5kcWBObET3LwkR0ObVjgzzzf28czqp161Nj
host_base = 192.168.164.221:5588
host_bucket = 192.168.164.221:5588/%(bucket)
cloudfront_host = 192.168.164.221:5588
use_https = False

运行s3cmd ls, 可显示结果说明配置成功

注：端口配置需要查看 /etc/ceph/ceph.conf, rgw frontends = "beast port=5588", 确认rgw服务启动

List Bucket

描述：列出桶

运行命令：s3cmd ls [s3://BUCKET[/PREFIX]]

示例：

Delete Bucket

描述：删除桶

运行命令：s3cmd rb s3://BUCKET

示例：

Create Bucket

描述：创建桶

运行命令：s3cmd mb s3://BUCKET

示例：

Bucket Lifecycle

设置桶生命周期

运行命令：s3cmd setlifecycle FILE s3://BUCKET

获取桶生命周期策略

运行命令：s3cmd getlifecycle s3://BUCKET

删除桶生命周期策略

运行命令：s3cmd dellifecycle s3://BUCKET

示例xml：

<?xml version="1.0"?>
<LifecycleConfiguration>
  <Rule>
    <ID>prefix expiration</ID>
    <Filter>
      <Prefix>dir1</Prefix>
    </Filter>
    <Status>Enabled</Status>
    <Expiration>
      <Days>120</Days>
    </Expiration>
    <Filter>
      <Prefix>dir2</Prefix>
    </Filter>
    <Status>Enabled</Status>
    <Expiration>
      <Days>1</Days>
    </Expiration>
  </Rule>
</LifecycleConfiguration>

示例：

Bucket Replication

描述：桶复制

运行命令：s3cmd cp s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]

示例：

Policy (Buckets, Objects)

描述：桶，对象策略 (ACLs(访问控制列表)支持)

添加,修改桶策略

运行命令：s3cmd setpolicy FILE s3://BUCKET

删除桶策略

运行命令：s3cmd delpolicy s3://BUCKET

示例：

  "Version": "2012-10-17",
  "Statement": [
    "Effect": "Allow",
    "Principal": "AWS": ["arn:aws:iam::usfolks:user/fred:subuser"],
    "Action": "s3:PutObjectAcl",
    "Resource": [
      "arn:aws:s3:::happybucket/*"
    ]
  ]

获取桶策略

运行命令: 

GET /bucket/?policy= HTTP/1.1
Host: Bucket.s3.amazonaws.com
x-amz-expected-bucket-owner: ExpectedBucketOwner

示例：

注：Principal字段见官网说明：

​​https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-bucket-user-policy-specifying-principal-intro.html​​

IAM user policies：IAM user policies for controlling user access to Amazon S3.

Bucket Website

描述：桶网站

创建website

运行命令：s3cmd ws-create s3://BUCKET

删除website

运行命令：s3cmd ws-delete s3://BUCKET

website信息

运行命令：s3cmd ws-info s3://BUCKET

Bucket ACLs (Get, Put)

获取桶访问控制列表

GET /?acl HTTP/1.1
Host: Bucket.s3.amazonaws.com
x-amz-expected-bucket-owner: ExpectedBucketOwner

例子：

设置桶访问控制列表

s3cmd setacl s3://BUCKET[/OBJECT]

PUT /?acl HTTP/1.1
Host: Bucket.s3.amazonaws.com
x-amz-acl: ACL
Content-MD5: ContentMD5
x-amz-sdk-checksum-algorithm: ChecksumAlgorithm
x-amz-grant-full-control: GrantFullControl
x-amz-grant-read: GrantRead
x-amz-grant-read-acp: GrantReadACP
x-amz-grant-write: GrantWrite
x-amz-grant-write-acp: GrantWriteACP
x-amz-expected-bucket-owner: ExpectedBucketOwner
<?xml versinotallow="1.0" encoding="UTF-8"?>
<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
   <AccessControlList>
      <Grant>
         <Grantee>
            <DisplayName>string</DisplayName>
            <EmailAddress>string</EmailAddress>
            <ID>string</ID>
            <xsi:type>string</xsi:type>
            <URI>string</URI>
         </Grantee>
         <Permission>string</Permission>
      </Grant>
   </AccessControlList>
   <Owner>
      <DisplayName>string</DisplayName>
      <ID>string</ID>
   </Owner>
</AccessControlPolicy>

Bucket Location

描述：获取桶的位置

运行命令：

GET /?location HTTP/1.1
Host: Bucket.s3.amazonaws.com
x-amz-expected-bucket-owner: ExpectedBucketOwner

存储URL: protocol://service-code.region-code.amazonaws.com

region-code对应的桶的位置

示例：

Bucket Notification

描述：桶通知

兼容性：AWS支持（SNS,SQS,Lambda）, HTTP/S, Kafka and AMQP

s3消息格式：(其中requestParameters.sourceIPAddress要为空 )

  
   "Records":[  
        
         "eventVersion":"2.2",
         "eventSource":"aws:s3",
         "awsRegion":"us-west-2",
         "eventTime":"The time, in ISO-8601 format, for example, 1970-01-01T00:00:00.000Z, when Amazon S3 finished processing the request",
         "eventName":"event-type",
         "userIdentity":  
            "principalId":"Amazon-customer-ID-of-the-user-who-caused-the-event"
         ,
         "requestParameters":  
            "sourceIPAddress":"ip-address-where-request-came-from"
         ,
         "responseElements":  
            "x-amz-request-id":"Amazon S3 generated request ID",
            "x-amz-id-2":"Amazon S3 host that processed the request"
         ,
         "s3":  
            "s3SchemaVersion":"1.0",
            "configurationId":"ID found in the bucket notification configuration",
            "bucket":  
               "name":"bucket-name",
               "ownerIdentity":  
                  "principalId":"Amazon-customer-ID-of-the-bucket-owner"
               ,
               "arn":"bucket-ARN"
            ,
            "object":  
               "key":"object-key",
               "size":"object-size in bytes",
               "eTag":"object eTag",
               "versionId":"object version if bucket is versioning-enabled, otherwise null",
               "sequencer": "a string representation of a hexadecimal value used to determine event sequence, only used with PUTs and DELETEs"
            
         ,
         "glacierEventData": 
            "restoreEventData": 
               "lifecycleRestorationExpiryTime": "The time, in ISO-8601 format, for example, 1970-01-01T00:00:00.000Z, of Restore Expiry",
               "lifecycleRestoreStorageClass": "Source storage class for restore"
            
         
      
   ]

事件不支持列表：

s3:ObjectSynced:Delete
s3:ObjectSynced:DeletionMarkerCreated
s3:ObjectLifecycle:Expiration:AbortMultipartUpload
s3:ObjectRestore:Post
s3:ObjectRestore:Complete
s3:ReducedRedundancyLostObject

消息队列TOPIC 行为仅支持:

CreateTopic
DeleteTopic
ListTopics

Bucket Object Versions

描述：桶对象版本

运行命令：

GET /?versions HTTP/1.1
Host: BucketName.s3.<Region>.amazonaws.com
Date: Wed, 28 Oct 2009 22:32:00 +0000
Authorization: authorization string (see Authenticating Requests (AWS Signature Version
    4))

运行结果：

Get Bucket Info (HEAD)

描述：获取桶信息

运行命令：s3cmd info s3://BUCKET

例子：

Bucket Request Payment

描述：获取桶请求支付配置

运行命令：

GET /?requestPayment HTTP/1.1
Host: Bucket.s3.amazonaws.com
x-amz-expected-bucket-owner: ExpectedBucketOwner

Put Object

描述：写入对象

运行命令：s3cmd put FILE [FILE...] s3://BUCKET[/PREFIX]

示例：

Delete Object

描述：删除对象

运行命令：

• s3cmd del s3://BUCKET/OBJECT
• s3cmd rm s3://BUCKET/OBJECT

示例：

Get Object

描述：获取对象

运行命令：s3cmd get s3://BUCKET/OBJECT LOCAL_FILE

Object ACLs (Get, Put)

得到对象访问控制列表

运行命令：

GET /Key+?acl&versionId=VersionId HTTP/1.1
Host: Bucket.s3.amazonaws.com
x-amz-request-payer: RequestPayer
x-amz-expected-bucket-owner: ExpectedBucketOwner

设置对象访问控制列表

运行命令：

s3cmd setacl s3://BUCKET[/OBJECT]

PUT /Key+?acl&versionId=VersionId HTTP/1.1
Host: Bucket.s3.amazonaws.com
x-amz-acl: ACL
Content-MD5: ContentMD5
x-amz-sdk-checksum-algorithm: ChecksumAlgorithm
x-amz-grant-full-control: GrantFullControl
x-amz-grant-read: GrantRead
x-amz-grant-read-acp: GrantReadACP
x-amz-grant-write: GrantWrite
x-amz-grant-write-acp: GrantWriteACP
x-amz-request-payer: RequestPayer
x-amz-expected-bucket-owner: ExpectedBucketOwner
<?xml version="1.0" encoding="UTF-8"?>
<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
   <AccessControlList>
      <Grant>
         <Grantee>
            <DisplayName>string</DisplayName>
            <EmailAddress>string</EmailAddress>
            <ID>string</ID>
            <xsi:type>string</xsi:type>
            <URI>string</URI>
         </Grantee>
         <Permission>string</Permission>
      </Grant>
   </AccessControlList>
   <Owner>
      <DisplayName>string</DisplayName>
      <ID>string</ID>
   </Owner>
</AccessControlPolicy>

Get Object Info (HEAD)

描述：获取桶对象信息

运行命令：s3cmd info s3://BUCKET[/OBJECT]

Copy Object

描述：拷贝桶对象信息

运行命令：s3cmd cp s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]

Multipart Uploads

描述：多分片上传

运行命令：s3cmd multipart s3://BUCKET [Id]

使用场景：对象超过100m的时候

好处：

• 通过并发提高吞吐量
• 在有网络问题的时候可以快速恢复
• 能暂停与恢复上传
• 可以在接受对象的同时上传对象

Object Tagging

获取对象标签

运行命令：

GET /Key+?tagging&versionId=VersionId HTTP/1.1
Host: Bucket.s3.amazonaws.com
x-amz-expected-bucket-owner: ExpectedBucketOwner
x-amz-request-payer: RequestPayer

删除对象标签

运行命令：

DELETE /Key+?tagging&versionId=VersionId HTTP/1.1
Host: Bucket.s3.amazonaws.com
x-amz-expected-bucket-owner: ExpectedBucketOwner

设置对象标签

运行命令：

PUT /Key+?tagging&versionId=VersionId HTTP/1.1
Host: Bucket.s3.amazonaws.com
Content-MD5: ContentMD5
x-amz-sdk-checksum-algorithm: ChecksumAlgorithm
x-amz-expected-bucket-owner: ExpectedBucketOwner
x-amz-request-payer: RequestPayer
<?xml versinotallow="1.0" encoding="UTF-8"?>
<Tagging xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
   <TagSet>
      <Tag>
         <Key>string</Key>
         <Value>string</Value>
      </Tag>
   </TagSet>

例子：

Bucket Tagging

获取桶标签

GET /?tagging HTTP/1.1
Host: Bucket.s3.amazonaws.com
x-amz-expected-bucket-owner: ExpectedBucketOwner

删除桶标签

DELETE /?tagging HTTP/1.1
Host: Bucket.s3.amazonaws.com
x-amz-expected-bucket-owner: ExpectedBucketOwner

设置桶标签

PUT /?tagging HTTP/1.1
Host: Bucket.s3.amazonaws.com
Content-MD5: ContentMD5
x-amz-sdk-checksum-algorithm: ChecksumAlgorithm
x-amz-expected-bucket-owner: ExpectedBucketOwner
<?xml versinotallow="1.0" encoding="UTF-8"?>
<Tagging xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
   <TagSet>
      <Tag>
         <Key>string</Key>
         <Value>string</Value>
      </Tag>
   </TagSet>
</Tagging>

例子：

Storage Class

描述：用来定制对象数据的放置(placement)，s3的桶生命周期的规则可以在storage classes之间自动进行对象转换

例子：

zonegroup的placement target 列举了一个初始化名字为STANDARD类

$radosgw-admin zonegroup get


    "id": "ab01123f-e0df-4f29-9d71-b44888d67cd5",
    "name": "default",
    "api_name": "default",
    ...
    "placement_targets": [
        
            "name": "default-placement",
            "tags": [],
            "storage_classes": [
                "STANDARD"
            ]
        
    ],
    "default_placement": "default-placement",
    ...

zone 配置负责提供一个data_pool 的名字为每个zonegroup 的存储类

$ radosgw-admin zone get

    "id": "557cdcee-3aae-4e9e-85c7-2f86f5eddb1f",
    "name": "default",
    "domain_root": "default.rgw.meta:root",
    ...
    "placement_pools": [
        
            "key": "default-placement",
            "val": 
                "index_pool": "default.rgw.buckets.index",
                "storage_classes": 
                    "STANDARD": 
                        "data_pool": "default.rgw.buckets.data"
                    
                ,
                "data_extra_pool": "default.rgw.buckets.non-ec",
                "index_type": 0,
                "inline_data": true
            
        
    ],
    ...

S3cmd 不支持的命令

参考该文档，用postman 进行调用，参数已经配置好，需要修改部分参数即可

​​​​​​https://documenter.getpostman.com/view/10394726/SzYbxHEf​​

例子（红框里的内容是需要修改的）：

特殊名词

SNS (Simple Notification Service)

是一种完全托管的发布/订阅消息收发和移动通知服务，用于协调向订阅终端节点和客户端的消息分发。

SQS (Simple Queue Service)

可以轻松分离和扩展微服务，分布式系统和无服务应用程序，对程序进行解耦

Lambda

Lambda 是一种高度可用、无服务器、事件驱动的计算服务

4ceph-deploy之配置使用对象存储

从firefly（v0.80）版本开始，ceph存储显著的简化了安装和配置Ceph Object Gateway，

Gateway进程嵌入到Civetweb，所以你需要安装一个web服务，或者配置FastCGI，

另外，ceph-deploy可以安装gateway包，生成key，配置数据目录，创建gateway实例。

Tip：Civetweb默认使用7480端口，您必须打开端口7480，或将端口设置为Ceph配置文件中的首选端口（例如端口80）。

安装Ceph Object Gateway

1、保证开启端口

2、在admin节点安装Ceph Object Gateway包到client-node节点

1. ceph-deploy install --rgw <client-node> [<client-node>...]

创建Ceph Object Gateway实例

1. ceph-deploy rgw create <client-node>

一旦gateway运行，你可以访问它的7480端口（例如：http://client-node:7480）

配置Ceph Object Gateway实例

1、通过修改ceph.conf配置文件修改默认端口，添加标题为 [client.rgw.<client-node>]的部分，

用ceph client节点hostname（hostname -s）替换<client-node>.

如果节点name是client-node，那么在[global]部分之后，添加如下：

1. [client.rgw.client-node]
2. rgw_frontends = "civetweb port=80"

注意：

确保在rgw_frontends键/值对中的 端口=端口号 之间不留空白，

如果您打算使用端口80，请确保Apache服务器未运行，否则将与Civetweb发生冲突。在这种情况下，我们建议删除Apache。

2、重启使新配置端口生效

1. systemctl restart cph-radosgw

3、如果开启了防火墙，检查端口在防火墙中打开。如果未打开，添加端口重载firewall生效

1. firewall-cmd --list-all
2. firewall-cmd --zone=public --add-port 80/tcp --permanent
3. firewall-cmd --reload

4、现在可以发起一个未认证请求，并且返回一个结果

1. #request
2. curl http://<client-node>:80
4. #result
5. <?xml version="1.0" encoding="UTF-8"?>
6. <ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
7. <Owner>
8. <ID>anonymous</ID>
9. <DisplayName></DisplayName>
10. </Owner>
11. <Buckets>
12. </Buckets>
13. </ListAllMyBucketsResult>