云原生通用存储框架：Rook介绍与使用
Posted 新钛云服
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了云原生通用存储框架：Rook介绍与使用相关的知识，希望对你有一定的参考价值。
新钛云服已为您服务1007天
Rook-Ceph架构
关于Ceph
-
MON（Ceph监视器）负责群集仲裁。所有群集节点都向MON报告，并共享有关其状态的每个更改的信息。 -
OSD（Ceph对象存储设备）负责存储对象并通过网络提供对它们的访问。 -
MGR（Ceph Manager）提供了额外的监视和与外部管理系统的接口。 -
RADOS（可靠的自治分布式对象存储）是Ceph集群的核心。RADOS确保所存储的数据始终与数据复制，故障检测和恢复保持一致。 -
LibRADOS是用于访问RADOS的库。借助对多种编程语言的支持，LibRADOS为RADOS提供了本机接口，并为其他高级服务（例如RBD，RGW和CephFS）提供了基础。 -
RBD（RADOS块设备）现在称为Ceph块设备，它提供了持久的块存储，它是精简配置的，可调整大小的，并存储在多个OSD上条带化的数据。 -
RGW（RADOS网关）是提供对象存储服务的接口。它使用libRGW（RGW库）和libRADOS来与应用程序之间的Ceph对象存储建立连接。RGW提供了与Amazon S3和OpenStack Swift兼容的RESTful API。 -
CephFS是提供POSIX兼容文件系统的Ceph文件系统。CephFS使用Ceph集群存储用户数据。 -
MDS（Ceph Metadata Server）跟踪文件层次结构，并仅存储CephFS的元数据。
Operator模式
Ceph Rook
-
Ceph CSI（容器存储接口）是一种标准，用于将任意块和文件存储系统暴露给容器编排系统（如Kubernetes）上的容器化工作负载。Ceph CSI与Rook集成在一起，并支持两种方案： -
RBD（块存储）：此驱动程序针对RWO pod访问进行了优化，其中只有一个pod可以访问存储。 -
CephFS（文件系统）：此驱动程序允许带有一个或多个Pod的RWX访问同一存储。 -
对于对象存储，Rook支持创建新存储桶以及通过两个自定义资源访问现有存储桶：对象存储桶声明（OBC）和对象存储桶（OB）。应用程序可以通过RGW访问对象。
-
Toolbox + Ceph CLI：推荐的方法是在Rook Toolbox终端中执行Ceph命令。 -
Ceph仪表板：第二种推荐方法是使用Ceph Dashboard。此方法的优先级与通过Ceph CLI进行配置的优先级相同。 -
通过ceph.conf的高级配置覆盖ConfigMap。某些设置无法通过CLI或信息中心轻松修改。例如，我们不能使用Ceph CLI直接删除MON。唯一的方法是覆盖configMap。
开始使用
软件配置
Rook-Ceph部署
准备Kubernetes集群
sudo apt-get update && sudo apt-get install -y apt-
transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo
apt-key add -
cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
kubeadm config images pull --kubernetes-version=v1.18.6
sudo swapoff -a
sudo kubeadm init --pod-network-cidr 10.244.0.0/16 --apiserver-advertise-address <your_host_ip> --node-name master --ignore-preflight-errors=all
kubectl apply -f https://docs.projectcalico.org/v3.14/manifests/calico.yaml
kubectl taint nodes master node-role.kubernetes.io/master:NoSchedule-
kubeadm join <master_host_ip>:6443 --token zifmp3.27h736nwdfjli6fi \
--discovery-token-ca-cert-hash sha256:5c11bfd28f016fd15b656850324de5d4d9a042c9a9e620aba3d1c959b7ac0ad5
Rook和Ceph集群配置
kubectl apply -f https://github.com/rook/rook/blob/release-1.3/cluster/examples/kubernetes/ceph/common.yaml
kubectl apply -f https://github.com/rook/rook/blob/release-1.3/cluster/examples/kubernetes/ceph/operator.yaml
kubectl apply -f https://github.com/rook/rook/blob/release-1.3/cluster/examples/kubernetes/ceph/cluster.yaml
mon:
count: 3
allowMultiplePerNode: false
storage: # cluster level storage configuration and selection
useAllNodes: true
useAllDevices: true
#deviceFilter:
config:
# metadataDevice: "md0" # specify a non-rotational storage so ceph-volume will use it as block db device of bluestore.
# databaseSizeMB: "1024" # uncomment if the disks are smaller than 100 GB
# journalSizeMB: "1024" # uncomment if the disks are 20 GB or smaller
# osdsPerDevice: "1" # this value can be overridden at the node or device level
# encryptedDevice: "true" # the default value for this option is "false"
# Individual nodes and their config can be specified as well, but 'useAllNodes' above must be set to false. Then, only the named
# nodes below will be used as storage resources. Each node's 'name' field should match their 'kubernetes.io/hostname' label.
# nodes:
# - name: "172.17.4.201"
# devices: # specific devices to use for storage can be specified for each node
# - name: "sdb"
# - name: "nvme01" # multiple osds can be created on high performance devices
# config:
# osdsPerDevice: "5"
# - name: "/dev/disk/by-id/ata-ST4000DM004-XXXX" # devices can be specified using full udev paths
# config: # configuration can be specified at the node level which overrides the cluster level config
# storeType: filestore
# - name: "172.17.4.301"
# deviceFilter: "^sd."
kubectl apply -f https://github.com/rook/rook/blob/release-1.3/cluster/examples/kubernetes/ceph/cluster-on-pvc.yaml
storage:
storageClassDeviceSets:
- name: set1
# The number of OSDs to create from this device set
count: 3
# IMPORTANT: If volumes specified by the storageClassName are not portable across nodes
# this needs to be set to false. For example, if using the local storage provisioner
# this should be false.
portable: true
# Certain storage class in the Cloud are slow
# Rook can configure the OSD running on PVC to accommodate that by tuning some of the Ceph internal
# Currently, "gp2" has been identified as such
tuneDeviceClass: true
# Since the OSDs could end up on any node, an effort needs to be made to spread the OSDs
# across nodes as much as possible. Unfortunately, the pod anti-affinity breaks down
# as soon as you have more than one OSD per node. If you have more OSDs than nodes, K8s may
# choose to schedule many of them on the same node. What we need is the Pod Topology
# Spread Constraints.
# Another approach for a small number of OSDs is to create a separate device set for each
# zone (or other set of nodes with a common label) so that the OSDs will end up on different
# nodes. This would require adding nodeAffinity to the placement here.
placement:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- rook-ceph-osd
- key: app
operator: In
values:
- rook-ceph-osd-prepare
topologyKey: kubernetes.io/hostname
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- rook-ceph-osd
- rook-ceph-osd-prepare
resources:
# limits:
# cpu: "500m"
# memory: "4Gi"
# requests:
# cpu: "500m"
# memory: "4Gi"
volumeClaimTemplates:
- metadata:
name: data
# if you are looking to give your OSD a different CRUSH device class than the one detected by Ceph
# annotations:
# crushDeviceClass: hybrid
spec:
resources:
requests:
storage: 64Gi
# IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, gp2)
storageClassName: local-sc
volumeMode: Block
accessModes:
- ReadWriteOnce
# dedicated block device to store bluestore database (block.db)
- metadata:
name: metadata
spec:
resources:
requests:
# Find the right size https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#sizing
storage: 3Gi
# IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, gp2)
storageClassName: local-sc
# volumeMode: Block
# accessModes:
# - ReadWriteOnce
kubectl apply -f https://github.com/rook/rook/blob/release-1.3/cluster/examples/kubernetes/ceph/toolbox.yaml
Volume Claim与应用程序
kubectl apply -f https://github.com/rook/rook/blob/release-1.3/cluster/examples/kubernetes/ceph/csi/rbd/storageclass.yaml
kubectl apply -f https://github.com/cockroachdb/cockroach/blob/master/cloud/kubernetes/cockroachdb-statefulset.yaml
volumeClaimTemplates:
- metadata:
name: datadir
spec:
accessModes:
- "ReadWriteOnce"
storageClassName: rook-ceph-block
resources:
requests:
storage: 16Gi
配置Ceph集群
kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash
kubectl edit configmap rook-ceph-mon-endpoints -n rook-ceph
kubectl edit configmap rook-ceph-override -n rook-ceph
-------------------------------------
apiVersion: v1
kind: ConfigMap
metadata:
name: rook-config-override
namespace: rook-ceph
data:
config: |
[global]
osd crush update on start = false
osd pool default size = 2
概括：
参考：
Rook.io: https://rook.io/docs/rook/v1.3/
Rook github: https://github.com/rook/rook
原文：https://01.org/kubernetes/blogs/tingjie/2020/introduction-cloud-native-storage-orchestrator-rook
了解新钛云服
往期技术干货
以上是关于云原生通用存储框架：Rook介绍与使用的主要内容，如果未能解决你的问题，请参考以下文章
Kuberntes云原生实战六 使用Rook搭建Ceph集群
Kuberntes云原生实战六 使用Rook搭建Ceph集群
Kubernetes云原生实战06 使用Rook搭建Ceph集群
Rook：Kubernetes上提供Kubernetes云原生本地持久存储