Kettle案例三十七：变量活动类型
Posted
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Kettle案例三十七：变量活动类型相关的知识，希望对你有一定的参考价值。
参考技术A Valid in the virtual machine:Valid in the parent job:
Valid in the grand-parent job:
Valid in the root job:
综合案例：使用Configmap-reload动态热加载Configmap中的配置文件（三十七）
综合案例：使用Configmap-reload动态热加载Configmap中的配置文件
文章目录
1.configmap-reload资源热更新机制
在k8s集群中，当configmap以volume的形式挂载到pod内时，更新configmap，k8s会自动将更改的配置文件内容同步到pod挂载的文件中，这个并不是立刻生效的，大约需要1分钟左右，实际案例中，如果应用程序支持热更新功能，所谓热更新就是通过http接口的方式就可以更新程序的配置，如果支持热更新机制就可以使用configmap-reload监控配置文件的变更，当配置文件变更后调用程序接口自动将应用程序进行更新
以prometheus为例配置当promet.yaml更新后立即进行热更新
注意：只有支持热更新的程序才能进行configmap-reload配置
整体实现思路：
1.首先准备一个configmap资源，用于保存prometheus的配置文件
2.准备一套pv、pvc，用于持久化prometheus监控系统的数据
3.准备一个statfulset控制器跑prometheus pod，configmap-reload必须依靠固定的主机名，因此需要采用statfulset资源
4.准备svc资源，用于在外部访问prometheus
5.在所有k8s节点上部署node_exporter，用于prometheus采集数据
6.编辑configmap资源增加k8s节点的配置内容，观察configmap-reload是否会自动更新
2.使用configmap-reload实现prometheus监控系统配置热更新
2.1.准备prometheus的configmap资源
将prometheus配置文件以configmap的方式挂载到容器中
1.编写yaml
[root@k8s-master ~/k8s_1.19_yaml/configmap]# vim prometheus-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-configmap
namespace: prometheus
data:
prometheus.yaml: |
global:
scrape_interval: 90s
evaluation_interval: 90s
scrape_configs:
- job_name: prometheus
static_configs:
- targets:
- 127.0.0.1:9090
2.创建yaml
[root@k8s-master ~/k8s_1.19_yaml/configmap]# kubectl create -f prometheus-configmap.yaml
configmap/prometheus-configmap created
3.查看资源状态
[root@k8s-master ~/k8s_1.19_yaml/configmap]# kubectl get cm -n prometheus
NAME DATA AGE
prometheus-configmap 1 4m56s
4.查看资源的详细信息
[root@k8s-master ~/k8s_1.19_yaml/configmap]# kubectl describe cm -n prometheus
Name: prometheus-configmap
Namespace: prometheus
Labels: <none>
Annotations: <none>
Data
====
prometheus.yaml:
----
global:
scrape_interval: 90s
evaluation_interval: 90s
scrape_configs:
- job_name: prometheus
static_configs:
- targets:
- 127.0.0.1:9090
Events: <none>
2.2.准备prometheus pv、pvc资源
为prometheus数据提供持久化存储
1.编写yaml
[root@k8s-master ~/k8s_1.19_yaml/configmap]# vim prometheus-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-2
labels:
pv: pv-2
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
nfs: #以nfs类型存储
path: /data/pv_2/prometheus
server: 192.168.81.210
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-2
namespace: prometheus
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
selector:
matchLabels:
pv: pv-2 #通过标签方式管理pv-2的pv存储
2.创建资源
[root@k8s-master ~/k8s_1.19_yaml/configmap]# kubectl create -f prometheus-pv.yaml
persistentvolume/pv-2 created
persistentvolumeclaim/pvc-2 created
2.3.编写prometheus statfulset、svc资源
使用statefulset资源控制器部署prometheus容器并使用svc进行暴露，由于configmap-reload必须要配合固定的主机名进行使用，因此采用statefulset控制器
1.编写statfulset控制器yaml文件
[root@k8s-master ~/k8s_1.19_yaml/configmap]# vim prometheus-statfulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: statefulset-prometheus
namespace: prometheus
spec:
serviceName: "prometheus"
replicas: 1
podManagementPolicy: "Parallel"
updateStrategy:
type: "RollingUpdate"
selector:
matchLabels:
app: prometheus-pod
template:
metadata:
labels:
app: prometheus-pod
spec:
initContainers: #定义一个初始化容器，主要是给/data目录赋权限并挂载pvc
- name: "init-chown-data"
image: "busybox:1.30"
command: ["chown", "-R", "65534:65534", "/data"]
volumeMounts:
- name: prometheus-data
mountPath: /data
containers:
- name: prometheus
image: prom/prometheus:v2.23.0
args: #定义prometheus启动参数
- --config.file=/etc/config/prometheus.yaml
- --storage.tsdb.path=/data
- --web.console.libraries=/etc/prometheus/console_libraries
- --web.console.templates=/etc/prometheus/consoles
- --web.enable-lifecycle #启动prometheus热更新机制
ports:
- containerPort: 9090
volumeMounts:
- name: prometheus-data
mountPath: /data
- name: prometheus-config
mountPath: /etc/config
volumes:
- name: prometheus-data
persistentVolumeClaim:
claimName: pvc-2
readOnly: false
- name: prometheus-config
configMap:
name: prometheus-configmap
2.编写service资源的yaml文件
[root@k8s-master ~/k8s_1.19_yaml/configmap]# vim prometheus-svc.yaml
apiVersion: v1
kind: Service
metadata:
name: prometheus-service
namespace: prometheus
spec:
selector:
app: prometheus-pod
type: NodePort
ports:
- port: 9090
targetPort: 9090
3.创建资源
[root@k8s-master ~/k8s_1.19_yaml/configmap]# kubectl create -f prometheus-statfulset.yaml
statefulset.apps/statefulset-prometheus created
[root@k8s-master ~/k8s_1.19_yaml/configmap]# kubectl create -f prometheus-svc.yaml
service/prometheus-service created
2.4.访问prometheus
1.查看prometheus创建的资源
[root@k8s-master ~/k8s_1.19_yaml/configmap]# kubectl get all -n prometheus
NAME READY STATUS RESTARTS AGE
pod/statefulset-prometheus-0 1/1 Running 1 133m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/prometheus-service NodePort 10.99.144.79 <none> 9090:30877/TCP 134m
NAME READY AGE
statefulset.apps/statefulset-prometheus 1/1 134m
2.访问node节点ip加30877即可访问prometheus
2.5.配置configmap热更新
主要用来当configmap资源文件更新后，prometheus pod会自动加载配置并进行更新
1.修改yaml文件增加configmap-reload配置
[root@k8s-master ~/k8s_1.19_yaml/configmap]# vim prometheus-statfulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: statefulset-prometheus
namespace: prometheus
spec:
serviceName: "prometheus"
replicas: 1
podManagementPolicy: "Parallel"
updateStrategy:
type: "RollingUpdate"
selector:
matchLabels:
app: prometheus-pod
template:
metadata:
labels:
app: prometheus-pod
spec:
initContainers:
- name: "init-chown-data"
image: "busybox:1.30"
command: ["chown", "-R", "65534:65534", "/data"]
volumeMounts:
- name: prometheus-data
mountPath: /data
containers:
- name: prometheus-configmap-reload #增加一个configmap-reload的容器即可
image: "jimmidyson/configmap-reload:v0.1"
imagePullPolicy: "IfNotPresent"
args: #configmap-reload需要增加的参数
- --volume-dir=/etc/prometheus
- --webhook-url=http://localhost:9090/-/reload
volumeMounts: #将configmap挂载到/etc/prometheus
- name: prometheus-config
mountPath: /etc/prometheus
readOnly: true
- name: prometheus
image: prom/prometheus:v2.23.0
args:
- --config.file=/etc/config/prometheus.yaml
- --storage.tsdb.path=/data
- --web.console.libraries=/etc/prometheus/console_libraries
- --web.console.templates=/etc/prometheus/consoles
- --web.enable-lifecycle
ports:
- containerPort: 9090
volumeMounts:
- name: prometheus-data
mountPath: /data
- name: prometheus-config
mountPath: /etc/config
volumes:
- name: prometheus-data
persistentVolumeClaim:
claimName: pvc-2
readOnly: false
- name: prometheus-config
configMap:
name: prometheus-configmap
2.更新yaml文件
[root@k8s-master ~/k8s_1.19_yaml/configmap]# kubectl apply -f prometheus-statfulset.yaml
statefulset.apps/statefulset-prometheus configured
2.6.更新configmap配置文件观察prometheus容器是否会自动更新
在node节点安装node_exporter监控采集器，并修改prometheus configmap的配置，等待一段时间后观察prometheus是否会自动更新配置
1.所有master、node节点安装并启动node_exporter
tar xf node_exporter-1.0.1.linux-amd64.tar.gz
cp node_exporter-1.0.1.linux-amd64/node_exporter /usr/bin/
nohup /usr/bin/node_exporter &
netstat -lnpt | grep 9100
2.修改configmap中prometheus配置文件的内容增加node节点的配置
#这里只增加了master和node-1节点
[root@k8s-master ~/k8s_1.19_yaml/configmap]# vim prometheus-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-configmap
namespace: prometheus
data:
prometheus.yaml: |
global:
scrape_interval: 90s
evaluation_interval: 90s
scrape_configs:
- job_name: prometheus
static_configs:
- targets:
- 127.0.0.1:9090
- job_name: k8s-node
static_configs:
- targets:
- 192.168.81.210:9100
- 192.168.81.220:9100
3.更新configmap资源
[root@k8s-master ~/k8s_1.19_yaml/configmap]# kubectl apply -f prometheus-configmap.yaml
configmap/prometheus-configmap configured
4.访问prometheus观察配置是否更新
直接在页面上查看prometheus的配置是否增加和targets里面是否有主机增加就可以看到效果了
以上是关于Kettle案例三十七：变量活动类型的主要内容，如果未能解决你的问题，请参考以下文章
2021年大数据Flink（三十七）：Table与SQL 案例四
综合案例：使用Configmap-reload动态热加载Configmap中的配置文件（三十七）