Flume-数据流监控 Ganglia
Posted jhxxb
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了Flume-数据流监控 Ganglia相关的知识,希望对你有一定的参考价值。
Ganglia 由 gmond、gmetad 和 gweb 三部分组成。
gmond(Ganglia Monitoring Daemon)是一种轻量级服务,安装在每台需要收集指标数据的节点主机上。使用 gmond,你可以很容易收集很多系统指标数据,如 CPU、内存、磁盘、网络和活跃进程的数据等。
gmetad(Ganglia Meta Daemon)整合所有信息,并将其以 RRD 格式存储至磁盘的服务。
gweb(Ganglia Web)Ganglia 可视化工具,gweb 是一种利用浏览器显示 gmetad 所存储数据的 php 前端。在 Web 界面中以图表方式展现集群的运行状态下收集的多种不同指标数据。
一、安装
# 替换阿里 yum 源 mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo sed -i -e ‘/mirrors.cloud.aliyuncs.com/d‘ -e ‘/mirrors.aliyuncs.com/d‘ /etc/yum.repos.d/CentOS-Base.repo # 安装 httpd 服务与 php sudo yum -y install httpd php # 安装其他依赖 sudo yum -y install rrdtool perl-rrdtool rrdtool-devel sudo yum -y install apr-devel # ganglia 在 epel 源中 mv /etc/yum.repos.d/epel.repo /etc/yum.repos.d/epel.repo.backup mv /etc/yum.repos.d/epel-testing.repo /etc/yum.repos.d/epel-testing.repo.backup curl -o /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo # 安装 ganglia sudo yum -y install ganglia-gmetad sudo yum -y install ganglia-web sudo yum -y install ganglia-gmond
二、配置
1.httpd 配置
https://blog.csdn.net/mxl00z/article/details/83041266
sudo vim /etc/httpd/conf.d/ganglia.conf
ganglia.conf
#
# Ganglia monitoring system php web frontend
#
Alias /ganglia /usr/share/ganglia
<Location /ganglia>
# Options Indexes FollowSymLinks
# AllowOverride None
Require all granted
# Order deny,allow
# Deny from all
# Allow from all
# Require local
# Require ip 10.1.2.3
# Require host example.org
</Location>
2.gmetad 配置
sudo vim /etc/ganglia/gmetad.conf
gmetad.conf
。。。。。。
# A list of machines which service the data source follows, in the # format ip:port, or name:port. If a port is not specified then 8649 # (the default gmond port) is assumed. # default: There is no default value # # data_source "my cluster" 10 localhost my.machine.edu:8649 1.2.3.5:8655 # data_source "my grid" 50 1.3.4.7:8655 grid.org:8651 grid-backup.org:8651 # data_source "another source" 1.3.4.7:8655 1.3.4.8 data_source "h136" 192.168.8.136 。。。。。。
3.gmond 配置
sudo vim /etc/ganglia/gmond.conf
gmond.conf
。。。。。。 /* * The cluster attributes specified will be used as part of the <CLUSTER> * tag that will wrap all hosts collected by this instance. */ cluster name = "h136" owner = "unspecified" latlong = "unspecified" url = "unspecified" /* The host section describes attributes of the host, like the location */ host location = "unspecified" /* Feel free to specify as many udp_send_channels as you like. Gmond used to only support having a single channel */ udp_send_channel #bind_hostname = yes # Highly recommended, soon to be default. # This option tells gmond to use a source address # that resolves to the machine‘s hostname. Without # this, the metrics may appear to come from any # interface and the DNS names associated with # those IPs will be used to create the RRDs. # mcast_join = 239.2.11.71 host = 192.168.8.136 port = 8649 ttl = 1 /* You can specify as many udp_recv_channels as you like as well. */ udp_recv_channel # mcast_join = 239.2.11.71 port = 8649 bind = 192.168.8.136 retry_bind = true # Size of the UDP buffer. If you are handling lots of metrics you really # should bump it up to e.g. 10MB or even higher. # buffer = 10485760
。。。。。。
4.selinux 配置
setenforce 0 sed -i ‘s/SELINUX=enforcing/SELINUX=disabled/g‘ /etc/selinux/config
三、启动
systemctl restart httpd
systemctl restart gmetad
systemctl restart gmond
浏览器访问:http://IP/ganglia/
若出现 You don‘t have permission to access /ganglia/ on this server,可以修改/var/lib/ganglia 目录的权限。
sudo chmod -R 0777 /var/lib/ganglia
四、操作 Flume 测试监控
修改 flume-env.sh,添加 JAVA_OPTS
vim /opt/apache-flume-1.9.0-bin/conf/flume-env.sh
JAVA_OPTS="-Dflume.monitoring.type=ganglia -Dflume.monitoring.hosts=192.168.8.136:8649 -Xms100m -Xmx200m"
测试任务文件
flume-netcat-logger.conf
# Name the components on this agent a1.sources = r1 a1.sinks = k1 a1.channels = c1 # Describe/configure the source a1.sources.r1.type = netcat a1.sources.r1.bind = 127.0.0.1 a1.sources.r1.port = 4444 # Describe the sink a1.sinks.k1.type = logger # Use a channel which buffers events in memory a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 # Bind the source and sink to the channel a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1
启动 Flume 任务
cd /opt/apache-flume-1.9.0-bin bin/flume-ng agent --conf conf/ --name a1 --conf-file /tmp/flume-job/flume-netcat-logger.conf -Dflume.root.logger==INFO,console -Dflume.monitoring.type=ganglia -Dflume.monitoring.hosts=192.168.8.136:8649 nc 127.0.0.1 4444 123
查看 Ganglia
图例说明:
字段(图表名称) | 字段含义 |
---|---|
EventPutAttemptCount | source 尝试写入 channel 的事件总数量 |
EventPutSuccessCount | 成功写入 channel 且提交的事件总数量 |
EventTakeAttemptCount | sink 尝试从 channel 拉取事件的总数量 |
EventTakeSuccessCount | sink 成功读取的事件的总数量 |
StartTime | channel 启动的时间(毫秒) |
StopTime | channel 停止的时间(毫秒) |
ChannelSize | 目前 channel 中事件的总数量 |
ChannelFillPercentage | channel 占用百分比 |
ChannelCapacity | channel 的容量 |
以上是关于Flume-数据流监控 Ganglia的主要内容,如果未能解决你的问题,请参考以下文章