supervisor管理hive服务(metastore,hiveserver2),使自启

Posted 2021-08-15 Mr.zhou_Zxy

使用supervisor管理hive服务

1.1 需求说明

			在做通过hudi采集数据并落地到hive的过程中，因为经常有Hive的服务被杀死，所以想到用supervisor来管理hive的进程，
当hive的服务被意外杀死后，实现自动启动

1.2 supervisor

管理进程的工具。

1.2.1 安装

##1. 安装
[root@hadoop prometheus-2.17.1]# yum -y install epel-release
[root@hadoop prometheus-2.17.1]# yum -y install supervisor

##2. 配置
[root@hadoop prometheus-2.17.1]# mv /etc/supervisord.conf /etc/supervisord.conf.bak

1.2.2 配置

[root@hadoop prometheus-2.17.1]# vi /etc/supervisord.conf

; filename:supervisord.conf
; author:zxy
; date:2021-07-18

[unix_http_server]
file=/var/run/supervisor/supervisor.sock   ; (the path to the socket file)
;chmod=0700                 ; sockef file mode (default 0700)
;chown=nobody:nogroup       ; socket file uid:gid owner
;username=user              ; (default is no username (open server))
;password=123               ; (default is no password (open server))

[inet_http_server]         ; inet (TCP) server disabled by default
port=0.0.0.0:9001        ; (ip_address:port specifier, *:port for all iface)
;username=user              ; (default is no username (open server))
;password=123               ; (default is no password (open server))

[supervisord]
logfile=/var/log/supervisor/supervisord.log  ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB       ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10          ; (num of main logfile rotation backups;default 10)
loglevel=info               ; (log level;default info; others: debug,warn,trace)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false              ; (start in foreground if true;default false)
minfds=1024                 ; (min. avail startup file descriptors;default 1024)
minprocs=1024                ; (min. avail process descriptors;default 200)
user=root

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///var/run/supervisor/supervisor.sock ; use a unix:// URL  for a unix socket
serverurl=http://0.0.0.0:9001 ; use an http:// url to specify an inet socket
;username=chris              ; should be same as http_username if set
;password=123                ; should be same as http_password if set

[include]
files = /etc/supervisord.d/*.conf

1.2.3 启动supervisord

[root@hadoop prometheus-2.17.1]# touch /var/run/supervisor/supervisord.pid
/etc/supervisord.d/ : 存放supervisor的配置文件
/var/run/supervisor/: 存放pid的进程
/var/log/supervisor/: 存放supervisor的日志文件

##2. 设置开机自启动
[root@hadoop supervisor]# systemctl enable supervisord
##3. 开启服务器
[root@hadoop supervisor]# systemctl start supervisord
[root@hadoop supervisor]# systemctl status supervisord
● supervisord.service - Process Monitoring and Control Daemon
   Loaded: loaded (/usr/lib/systemd/system/supervisord.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-05-27 14:50:01 CST; 12s ago
  Process: 16460 ExecStart=/usr/bin/supervisord -c /etc/supervisord.conf (code=exited, status=0/SUCCESS)
 Main PID: 16481 (supervisord)
   CGroup: /system.slice/supervisord.service
           └─16481 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf

May 27 14:50:01 hadoop systemd[1]: Starting Process Monitoring and Control Daemon...
May 27 14:50:01 hadoop systemd[1]: Started Process Monitoring and Control Daemon.

1.3 supervisor管理进程

hive_metastore.conf

[root@hadoop supervisord.d]# vim hive_metastore.conf

; filename:hive_metastore.conf
; author:zxy
; date:2021-07-18
; desc:配置supervisor管理metastore

[program:metastore]
directory=/opt/apps/hive-1.2.1   ;软件的前缀
command=/opt/apps/hive-1.2.1/bin/hive --service metastore &       ; 配置启动Prometheus的命令
stderr_logfile=/var/log/supervisor/hivemetastore.err       ; 错误日志存放路径
stdout_logfile=/var/log/supervisor/hivemetastore.log         ; 标准的日志存放路径
stdout_logfile_maxbytes=10MB   ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=10     ; # of stdout logfile backups (default 10)
user=root ; 只能root用户启动
autostart=true ; 自动启动
autorestart=true        ; 自动重启
startsecs=10                  ; 开机10s之后如果应用程序是running状态，supervisor就认为这个启动是成功
startretries=3                ; 重启3此之后还是没有成功，就标记为失败
redirect_stderr=false ;    如果是true，那么stderr日志就会写入到stdout中

hive_hiveserver2.conf

[root@hadoop supervisord.d]# vim hive_hiveserver2.conf

; filename:hive_hiveserver2.conf
; author:zxy
; date:2021-07-18
; desc:配置supervisor管理hiveserver2

[program:hiveserver]
directory=/opt/apps/hive-1.2.1   ;软件的前缀
command=/opt/apps/hive-1.2.1/bin/hive --service hiveserver2 &       ; 配置启动Prometheus的命令
stderr_logfile=/var/log/supervisor/hiveserver.err       ; 错误日志存放路径
stdout_logfile=/var/log/supervisor/hiveserver.log         ; 标准的日志存放路径
stdout_logfile_maxbytes=10MB   ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=10     ; # of stdout logfile backups (default 10)
user=root ; 只能root用户启动
autostart=true ; 自动启动
autorestart=true        ; 自动重启
startsecs=10                  ; 开机10s之后如果应用程序是running状态，supervisor就认为这个启动是成功
startretries=3                ; 重启3此之后还是没有成功，就标记为失败
redirect_stderr=false ;    如果是true，那么stderr日志就会写入到stdout中

1.4通过supervisor开启/关闭/重启hive服务

##1. 查看supervisor下的所有的应用程序的状态
[root@hadoop azkaban-solo-server]# supervisorctl reread
prometheus: available

##2. 更新supervisor中的指定程序
[root@hadoop azkaban-solo-server]# supervisorctl update prometheus
prometheus: added process group

##3. 查看指定程序的状态
[root@hadoop azkaban-solo-server]# supervisorctl status prometheus
prometheus                       RUNNING   pid 25360, uptime 0:01:05

##4. 关闭指定的程序
[root@hadoop azkaban-solo-server]# supervisorctl stop prometheus
prometheus: stopped

##5. 开启指定的程序
[root@hadoop azkaban-solo-server]# supervisorctl start prometheus
prometheus: stopped

1.4.1 supervisorctl 启动hive的进程

[root@hadoop supervisord.d]# supervisorctl start metastore
[root@hadoop supervisord.d]# supervisorctl start hiveserver
[root@hadoop supervisord.d]# jps
3361 NodeManager
2456 NameNode
2601 DataNode
48601 RunJar
3242 ResourceManager
48602 RunJar
2845 SecondaryNameNode
48831 Jps
[root@hadoop supervisord.d]# hive

Logging initialized using configuration in file:/data/apps/hive-1.2.1/conf/hive-log4j.properties
hive (default)>