GreenPlum 大数据平台--安装
Posted kingle66
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了GreenPlum 大数据平台--安装相关的知识，希望对你有一定的参考价值。
1. 环境准备
01, 安装包准备:
Greenplum : >>>>链接地址
Pgadmin客户端 : >>>链接地址
greenplum-cc-web监控: >>>>链接地址
02,节点说明
| 服务器ip | 主机名 | 角色 | 系统版本 |
| 192.168.0.221 | greenplum01 | master | centos7 |
| 192.168.0.222 | greenplum02 | Segment/standby | centos7 |
| 192.168.0.223 | greenplum03 | Segment | centos7 |
03,host文件配置(All)
echo -e "192.168.0.221 greenplum01\\n192.168.0.222 greenplum02\\n192.168.0.223 greenplum03" >>/etc/hosts scp /etc/hosts 192.168.0.222:/etc/hosts scp /etc/hosts 192.168.0.223:/etc/hosts
04,创建用户(ALL)
[root@greenplum01 ~]# groupadd -g 530 gpadmin [root@greenplum01 ~]# useradd -g 530 -m -d /home/gpadmin -s /bin/bash gpadmin [root@greenplum01 ~]# passwd gpadmin Changing password for user gpadmin. New password: BAD PASSWORD: The password is shorter than 8 characters Retype new password: passwd: all authentication tokens updated successfully. [root@greenplum02 ~]# groupadd -g 530 gpadmin [root@greenplum02 ~]# useradd -g 530 -m -d /home/gpadmin -s /bin/bash gpadmin [root@greenplum02 ~]# passwd gpadmin Changing password for user gpadmin. New password: BAD PASSWORD: The password is shorter than 8 characters Retype new password: passwd: all authentication tokens updated successfully. [root@greenplum03 ~]# groupadd -g 530 gpadmin [root@greenplum03 ~]# useradd -g 530 -m -d /home/gpadmin -s /bin/bash gpadmin [root@greenplum03 ~]# passwd gpadmin Changing password for user gpadmin. New password: BAD PASSWORD: The password is shorter than 8 characters Retype new password: passwd: all authentication tokens updated successfully.
05,内核参数修改(ALL)
echo \' kernel.shmmax = 500000000 kernel.shmmni = 4096 kernel.shmall = 4000000000 kernel.sem = 250 512000 100 2048 kernel.sysrq = 1 kernel.core_uses_pid = 1 kernel.msgmnb = 65536 kernel.msgmax = 65536 kernel.msgmni = 2048 net.ipv4.tcp_syncookies = 1 net.ipv4.ip_forward = 0 net.ipv4.conf.default.accept_source_route = 0 net.ipv4.tcp_tw_recycle = 1 net.ipv4.tcp_max_syn_backlog = 4096 net.ipv4.conf.all.arp_filter = 1 net.ipv4.ip_local_port_range = 1025 65535 net.core.netdev_max_backlog = 10000 net.core.rmem_max = 2097152 net.core.wmem_max = 2097152 vm.overcommit_memory = 2 \'>>/etc/sysctl.conf
sysctl -
06.打开数限制(ALL)
echo \' * soft nofile 65536 * hard nofile 65536 * soft nproc 131072 * hard nproc 131072 \'>>/etc/security/limits.conf
07,关闭防火墙,selinux(ALL)
systemctl disable firewalld.service systemctl stop firewalld.service setenforce 0
08,创建需要的目录(ALL)
mkdir /greenplum
chown -R gpadmin:gpadmin /greenplum
二,安装greenplum(master节点)
01,安装
[root@greenplum01 ~]# ls anaconda-ks.cfg greenplum-cc-web-4.6.1-LINUX-x86_64 initial-setup-ks.cfg Templates Desktop greenplum-cc-web-4.6.1-LINUX-x86_64.zip Music Videos Documents greenplum-db-5.16.0-rhel7-x86_64.rpm Pictures Downloads Public [root@greenplum01 ~]# rpm -ivh greenplum-db-5.16.0-rhel7-x86_64.rpm Preparing... ################################# [100%] Updating / installing... 1:greenplum-db-5.16.0-1 ################################# [100%]
02,配置greeplum
chown -R gpadmin:gpadmin /usr/local/ su - gpadmin mkdir -p /home/gpadmin/conf vi /home/gpadmin/conf/hostlist greenplum01 greenplum02 greenplum03
再创建一个seg_hosts ，包含所有的Segment Host的主机名：
vi /home/gpadmin/conf/seg_hosts
greenplum02
greenplum03
03,配置链接
[gpadmin@greenplum01 ~]$ source /usr/local/greenplum-db/greenplum_path.sh [gpadmin@greenplum01 ~]$ gpssh-exkeys -f /home/gpadmin/conf/hostlist [STEP 1 of 5] create local ID and authorize on local host [STEP 2 of 5] keyscan all hosts and update known_hosts file [STEP 3 of 5] authorize current user on remote hosts ... send to greenplum02 *** *** Enter password for greenplum02: ... send to greenplum03 [STEP 4 of 5] determine common authentication file content [STEP 5 of 5] copy authentication files to all remote hosts ... finished key exchange with greenplum02 ... finished key exchange with greenplum03 [INFO] completed successfully
验证链接:
[gpadmin@greenplum01 ~]$ ssh greenplum02 [gpadmin@greenplum02 ~]$ exit logout Connection to greenplum02 clsed. [gpadmin@greenplum01 ~]$ ssh greenplum03 [gpadmin@greenplum02 ~]$ exit logout Connection to greenplum03 clsed.
三,安装其他节点的DB
01,权限授予
chown -R gpadmin:gpadmin /usr/local chown -R gpadmin:gpadmin /greenplum
02,数据打包
cd /usr/local
tar zcvf gp.tar greenplum-db-5.16.0/ gpscp -f /home/gpadmin/conf/seg_hosts gp.tar =:/usr/local/
03,解压缩
[gpadmin@greenplum01 ~]$ source /usr/local/greenplum-db/greenplum_path.sh [gpadmin@greenplum01 ~]$ gpssh -f /home/gpadmin/conf/seg_hosts => cd /usr/local [greenplum03] [greenplum02] => tar -xf gp.tar [greenplum03] [greenplum02] => ln -s ./greenplum-db-5.16.0/ greenplum-db [greenplum03] [greenplum02] => ll [greenplum03] total 271720 [greenplum03] drwxr-xr-x. 2 gpadmin gpadmin 6 Nov 5 2016 bin [greenplum03] drwxr-xr-x. 2 gpadmin gpadmin 6 Nov 5 2016 etc [greenplum03] drwxr-xr-x. 2 gpadmin gpadmin 6 Nov 5 2016 games [greenplum03] -rw-rw-r-- 1 gpadmin gpadmin 278239886 Mar 12 23:18 gp.tar [greenplum03] lrwxrwxrwx 1 gpadmin gpadmin 22 Mar 12 23:21 greenplum-db -> ./greenplum-db-5.16.0/ [greenplum03] drwxr-xr-x 11 gpadmin gpadmin 138 Mar 12 23:03 greenplum-db-5.16.0 [greenplum03] drwxr-xr-x. 2 gpadmin gpadmin 6 Nov 5 2016 include [greenplum03] drwxr-xr-x. 2 gpadmin gpadmin 6 Nov 5 2016 lib [greenplum03] drwxr-xr-x. 2 gpadmin gpadmin 6 Nov 5 2016 lib64 [greenplum03] drwxr-xr-x. 2 gpadmin gpadmin 6 Nov 5 2016 libexec [greenplum03] drwxr-xr-x. 2 gpadmin gpadmin 6 Nov 5 2016 sbin [greenplum03] drwxr-xr-x. 5 gpadmin gpadmin 49 Feb 18 22:35 share [greenplum03] drwxr-xr-x. 2 gpadmin gpadmin 6 Nov 5 2016 src [greenplum03] drwxrwxr-x 3 gpadmin gpadmin 19 Mar 12 23:21 usr [greenplum02] total 271720 [greenplum02] drwxr-xr-x. 2 gpadmin gpadmin 6 Nov 5 2016 bin [greenplum02] drwxr-xr-x. 2 gpadmin gpadmin 6 Nov 5 2016 etc [greenplum02] drwxr-xr-x. 2 gpadmin gpadmin 6 Nov 5 2016 games [greenplum02] -rw-rw-r-- 1 gpadmin gpadmin 278239886 Mar 12 23:18 gp.tar [greenplum02] lrwxrwxrwx 1 gpadmin gpadmin 22 Mar 12 23:21 greenplum-db -> ./greenplum-db-5.16.0/ [greenplum02] drwxr-xr-x 11 gpadmin gpadmin 138 Mar 12 23:03 greenplum-db-5.16.0 [greenplum02] drwxr-xr-x. 2 gpadmin gpadmin 6 Nov 5 2016 include [greenplum02] drwxr-xr-x. 2 gpadmin gpadmin 6 Nov 5 2016 lib [greenplum02] drwxr-xr-x. 2 gpadmin gpadmin 6 Nov 5 2016 lib64 [greenplum02] drwxr-xr-x. 2 gpadmin gpadmin 6 Nov 5 2016 libexec [greenplum02] drwxr-xr-x. 2 gpadmin gpadmin 6 Nov 5 2016 sbin [greenplum02] drwxr-xr-x. 5 gpadmin gpadmin 49 Feb 18 22:35 share [greenplum02] drwxr-xr-x. 2 gpadmin gpadmin 6 Nov 5 2016 src [greenplum02] drwxrwxr-x 3 gpadmin gpadmin 19 Mar 12 23:21 usr
四,初始化数据库
01,创建需要的资源目录
[gpadmin@greenplum01 ~]$ source /usr/local/greenplum-db/greenplum_path.sh [gpadmin@greenplum01 ~]$ gpssh -f /home/gpadmin/conf/hostlist => mkdir -p /greenplum/data/master [greenplum01] [greenplum02] [greenplum03] => mkdir -p /greenplum/data/primary [greenplum01] [greenplum02] [greenplum03] => mkdir -p /greenplum/data/mirror [greenplum01] [greenplum02] [greenplum03] => mkdir -p /greenplum/data2/primary [greenplum01] [greenplum02] [greenplum03] => mkdir -p /greenplum/data2/mirror [greenplum01] [greenplum02] [greenplum03] =>
02,节点配置环境(all)
[gpadmin@greenplum01 ~]$ vim .bash_profile source /usr/local/greenplum-db/greenplum_path.sh export MASTER_DATA_DIRECTORY=/greenplum/data/master/gpseg-1 export GPPORT=5432 export PGDATABASE=gp_db [gpadmin@greenplum01 ~]$ scp .bash_profile greenplum02:/home/gpadmin/ .bash_profile 100% 344 738.4KB/s 00:00 [gpadmin@greenplum01 ~]$ scp .bash_profile greenplum03:/home/gpadmin/ .bash_profile 100% 344 828.8KB/s 00:00 [gpadmin@greenplum01 ~]$ source .bash_profile
[gpadmin@greenplum02 ~]$ source .bash_profile
[gpadmin@greenplum03 ~]$ source .bash_profile
03,初始化检查
[gpadmin@greenplum01 ~]$ cd /usr/local/greenplum-db/bin/ [gpadmin@greenplum01 bin]$ gpcheckperf -f /home/gpadmin/conf/hostlist -r N -d /tmp /usr/local/greenplum-db/./bin/gpcheckperf -f /home/gpadmin/conf/hostlist -r N -d /tmp ------------------- -- NETPERF TEST ------------------- ==================== == RESULT ==================== Netperf bisection bandwidth test greenplum01 -> greenplum02 = 1654.370000 greenplum03 -> greenplum01 = 1156.950000 greenplum02 -> greenplum01 = 1183.980000 greenplum01 -> greenplum03 = 1363.620000 Summary: sum = 5358.92 MB/sec min = 1156.95 MB/sec max = 1654.37 MB/sec avg = 1339.73 MB/sec median = 1363.62 MB/sec
04,创建初始化文件
[gpadmin@greenplum01 conf]$ cp /usr/local/greenplum-db/docs/cli_help/gpconfigs/gpinitsystem_config ./initgp_config [gpadmin@greenplum01 conf]$ vim initgp_config [gpadmin@greenplum01 conf]$ grep -Ev \'^#|^$\' initgp_config ARRAY_NAME="Greenplum" SEG_PREFIX=gpseg PORT_BASE=6000 declare -a DATA_DIRECTORY=(/greenplum/data/primary /greenplum/data/primary /greenplum/data/primary /greenplum/data2/primary /greenplum/data2/primary /greenplum/data2/primary) declare -a MIRROR_DATA_DIRECTORY=(/greenplum/data/mirror /greenplum/data/mirror /greenplum/data/mirror /greenplum/data2/mirror /greenplum/data2/mirror /greenplum/data2/mirror) MASTER_HOSTNAME=greenplum01 MASTER_DIRECTORY=/greenplum/data/master MASTER_PORT=5432 TRUSTED_SHELL=ssh CHECK_POINT_SEGMENTS=8 ENCODING=UNICODE MIRROR_PORT_BASE=43000 REPLICATION_PORT_BASE=34000 MIRROR_REPLICATION_PORT_BASE=44000 MACHINE_LIST_FILE=/home/gpadmin/conf/seg_hosts
05,初始化
gpinitsystem -c /home/gpadmin/conf/initgp_config -s greenplum02
附上过程
1 [gpadmin@greenplum01 conf]$ gpinitsystem -c /home/gpadmin/conf/initgp_config -s greenplum02 2 20190313:01:49:23:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Checking configuration parameters, please wait... 3 20190313:01:49:23:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Reading Greenplum configuration file /home/gpadmin/conf/initgp_config 4 20190313:01:49:23:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Locale has not been set in /home/gpadmin/conf/initgp_config, will set to default value 5 20190313:01:49:23:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Locale set to en_US.utf8 6 20190313:01:49:23:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-No DATABASE_NAME set, will exit following template1 updates 7 20190313:01:49:23:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-MASTER_MAX_CONNECT not set, will set to default value 250 8 20190313:01:49:23:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Checking configuration parameters, Completed 9 20190313:01:49:23:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Commencing multi-home checks, please wait... 10 .. 11 20190313:01:49:24:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Configuring build for standard array 12 20190313:01:49:24:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Commencing multi-home checks, Completed 13 20190313:01:49:24:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Building primary segment instance array, please wait... 14 ............ 15 20190313:01:49:28:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Building group mirror array type , please wait... 16 ............ 17 20190313:01:49:32:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Checking Master host 18 20190313:01:49:32:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Checking new segment hosts, please wait... 19 ........................ 20 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Checking new segment hosts, Completed 21 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Greenplum Database Creation Parameters 22 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:--------------------------------------- 23 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Master Configuration 24 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:--------------------------------------- 25 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Master instance name = Greenplum 26 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Master hostname = greenplum01 27 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Master port = 5432 28 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Master instance dir = /greenplum/data/master/gpseg-1 29 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Master LOCALE = en_US.utf8 30 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Greenplum segment prefix = gpseg 31 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Master Database = 32 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Master connections = 250 33 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Master buffers = 128000kB 34 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Segment connections = 750 35 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Segment buffers = 128000kB 36 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Checkpoint segments = 8 37 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Encoding = UNICODE 38 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Postgres param file = Off 39 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Initdb to be used = /usr/local/greenplum-db/./bin/initdb 40 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-GP_LIBRARY_PATH is = /usr/local/greenplum-db/./lib 41 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-HEAP_CHECKSUM is = on 42 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-HBA_HOSTNAMES is = 0 43 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Ulimit check = Passed 44 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Array host connect type = Single hostname per node 45 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Master IP address [1] = ::1 46 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Master IP address [2] = 172.168.0.221 47 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Master IP address [3] = 192.168.0.221 48 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Master IP address [4] = fe80::58d2:b017:6803:188 49 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Master IP address [5] = fe80::5b46:68af:2341:babe 50 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Master IP address [6] = fe80::6b83:cde0:ac33:87bf 51 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Master IP address [7] = fe80::b210:3bd4:b0c0:2e17 52 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Master IP address [8] = fe80::c0ae:2a2:82a5:5f12 53 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Master IP address [9] = fe80::f144:3513:68ac:3356 54 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Standby Master = greenplum02 55 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Primary segment # = 6 56 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Standby IP address = ::1 57 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Standby IP address = 172.168.0.222 58 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Standby IP address = 192.168.0.222 59 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Standby IP address = fe80::58d2:b017:6803:188 60 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Standby IP address = fe80::5b46:68af:2341:babe 61 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Standby IP address = fe80::6b83:cde0:ac33:87bf 62 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Standby IP address = fe80::b210:3bd4:b0c0:2e17 63 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Standby IP address = fe80::c0ae:2a2:82a5:5f12 64 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Standby IP address = fe80::f144:3513:68ac:3356 65 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Total Database segments = 12 66 20190313:01:49:48:007228 gpinitsystem:greenplum01:gpadmin-[INFO]:-Trusted shell = ssh 67 20190313:01:超强数据源覆盖能力，永洪科技BI产品全面支持GreenplumGreenplum 实时数据仓库实践——Greenplum安装部署