部署Greenplum数据库6.13.0单机版重启实录（5次报错问题及解决避坑指南）

Posted 2021-09-16 シ風箏

GP数据库安装后正常运行，服务器被恶意连接故修改密码后重启，GP数据库也需要重新启动，这个重新启动的过程坑坑洼洼 🐾 打印的日志信息进行了处理，删除了一些冗余。

5次报错及解决

第 1️⃣ 次报错

# 服务器重启后 启动GP
[gpadmin@tcloud /]$ /usr/local/greenplum-db/bin/gpstart
Traceback (most recent call last):
  File "./bin/gpstart", line 12, in <module>
    from gppylib.mainUtils import *
ImportError: No module named gppylib.mainUtils

第 2️⃣ 次报错

第1次报错的原因是 greenplum_path.sh 配置文件未生效。

# 生效 greenplum_path.sh后 启动GP
[gpadmin@tcloud /]$ source /usr/local/greenplum-db/greenplum_path.sh
[gpadmin@tcloud /]$ /usr/local/greenplum-db/bin/gpstart
20210901:09:16:59:028877 gpstart:tcloud:gpadmin-[INFO]:
-Starting gpstart with args:
-Gathering information and validating the environment...
20210901:09:16:59:028877 gpstart:tcloud:gpadmin-[CRITICAL]:
-gpstart failed. (Reason='Environment Variable MASTER_DATA_DIRECTORY not set!') exiting...

第 3️⃣ 次报错

第2次报错的原因是 .bashrc 配置文件未生效。

# 生效.bashrc后 启动GP
[gpadmin@tcloud /]$ source /home/greenplum/.bashrc
[gpadmin@tcloud /]$ /usr/local/greenplum-db/bin/gpstart
20210901:09:23:19:030476 gpstart:tcloud:gpadmin-[INFO]:
-Starting gpstart with args:
-Gathering information and validating the environment...
20210901:09:23:19:030476 gpstart:tcloud:gpadmin-[CRITICAL]:
-gpstart failed. (Reason='[Errno 2] No such file or directory: 
'/home/greenplum/gpdata/master/postgresql.conf'') exiting...

第 4️⃣ 次报错

第3次报错的原因是 MASTER_DATA_DIRECTORY 配置错误，重新配置文件夹位置。

# 修改.bash_profile文件
[gpadmin@tcloud /]$ vim /home/gpadmin/.bash_profile
source /usr/local/greenplum-db/greenplum_path.sh
export MASTER_DATA_DIRECTORY=/home/greenplum/gpdata/master/gpseg-1

# 生效.bash_profile后 启动GP
[gpadmin@tcloud /]$ source /home/gpadmin/.bash_profile
[gpadmin@tcloud /]$ /usr/local/greenplum-db/bin/gpstart
20210901:10:04:58:007993 gpstart:tcloud:gpadmin-[INFO]:
-Starting gpstart with args:
-Gathering information and validating the environment...
-Greenplum Binary Version: 'postgres (Greenplum Database) 6.13.0 build 
commit:4f1adf8e247a9685c19ea02bcaddfdc200937ecd Open Source'
-Greenplum Catalog Version: '301908232'
20210901:10:04:58:007993 gpstart:tcloud:gpadmin-[WARNING]:
-postmaster.pid file exists on Master, checking if recovery startup required
20210901:10:04:58:007993 gpstart:tcloud:gpadmin-[INFO]:
-Commencing recovery startup checks
20210901:10:04:58:007993 gpstart:tcloud:gpadmin-[CRITICAL]:
-Command netstat not found
-gpstart failed. (Reason='Could not locate command: 'netstat' in this set of paths: 
['/usr/kerberos/bin', '/usr/sfw/bin', '/opt/sfw/bin', '/bin', '/usr/local/bin', '/usr/bin', 
'/sbin', '/usr/sbin', '/usr/ucb', '/sw/bin', '/opt/Navisphere/bin', '/usr/local/greenplum-db-
6.13.0']') exiting...

第 5️⃣ 次报错

第4次报错的原因是缺少 net-tools 切换回root用户安装它，我极度疑惑第一次是如何启动成功的 ❔

# 切换回root用户 安装 net-tools 后用gpadmin用户 启动GP
[root@tcloud ~]# yum install net-tools
[gpadmin@tcloud root]$ /usr/local/greenplum-db/bin/gpstart
20210901:10:15:32:010546 gpstart:tcloud:gpadmin-[INFO]:
-Starting gpstart with args:
-Gathering information and validating the environment...
-Greenplum Binary Version: 'postgres (Greenplum Database) 6.13.0 build 
commit:4f1adf8e247a9685c19ea02bcaddfdc200937ecd Open Source'
-Greenplum Catalog Version: '301908232'
20210901:10:15:32:010546 gpstart:tcloud:gpadmin-[WARNING]:
-postmaster.pid file exists on Master, checking if recovery startup required
20210901:10:15:32:010546 gpstart:tcloud:gpadmin-[INFO]:
-Commencing recovery startup checks
-No socket connection or lock file in /tmp found for port=2345
-No Master instance process, entering recovery startup mode
-Clearing Master instance pid file
-Starting Master instance in admin mode
-Obtaining Greenplum Master catalog information
-Obtaining Segment details from master...
-Setting new master era
-Commencing forced instance shutdown
-Starting Master instance in admin mode
-Obtaining Greenplum Master catalog information
-Obtaining Segment details from master...
-Setting new master era
-Master Started...
20210901:10:15:34:010546 gpstart:tcloud:gpadmin-[WARNING]:
-One or more hosts are not reachable via SSH.  
Any segments on those hosts will be marked down
-Host tcloud is unreachable
-Marking segment 2 down because tcloud is unreachable
20210901:10:15:34:010546 gpstart:tcloud:gpadmin-[CRITICAL]:
-gpstart failed. 
(Reason=''NoneType' object has no attribute 'getSegmentHostName'') exiting...

这个时候用Navicat连接数据库时报 System was started in master-only utility mode - only utility mode connections are allowed 实际上是启动了 master 但是 由于 Host tcloud is unreachable 导致 segment 数据没能读取。

🔥 第 6️⃣ 次成功 🔥

第5次报错的信息是 One or more hosts are not reachable via SSH 解决这个问题。

# 运行 gpssh-exkeys -f /home/greenplum/hostfile_exkeys
[gpadmin@tcloud root]$ gpssh-exkeys -f /home/greenplum/hostfile_exkeys
[STEP 1 of 5] create local ID and authorize on local host
  ... /home/gpadmin/.ssh/id_rsa file exists ... key generation skipped
[STEP 2 of 5] keyscan all hosts and update known_hosts file
[STEP 3 of 5] retrieving credentials from remote hosts
[STEP 4 of 5] determine common authentication file content
[STEP 5 of 5] copy authentication files to all remote hosts
[INFO] completed successfully

# 启动GP
[gpadmin@tcloud root]$ /usr/local/greenplum-db/bin/gpstart

20210901:10:30:11:014617 gpstart:tcloud:gpadmin-[INFO]:
-Starting gpstart with args:
-Gathering information and validating the environment...
-Greenplum Binary Version: 'postgres (Greenplum Database) 6.13.0 build 
commit:4f1adf8e247a9685c19ea02bcaddfdc200937ecd Open Source'
-Greenplum Catalog Version: '301908232'
-Starting Master instance in admin mode
-Obtaining Greenplum Master catalog information
-Obtaining Segment details from master...
-Setting new master era
-Master Started...
Warning: the RSA host key for 'tcloud' differs from the key for the IP address 'xxx.xx.x.x'
Offending key for IP in /home/gpadmin/.ssh/known_hosts:2
Matching host key in /home/gpadmin/.ssh/known_hosts:1
Are you sure you want to continue connecting (yes/no)? yes

20210901:10:30:20:014617 gpstart:tcloud:gpadmin-[INFO]:
-Shutting down master
---------------------------
-Master instance parameters
---------------------------
-Database                 = template1
-Master Port              = 2345
-Master directory         = /home/greenplum/gpdata/master/gpseg-1
-Timeout                  = 600 seconds
-Master standby           = Off
---------------------------------------
-Segment instances that will be started
---------------------------------------
-   Host     Datadir                                 Port
-   tcloud   /home/greenplum/gpdata/gpnode1/gpseg0   6000
-   tcloud   /home/greenplum/gpdata/gpnode2/gpseg1   6001

Continue with Greenplum instance startup Yy|Nn (default=N):
> y

20210901:10:30:25:014617 gpstart:tcloud:gpadmin-[INFO]:
-Commencing parallel segment instance startup, please wait...
.
-Process results...
-----------------------------------------------------
-   Successful segment starts                                            = 2
-   Failed segment starts                                                = 0
-   Skipped segment starts (segments are marked down in configuration)   = 0
-----------------------------------------------------
-Successfully started 2 of 2 segment instances
-----------------------------------------------------
-Starting Master instance tcloud directory /home/greenplum/gpdata/master/gpseg-1
-Command pg_ctl reports Master tcloud instance active
-Connecting to dbname='template1' connect_timeout=15
-No standby master configured.  skipping...
-Database successfully started