hive+postgres安装部署过程

Posted 2020-09-30

master节点安装元数据库，采用postgres:
#useradd postgres
#password postgres
su - postgres
wget https://ftp.postgresql.org/pub/source/v10beta2/postgresql-10beta2.tar.gz
tar zxvf postgresql-10beta2.tar.gz
cd postgresql-10beta2

./configure
make
su
make install

mkdir /usr/local/pgsql/data
chown postgres /usr/local/pgsql/data
su - postgres
/usr/local/pgsql/bin/initdb -D /usr/local/pgsql/data
/usr/local/pgsql/bin/postgres -D /usr/local/pgsql/data >logfile 2>&1 &
/usr/local/pgsql/bin/createdb test
/usr/local/pgsql/bin/psql test
----------------------------------------------------------------------------------------
#启动数据库方法 /usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data -l logfile start
#停止数据库的方法 /usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data -l logfile stop
----------------------------------------------------------------------------------------
安装完毕后配置
--以管理员身份登入PG：
psql postgres -U postgres

--创建用户hive_user:
Create user hive_user;

--创建DB metastore_db，owner为hive_user:
Create database metastore_db with owner=hive_user;

--设置hive_user的密码：
\password hive_user
安装hive:
tar zxvf apache-hive-2.3.0-bin.tar.gz
vi .bash_profile
export HADOOP_HOME=$HOME/hadoop-3.0.0-alpha4
export HIVE_HOME=$HOME/apache-hive-2.3.0-bin
PATH=$PATH:$HOME/.local/bin:$HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HIVE_HOME/bin
. .bash_profile
[[email protected] ~]$ $HADOOP_HOME/bin/hadoop fs -mkdir /tmp
[[email protected] ~]$ $HADOOP_HOME/bin/hadoop fs -mkdir -p /user/hive/warehouse
[[email protected] ~]$ $HADOOP_HOME/bin/hadoop fs -chmod g+w /tmp
[[email protected] ~]$ $HADOOP_HOME/bin/hadoop fs -chmod g+w /user/hive/warehouse

 

 

cd apache-hive-2.3.0-bin
[[email protected] apache-hive-2.3.0-bin]$ ls
bin binary-package-licenses conf examples hcatalog jdbc lib LICENSE NOTICE RELEASE_NOTES.txt scripts
[[email protected] apache-hive-2.3.0-bin]$ cd conf
[[email protected] conf]$ cp hive-env.sh.template hive-env.sh
[[email protected] conf]$ cp hive-default.xml.template hive-site.xml
vi hive-env.sh
HADOOP_HOME
vi hive-site.xml
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:postgresql://master:5432/metastore_db?</value>
jdbc:postgresql://master:5432/metastore_db?
<name>javax.jdo.option.ConnectionDriverName</name>
<value>org.postgresql.Driver</value>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive_user</value>
<name>javax.jdo.option.ConnectionPassword</name>
<value>111111</value>

hive.metastore.warehouse.dir：（HDFS上的）数据目录
hive.exec.scratchdir：（HDFS上的）临时文件目录
hive.metastore.warehouse.dir默认值是/user/hive/warehouse
hive.exec.scratchdir默认值是/tmp/hive-${user.name}
以上是默认值，暂时不改。
[[email protected] lib]$ pwd
/home/hd/apache-hive-2.3.0-bin/lib
[[email protected] lib]$ wget https://jdbc.postgresql.org/download/postgresql-42.1.3.jar

[[email protected] ~]$ $HIVE_HOME/bin/schematool -dbType postgres -initSchema
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hd/apache-hive-2.3.0-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hd/hadoop-3.0.0-alpha4/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:postgresql://master:5432/metastore_db?
Metastore Connection Driver : org.postgresql.Driver
Metastore connection User: hive_user
Starting metastore schema initialization to 2.3.0
Initialization script hive-schema-2.3.0.postgres.sql
Initialization script completed
schemaTool completed

[[email protected] ~]$ hive
which: no hbase in (/usr/java/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/hd/.local/bin:/home/hd/bin:/home/hd/hadoop-3.0.0-alpha4/bin:/home/hd/hadoop-3.0.0-alpha4/sbin:apache-hive-2.3.0-bin/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hd/apache-hive-2.3.0-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hd/hadoop-3.0.0-alpha4/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/home/hd/apache-hive-2.3.0-bin/lib/hive-common-2.3.0.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. hive-on-spark
hive> show tables;
OK
Time taken: 6.186 seconds
hive> select * from test;
FAILED: SemanticException [Error 10001]: Line 1:14 Table not found ‘test‘
hive> create table tt (a char(3),b char(4));
OK
Time taken: 0.737 seconds
hive> show tables;
OK
tt
Time taken: 0.078 seconds, Fetched: 1 row(s)
hive>