Presto的坑记录

Posted 2022-12-02 北京小辉

Presto的坑记录

• 问题1 Java版本:
• 问题2 Hive连接:
• 问题3 单机没有work:
• 问题4 ： java.net.UnknownHostException: ns2
• 问题5 ： Unable to create input format org.apache.hadoop.mapred.TextInputFormat

问题1 Java版本:

Unrecognized VM option 'ExitOnOutOfMemoryError'
Did you mean 'OnOutOfMemoryError=<value>'?
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

回答：
安装要求如下：

Mac OS X or Linux
Java 8 Update 151 or higher (8u151+), 64-bit. Both Oracle JDK and OpenJDK are supported.
Maven 3.3.9+ (for building)
Python 2.4+ (for running with the launcher script)

bin/launcher run
ERROR: [Errno 2] No such file or directory

解决：
1、看看java版本是否有，java -version。
2、用root用户启动 bin/launcher run

问题2 Hive连接:

2020-09-15T14:56:32.783+0800	ERROR	main	com.facebook.presto.server.PrestoServer	No factory for connector xxx.xxx.xxx.xxx
java.lang.IllegalArgumentException: No factory for connector xxx.xxx.xxx.xxx
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:216)
	at com.facebook.presto.connector.ConnectorManager.createConnection(ConnectorManager.java:203)
	at com.facebook.presto.metadata.StaticCatalogStore.loadCatalog(StaticCatalogStore.java:123)
	at com.facebook.presto.metadata.StaticCatalogStore.loadCatalog(StaticCatalogStore.java:98)
	at com.facebook.presto.metadata.StaticCatalogStore.loadCatalogs(StaticCatalogStore.java:80)
	at com.facebook.presto.metadata.StaticCatalogStore.loadCatalogs(StaticCatalogStore.java:68)
	at com.facebook.presto.server.PrestoServer.run(PrestoServer.java:135)
	at com.facebook.presto.server.PrestoServer.main(PrestoServer.java:77)

解决：
在hive.properties中 connector.name写错了，应该为指定的版本，以便于presto使用对应的适配器，修改为：
connector.name=hive-hadoop2

[root@slave2 presto-server-0.240]# cat etc/catalog/hive.properties
connector.name=hive-hadoop2
hive.metastore.uri=thrift://xxx.xxx.xxx.xxx:9083 # Hive的地址

问题3 单机没有work:

自己的角色既是coordinator 也是work

coordinator=true
node-scheduler.include-coordinator=true  // 这里设置为true
http-server.http.port=8080
query.max-memory=50GB
query.max-memory-per-node=1GB
query.max-total-memory-per-node=2GB
discovery-server.enabled=true
discovery.uri=http://localhost:8080

问题4 ： java.net.UnknownHostException: ns2

presto:default> select * from test;

Query 20200916_090315_00014_xm34z, FAILED, 1 node
Splits: 16 total, 0 done (0.00%)
0:00 [0 rows, 0B] [0 rows/s, 0B/s]

Query 20200916_090315_00014_xm34z failed: java.net.UnknownHostException: ns2

在【 hive.config.resources】和【hive.properties】两个文件都要加入下面的配置

hive.config.resources=/data/data_big/webserver/presto/hdfs-site.xml,/data/data_big/webserver/presto/core-site.xml

问题5 ： Unable to create input format org.apache.hadoop.mapred.TextInputFormat

解决：
将集群中的hadoop依赖包拷贝到/var/presto/data/plugin/hive-hadoop2.
查找包命令：
find /usr/hdp/2.5.0.0-1245/hadoop -name lib_name*
包列表如下：

-rw-r--r-- 1 yuhui yuhui 3607775 Sep 27 17:14 hadoop-common-2.7.3.2.5.0.0-1245.jar
-rw-r--r-- 1 yuhui yuhui 8517435 Sep 27 17:15 hadoop-hdfs-2.7.3.2.5.0.0-1245.jar
-rw-r--r-- 1 yuhui yuhui 4093994 Sep 27 17:19 hadoop-hdfs-client-2.8.5.jar
-rw-r--r-- 1 yuhui yuhui   72076 Sep 27 17:12 hadoop-lzo-0.6.0.2.5.0.0-1245.jar
-rw-r--r-- 1 yuhui yuhui  516260 Sep 27 17:21 hadoop-mapreduce-client-app-2.7.3.2.5.0.0-1245.jar
-rw-r--r-- 1 yuhui yuhui  751958 Sep 27 17:21 hadoop-mapreduce-client-common-2.7.3.2.5.0.0-1245.jar
-rw-r--r-- 1 yuhui yuhui   38551 Sep 27 17:22 hadoop-mapreduce-client-jobclient-2.7.3.2.5.0.0-1245.jar
-rw-r--r-- 1 yuhui yuhui   54075 Sep 27 17:22 hadoop-mapreduce-client-shuffle-2.7.3.2.5.0.0-1245.jar
-rw-r--r-- 1 yuhui yuhui  105895 Sep 27 17:23 hadoop-streaming-2.7.3.2.5.0.0-1245.jar