Apache Pig如何与Apache Lucene集成？

Posted 2021-04-24 我是攻城师

Java代码

1. ---注册依赖相关的包

2. REGISTER /home/search/nsconvent/spig/20150112/lucenepig/pigudf.jar;

3. REGISTER /home/search/nsconvent/spig/20150112/lucenepig/lucene-analyzers-common-4.10.2.jar;

4. REGISTER /home/search/nsconvent/spig/20150112/lucenepig/lucene-core-4.10.2.jar;

5. REGISTER /home/search/nsconvent/spig/20150112/lucenepig/lucene-queryparser-4.10.2.jar;

7. --声明别名引用（注意只能无参的引用，带参数的在反射时候，会出现异常）

8. DEFINE LuceneStore com.pig.support.lucene.LuceneStore;

9. --加载数据

10. a = load '/tmp/data/20150303/tt.txt' using PigStorage(',') as (lbl:chararray,desc:chararray,score:int); ;

11. --生成索引并存储在HDFS上，注意需要配置简单lucene索引方式（是否存储？是否索引？）

12. store a into '/tmp/data/20150303/luceneindex' using LuceneStore('store[true]:tokenize[true]');