CSV 将大量数据加载到 Pig 中

Posted 2023-04-18

【中文标题】CSV 将大量数据加载到 Pig 中

【英文标题】：CSV loading large amount of data into Pig

【发布时间】：2016-03-26 21:45:26

【问题描述】：

我在 pig 中使用此查询从包含 50000 条记录的 CSV 文件中加载数据。

A = LOAD '/home/user/q2.csv' using org.apache.pig.piggybank.storage.CSVExcelStorage(',', 'YES_MULTILINE') as (Id:chararray,
PostTypeId:chararray, 
AcceptedAnswerId:chararray, 
ParentId:chararray, 
CreationDate:chararray, 
DeletionDate:chararray, 
Score:chararray, 
ViewCount:chararray, 
Body:chararray, 
OwnerUserId:chararray, 
OwnerDisplayName:chararray, 
LastEditorUserId:chararray, 
LastEditorDisplayName:chararray, 
LastEditDate:chararray, 
LastActivityDate:chararray, 
Title:chararray, 
Tags:chararray, 
AnswerCount:chararray, 
CommentCount:chararray, 
FavoriteCount:chararray, 
ClosedDate:chararray, 
CommunityOwnedDate:chararray);

这里是清除 \n & 的数据的查询，在 body 字段中等等。

Q2Clean = FOREACH Q2 GENERATE
Id as Id, 
PostTypeId as PostTypeId, 
AcceptedAnswerId as AcceptedAnswerId, 
(chararray)REPLACE(ParentId,'"','')  as ParentId, 
CreationDate as CreationDate, 
(chararray)REPLACE(DeletionDate,'"','') as DeletionDate, 
Score as Score, 
ViewCount as ViewCount,  
(chararray)REPLACE(REPLACE(Body,'\n',''),',','')as Body, 
OwnerUserId as OwnerUserId, 
(chararray)REPLACE(OwnerDisplayName,'"','') as OwnerDisplayName, 
LastEditorUserId as LastEditorUserId, 
(chararray)REPLACE(LastEditorDisplayName,'"','') as LastEditorDisplayName, 
LastEditDate as LastEditDate, 
LastActivityDate as LastActivityDate, 
(chararray)REPLACE(Title,',','') as Title, 
(chararray)REPLACE(Tags,',','') as Tags, 
AnswerCount as AnswerCount, 
CommentCount as CommentCount, 
FavoriteCount as FavoriteCount, 
(chararray)REPLACE(ClosedDate,'"','') as ClosedDate, 
(chararray)REPLACE(CommunityOwnedDate,'"','') as CommunityOwnedDate;

现在的问题是，当我存储输出时显示 617538 行已写入。它创建了两个文件。第一个文件有 27000 条格式正确的记录，但第二个文件没有正确存储。它包含大约 610000 行和许多行，其中只有 ,。如何正确加载数据，以便输出显示 50000 而不是 617538 行。

Here's the load status

【参考方案1】：

我认为问题出在脚本的以下部分。

(chararray)REPLACE(REPLACE(Body,'\n',''),',','')as Body, 

您必须添加另一个反斜杠来替换 '\n'

(chararray)REPLACE(REPLACE(Body,'\\n',''),',','')as Body, 

【讨论】：

我厌倦了用另一个反斜杠替换 \n 但仍然显示相同数量的记录。

@user6118910 你能发布示例数据吗？