使用自定义换行符加载 CSV

Posted 2023-04-18

【中文标题】使用自定义换行符加载 CSV

【英文标题】：Loading CSV using Custom Line break

【发布时间】：2014-11-04 06:58:48

【问题描述】：

我有一个如下的 CSV，其中该行以“+++”而不是新行终止。如何通过在字符串“+++”存在的地方换行来加载csv？

VTS,51,0071,9739965515,NM,GP,INF01,V,19,072219,291014,0000.0000,N,00000.0000,E,07AE

VTS,01,0097,9739965515,SP,GP,18,072253,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,169,B205+++VTS,51,0071,9739965515,NM,GP,INF01,V,18,072311,291014,0000.0000,N,00000.0000,E,C24E+++VTS,01,0097,9739965515,NM,GP,19,072311,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,171,B358

VTS,51,0071,9739965515,NM,GP,INF01,V,18,072319,291014,0000.0000,N,00000.0000,E,012F
VTS,51,0071,9739965515,NM,GP,INF01,V,19,072326,291014,0000.0000,N,00000.0000,E,B2E6+++VTS,01,0097,9739965515,NM,GP,18,072326,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,173,EAA0
VTS,51,0071,9739965515,NM,GP,INF01,V,18,072333,291014,0000.0000,N,00000.0000,E,9896
VTS,51,0071,9739965515,NM,GP,INF01,V,18,072340,291014,0000.0000,N,00000.0000,E,9B23

首先，我需要在新行或“+++”符号所在的行换行并加载数据。然后，再次使用第二列中的值 01 进行过滤。

预期输出：

VTS,01,0097,9739965515,SP,GP,18,072253,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,169,B205
VTS,01,0097,9739965515,NM,GP,19,072311,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,171,B358
VTS,01,0097,9739965515,NM,GP,18,072326,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,173,EAA0

【问题讨论】：

您的预期输出是什么？

@SivasakthiJayaraman 给出了预期的输出

我更新了解决方案，请验证并告诉我这是否适合您。

【参考方案1】：

PigScript：

A = LOAD 'input.csv' AS (line:chararray);
B = FOREACH A 
                 splitRow = TOKENIZE(line,'+++');
                 GENERATE FLATTEN(splitRow) AS newList;
              
C = FOREACH B GENERATE FLATTEN(STRSPLIT(newList,',',16));
D = FILTER C BY $1==01;
DUMP D;

输出：

(VTS,01,0097,9739965515,SP,GP,18,072253,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,169,B205)
(VTS,01,0097,9739965515,NM,GP,19,072311,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,171,B358)
(VTS,01,0097,9739965515,NM,GP,18,072326,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,173,EAA0)

【讨论】：

你能解释一下上面的步骤，以便我能理解内部工作是什么

当我添加额外的一行时 E = foreach D 生成 $7,$15;转储 E；我得到的结果是 (060037,061114,0068,00,4000,00,999,149,9594) (060113,061114,0068,00,4000,00,999,152,B927) 为什么不必要地提交所有其余的文件而不是 7 美元和 15 美元

更改行“C = FOREACH B GENERATE FLATTEN(STRSPLIT(newList,',',23));”即，而不是 16 给出 23。基本上它的总列数。我错误地给了 16。