无法通过级联一键连接两个文件

Posted 2023-04-17

【中文标题】无法通过级联一键连接两个文件

【英文标题】：Couldn`t join two files with one key via Cascading

【发布时间】：2016-06-22 16:57:22

【问题描述】：

让我们看看我们有什么。第一个文件【接口类】：

list arrayList
list linkedList

第二档[Class1数量]：

arrayList 120
linkedList 4

我想通过 key[Class] 加入这两个文件并获取每个接口的计数：

list arraylist 120
list linkedlist 4

代码：

public class Main

public static void main( String[] args )

String docPath = args[ 0 ];
String wcPath = args[ 1 ];
String doc2Path = args[ 2 ];

Properties properties = new Properties();
AppProps.setApplicationJarClass( properties, Main.class );
AppProps.setApplicationName( properties, "Part 1" );
AppProps.addApplicationTag( properties, "lets:do:it" );
AppProps.addApplicationTag( properties, "technology:Cascading" );
FlowConnector flowConnector = new Hadoop2MR1FlowConnector( properties );

// create source and sink taps
Tap wcTap = new Hfs(new TextDelimited(true, ","), wcPath);


    Fields classInterfaceFiles = new Fields("interface", "class");
    Tap classInterfaceTap = new Hfs(new TextDelimited(classInterfaceFiles, true, ","), docPath);

    Fields classAmountFields = new Fields("class1", "amount");
    Tap classAmountFileTap = new Hfs(new TextDelimited(classAmountFields, true, ","), doc2Path);

    Tap outTap = new MultiSinkTap(); // just saying, create your own tap
    Pipe classInterfaceFilePipe = new Pipe("classInterfaceFilePipe");

    Pipe classIAmountFilePipe = new Pipe("classIAmountFilePipe");

    Fields groupFields = new Fields("class");
    Fields groupFields1 = new Fields("class1"); // fields used as joining keys
    Pipe outPipe = new CoGroup(classInterfaceFilePipe, groupFields, classIAmountFilePipe, groupFields1, new InnerJoin());


 // build flow definition
    FlowDef flowDef = FlowDef.flowDef().setName("myFlow")
            .addSource(classInterfaceFilePipe, classInterfaceTap)
            .addSource(classIAmountFilePipe, classAmountFileTap)
            .addTailSink(outPipe, wcTap);
         //   .addTailSink( outPipe, wcTap );


// write a DOT file and run the flow
Flow wcFlow = flowConnector.connect( flowDef );
wcFlow.writeDOT( "dot/wc.dot" );
wcFlow.complete();

[这是更大任务的一步]

【参考方案1】：

发生这种情况是因为您在连接在一起的两个管道中具有相同的字段，即“类”。可能您可以将它们重命名为“class_interface”和“class_amount”。您还必须更改在 CoGroup 管道中使用的 groupFields。

【讨论】：

您的建议和一些改进代码运行良好。所以我提供了更新的代码，并且在问题中是工作代码。