hadoop文件存储格式

Posted 2021-04-13 大数据猿群

1. txt

1. 纯文本格式，若干行记录。默认用字符编码存储

2. SequenceFile

   
       Configuration configuration = new Configuration();
       configuration.set("fs.defaultFS","hdfs://s100:8020");
       FileSystem fileSystem = FileSystem.get(configuration);
       Path path = new Path("hdfs://s100:8020/user/seqmyfile.seq");
       SequenceFile.Writer writer = SequenceFile.createWriter(fileSystem, configuration, path, IntWritable.class, Text.class);
   
   
       writer.append(new IntWritable(1),new Text("gg1"));
       writer.append(new IntWritable(1),new Text("gg2"));
       writer.append(new IntWritable(1),new Text("gg3"));
       writer.append(new IntWritable(1),new Text("gg4"));
       writer.close();    

    Configuration configuration = new Configuration();
       configuration.set("fs.defaultFS","hdfs://s100:8020");
       FileSystem fileSystem = FileSystem.get(configuration);
       Path path = new Path("hdfs://s100:8020/user/seqmyfile.seq");
       SequenceFile.Reader sr = new SequenceFile.Reader(fileSystem,path,configuration);
   
       IntWritable key = new IntWritable();
       Text value = new Text();    while (sr.next(key,value)){
           System.out.println(key +":"+value );
       }    

3. 编写读取 seq 文件案例

4. key-value 格式进行存储，最终形成的是一个二进制文件， 需用hadoop提供的api进行写入存储。

5. 编写 写入 seq文件案例。