如何使用kettle 做数据清洗

Posted 2023-04-08

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了如何使用kettle 做数据清洗相关的知识，希望对你有一定的参考价值。

参考技术A 在kettle常常有处理从一个源数据中做转换.做转换的时候, 需要去查另一个数据库. 这种问题遇到数据小时候还好办. 但是数据魇时候就麻烦来了. 下面针对三种情况做具体情况的选择办法先上一个图 [img] [/img] 1. 当需要转换的数据特别大的时候,

kettle - 清洗 mongodb 数据案例

文章目录

前言
- - kettle - 清洗 mongodb 数据案例

前言

如果您觉得有用的话，记得给博主点个赞，评论，收藏一键三连啊，写作不易啊^ _ ^。
而且听说点赞的人每天的运气都不会太差，实在白嫖的话，那欢迎常来啊!!!

kettle - 清洗 mongodb 数据案例

一、需求

需求:将 mongodb kettleTest集合脏数据整理到 kettleTestClear集合中
mongodb 待清洗的数据:
db.kettleTest.find();

[
  
    "_id": "12",
    "_class": "com.yzy.mongodb.model.KettleTestModel",
    "createTime": "$date": "2022-12-28T10:01:38.989Z",
    "delete": false,
    "kettleVo": 
      "list": ["1"],
      "key": "a",
      "code": "A"
    ,
    "name": "哈哈"
  ,
  
    "_id": "13",
    "_class": "com.yzy.mongodb.model.KettleTestModel",
    "createTime": "$date": "2022-12-28T10:01:42.001Z",
    "delete": false,
    "kettleVo": 
      "list": ["1", "2"],
      "code": "B"
    ,
    "name": "一模块"
  ,
  
    "_id": "14",
    "_class": "com.yzy.mongodb.model.KettleTestModel",
    "createTime": "$date": "2022-12-28T10:01:45.004Z",
    "delete": false,
    "kettleVo": 
      "key": "c",
      "code": "C"
    ,
    "name": "萨迪"
  ,
  
    "_id": "23",
    "_class": "com.yzy.mongodb.model.KettleTestModel",
    "delete": false,
    "kettleVo": 
      "key": "d",
      "code": "D"
    ,
    "name": "都是"
  ,
  
    "_id": "15",
    "_class": "com.yzy.mongodb.model.KettleTestModel",
    "createTime": "$date": "2022-12-28T10:01:51.013Z",
    "delete": false,
    "name": "是南方"
  
]

需求点:
1)待清洗集合和清洗后集合字段映射关系:
kettleTest.createTime -> kettleTestClear.createTime
kettleTest.name -> kettleTestClear.name
kettleTest.kettleVo.key -> kettleTestClear.key
kettleTest.kettleVo.code-> kettleTestClear.code
kettleTest.kettleVo.list -> kettleTestClear.list
kettleTest.delete -> kettleTestClear.delete
2)只清洗kettleTest.delete:false 的数据;
3)同时将delete字段的值映射后导入kettleTestClear集合中，映射关系: true:0,false:1;
4)根据_id进行排序后存入kettleTestClear集合;

二、kettle开发

整体流程:

1、新建mongodb数据查询

查询kettleTest.delete:false的数据:

2、配置kettleTest集合与清洗后kettleTestClear集合字段映射

注意:
1、_id放在最上面，因为它永远不会为空;
2、其中createTime这里是重点要加.$date;

3、根据_id进行排序

4、使用java脚本将日期格式化

import java.text.ParseException;
import java.util.Date;
import java.text.DateFormat;
import java.text.ParseException;
import java.text.SimpleDateFormat;
private static final SimpleDateFormat sdf = new SimpleDateFormat("yyyy/MM/dd  HH:mm:ss.SSS");
private static final String TIME_LINE = "yyyy-MM-dd'T'HH:mm:ss.SSSXX";
public static String dateToString(Date date) 
           return sdf.format(date);

public static Date stringToDate(String date, String format) throws  ParseException 
      DateFormat dateFormat = new SimpleDateFormat(format);
      dateFormat.setLenient(false);
      return dateFormat.parse(date);

public boolean processRow(StepMetaInterface smi, StepDataInterface sdi) throws  KettleException,ParseException 
  if (first) 
    first = false;
  
  Object[] r = getRow();
  if (r == null ) 
    setOutputDone();
    return false;
  
   
  r = createOutputRow(r, data.outputRowMeta.size());
  String createTime = get(Fields.In, "createTime").getString(r);
  if(null != createTime && "" != createTime)
    //时间格式化
    get(Fields.Out, "createTime").setValue(r,  dateToString(stringToDate(createTime,TIME_LINE)));
  else
    get(Fields.Out, "createTime").setValue(r, null);
  
  
  logBasic("----------"+createTime+"---------------");
  putRow(data.outputRowMeta, r);
  return true;