怎么把impala表导入kudu

Posted 2023-03-15

可以通过映射表方式实现。下面说得略细，需要耐心看并做参考：
首先要知道目前（2017.05.04）kudu没有timestamp与decimal类型，需要用其它类型代替，比如timestamp可以用长bigint代替，decimal可以用double型代替。
如果想在kudu中创建一个新表，并将数据导入到这个新表中，那么比较简单的方式是通过impala建立kudo的内部映射表，因为建立内部映射表过程会同时建立一个kudu新表并在impala中建立一个内部映射表，建立方法如下:
首先保证kudu正确安装并且master与tserver服务都正常运行，然后在impala shell中输入
create table table_name (
column1 type primary key ...,
column2 type,
...
)
partition by hash(column1) partitions n stored as kudu;
这样就在kudu中建立好表了，下一步就可以通过impala的映射表向这个kudu表中导入数据了（table_name要用自己取的表名代替，columnN也要用自己取的列名代替。）（如果建立列集主键，方式为primary key(column1, column2, column3)，具体如何建立这里就不细说了）（顺便说一下：这样建立的kudu中的表名为：impala::database_name.table_name）。
如果kudu中已经存在一个现有表，您想将impala表中数据导入到这个现有kudu表中，那么需要在impala中建立外部映射表，方法是：
create external table table_name
stored as kudu
tblproperties('kudu.table_name' = 'kudu中的table_name');
这样就建立完成了。
在impala shell 中执行：
insert into table_name select * from 你的impala表;

就可以将你的表数据导入到新表了。
最后提及：kudu表必须有主键或者列集主键，可能你的表没有主键或列集主键，而你的表第一列或对应列集的前几列又有重复数据，这种情况下你的表导入后会有数据丢失，比较简单的解决方法是：
在建立kudu表时增加一个专门的主键，放在第一列。然后：
insert into table_name select row_number() over(order by 你的impala表中任意一列名), * from 你的impala表; 参考技术A HDFS和HBase两个系统把持着，一直没有太大突破。在追求高吞吐的批处理场景下，我们选用HDFS，在追求低延迟，有随机读写需求的场景下，我们选用HBase，那么是否存在一种系统，能结合两个系统优点，同时支持高吞吐率和低延迟呢？有人尝试修改HBas

客快物流大数据项目（七十九）：Impala映射kudu表

文章目录

Impala映射kudu表

一、​​​​​​​​​​​​​​登录Hue页面

1、选择Impala

2、登录Hue页面

二、选择Impala执行引擎

1、选择Impala执行引擎

2、进入编写执行sql语句窗口

三、执行sql语句映射Kudu表

---

Impala映射kudu表

一、​​​​​​​​​​​​​​登录Hue页面

1、选择Impala

 

2、登录Hue页面

 

二、选择Impala执行引擎

1、选择Impala执行引擎

2、进入编写执行sql语句窗口

 

三、执行sql语句映射Kudu表

若数据库已经存在，则使用级联的方式删除：

DROP DATABASE logistics cascade;

编写kudu与impala表的映射关系，Sql语句如下：

CREATE DATABASE IF NOT EXISTS logistics;

CREATE EXTERNAL TABLE logistics.tbl_warehouse_transport_tool_detail

 STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_warehouse_transport_tool_detail',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_dot_transport_tool_detail STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_dot_transport_tool_detail',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');
  
  
CREATE EXTERNAL TABLE logistics.tbl_express_bill_detail STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_express_bill_detail',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_waybill STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_waybill',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_collect_package STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_collect_package',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_store_grid STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_store_grid',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_pkg STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_pkg',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_out_warehouse_detail STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_out_warehouse_detail',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_goods_rack STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_goods_rack',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_warehouse_receipt_detail STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_warehouse_receipt_detail',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_fixed_area STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_fixed_area',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_department STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_department',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_express_bill STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_express_bill',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_delivery_record STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_delivery_record',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_waybill_detail STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_waybill_detail',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_warehouse_emp STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_warehouse_emp',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_emp_info_map STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_emp_info_map',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_service_evaluation STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_service_evaluation',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_dot_transport_tool STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_dot_transport_tool',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_postal_standard STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_postal_standard',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_emp STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_emp',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_push_warehouse_detail STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_push_warehouse_detail',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_customer STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_customer',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_warehouse_transport_tool STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_warehouse_transport_tool',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_deliver_region STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_deliver_region',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_waybill_state_record STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_waybill_state_record',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_company_dot_map STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_company_dot_map',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_areas STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_areas',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_company_warehouse_map STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_company_warehouse_map',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_warehouse_send_vehicle STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_warehouse_send_vehicle',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_warehouse_vehicle_map STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_warehouse_vehicle_map',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_customer_detail STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_customer_detail',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_company STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_company',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_courier STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_courier',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_vehicle_monitor STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_vehicle_monitor',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_deliver_package STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_deliver_package',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_charge_standard STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_charge_standard',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_warehouse_rack_map STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_warehouse_rack_map',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_codes STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_codes',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_warehouse STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_warehouse',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_job STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_job',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_waybill_line STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_waybill_line',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_work_time STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_work_time',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_consumer_sender_info STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_consumer_sender_info',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_driver STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_driver',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_consumer_address_map STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_consumer_address_map',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_warehouse_receipt STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_warehouse_receipt',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_dot STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_dot',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_transport_tool STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_transport_tool',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_out_warehouse STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_out_warehouse',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');


CREATE EXTERNAL TABLE logistics.tbl_address STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_address',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');  

CREATE EXTERNAL TABLE logistics.tbl_route STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_route',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_push_warehouse STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_push_warehouse',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_company_transport_route_ma STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_company_transport_route_ma',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE logistics.tbl_express_package STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_express_package',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

CREATE EXTERNAL TABLE `tbl_warehouse_detail` STORED AS KUDU TBLPROPERTIES(
  'kudu.table_name' = 'tbl_warehouse_detail',
  'kudu.master_addresses' = 'node2.lanson.cn:7051');

---

• 📢博客主页：https://lansonli.blog.csdn.net
• 📢欢迎点赞 👍 收藏 ⭐留言 📝 如有错误敬请指正！
• 📢本文由 Lansonli 原创，首发于 CSDN博客🙉
• 📢停下休息的时候不要忘了别人还在奔跑，希望大家抓紧时间学习，全力奔赴更美好的生活✨