mysql调优小记

Posted 2020-10-04 我的空中楼阁

对于INNODB,主键就是聚集索引,如果没有主键定义,则第一个唯一非空索引被作为聚集索引.
如果没有主键也没有合适的唯一索引,那么innodb内部会生成一个隐藏的主键作为聚集索引,
这个隐藏的主键类似一个自增的id(int).

删除与重新添加主键
alter table tbname drop primary key;

alter table tbname add primary key(col1,col2,col3);

CREATE INDEX IDX_METE_BASE_AREA_STATION_CABINET ON CMS.T_CABINET_ELECTRICITY_HH_HIVE (
BASE_NAME ASC,
AREA_NAME ASC,
STATION_NAME ASC,
CABINET_NAME ASC
);

SELECT DISTINCT DEVICE_NAME ,REGEXP_EXTRACT(DEVICE_NAME,"-(.*)",0)
FROM T_METE_CONFIG
WHERE LSC_ID IN (30,31)
AND DEVICE_TYPE_NAME=‘UPS配电‘
AND STATION_NAME LIKE ‘%数据机房%‘
ORDER BY DEVICE_NAME;

SELECT REGEXP_EXTRACT(‘ASDFSADF-JOB‘,‘-([A-Z])‘,0) FROM CMS.DUAL LIMIT 1;

复合分区示例,
先以时间粒度天为例进行外分区,然后根据地域id进行分区,
mysql仅支持range和list的子分区,而且子分区只能是hash或key.

CREATE TABLE t_cabinet_electricity_hh_hive2
(
   base_id int NOT NULL,
   base_name varchar(20),
   lsc_name varchar(255),
   area_id bigint,
   area_name varchar(255),
   station_id int NOT NULL,
   station_name varchar(255),
   device_type_id varchar(255),
   device_type_name varchar(255),
   device_id varchar(255) NOT NULL,
   device_name varchar(255),
   mete_id int NOT NULL,
   mete_name varchar(255),
   max_report_time varchar(20),
   max_mete_value double,
   min_report_time varchar(20),
   min_mete_value double,
   avg_mete_value double,
   hour_sum_value double,
   hour_times int,
   dt_hh varchar(10) NOT NULL,
   hour_last_value double,
   cabinet_name varchar(255),
   dt varchar(20) NOT NULL,
   lsc_id int NOT NULL,
   PRIMARY KEY (dt,dt_hh,base_id,lsc_id,mete_id,device_id,station_id,mete_name)
)
PARTITION BY RANGE(TO_DAYS(DT))
SUBPARTITION BY HASH(LSC_ID)
SUBPARTITIONS 6
(
PARTITION P0  VALUES LESS THAN (TO_DAYS(‘20170810‘)),
PARTITION P1  VALUES LESS THAN (TO_DAYS(‘20170811‘)),
PARTITION P2  VALUES LESS THAN (TO_DAYS(‘20170812‘)),
PARTITION P3  VALUES LESS THAN (TO_DAYS(‘20170813‘)),
PARTITION P4  VALUES LESS THAN (TO_DAYS(‘20170814‘)),
PARTITION P5  VALUES LESS THAN (TO_DAYS(‘20170815‘)),
PARTITION P6  VALUES LESS THAN (TO_DAYS(‘20170816‘)),
PARTITION P7  VALUES LESS THAN (TO_DAYS(‘20170817‘)),
PARTITION P8  VALUES LESS THAN (TO_DAYS(‘20170818‘)),
PARTITION P9  VALUES LESS THAN (TO_DAYS(‘20170819‘)),
PARTITION P10  VALUES LESS THAN (TO_DAYS(‘20170820‘))
)
;

ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P11  VALUES LESS THAN (TO_DAYS(‘20170821‘)));         
ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P12  VALUES LESS THAN (TO_DAYS(‘20170822‘)));
ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P13  VALUES LESS THAN (TO_DAYS(‘20170823‘)));
ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P14  VALUES LESS THAN (TO_DAYS(‘20170824‘)));
ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P15  VALUES LESS THAN (TO_DAYS(‘20170825‘)));
ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P16  VALUES LESS THAN (TO_DAYS(‘20170826‘)));
ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P17  VALUES LESS THAN (TO_DAYS(‘20170827‘)));
ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P18  VALUES LESS THAN (TO_DAYS(‘20170828‘)));
ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P19  VALUES LESS THAN (TO_DAYS(‘20170829‘)));
ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P20  VALUES LESS THAN (TO_DAYS(‘20170830‘)));
ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P21  VALUES LESS THAN (TO_DAYS(‘20170831‘)));
ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P22  VALUES LESS THAN (TO_DAYS(‘20170901‘)));
        
--查看分区信息
select
partition_name,table_rows
from information_schema.partitions where table_name=‘xxx‘;
 
 
问题,hive表中的日期类型 yyyy-mm-dd 到mysql中对应的date字段,但是无法导入成功
其实看sqoop转换的java代码,可以看出这是sqoop的一个bug.,
里面有类似 (java.sql.Date)value
转换失败为空,这样进入我含有非空约束的字段就失败了.
原因是日期格式是不确定的,所以直接转有问题,
建议是直接设置临时表的字段为varchar,后面带做处理.
 

sqoop eval --connect jdbc:mysql://xx:3306/cms --username root --password xxx \
--query "truncate table xx.stage_t_xx"

sqoop export --connect jdbc:mysql://xxx:3306/xxx\
--username root --password xxx --table stage_t_xxe \
--hcatalog-database xxx--hcatalog-table stage_txxx

sqoop eval --connect jdbc:mysql://xxx:3306/cms --username root --password xxx \
--query "insert into xxx.t_xxx_xx_hh_xx select * from xxx.stage_t_xx_xx_hh_hive"