如何提高跨三个连接表的 DISTINCT 选择的性能？
Posted
技术标签:
【中文标题】如何提高跨三个连接表的 DISTINCT 选择的性能？【英文标题】：How do I improve performance on a DISTINCT select across three joined tables? 【发布时间】：2016-12-07 09:34:28 【问题描述】：我有以下表格：
角色 ImpressionsPersonas [连接表 - Personas ManyToMany Impressions] 展示次数我的查询是这样的，EXPLAIN 结果附在下面：
SELECT
DISTINCT (Personas.id),
Personas.parent_id,
Personas.persona,
Personas.subpersonas_count,
Personas.is_subpersona,
Personas.impressions_count,
Personas.created,
Personas.modified
FROM personas as Personas
INNER JOIN
impressions_personas ImpressionsPersonas ON (
Personas.id = ImpressionsPersonas.persona_id
)
inner JOIN impressions Impressions ON (Impressions.id = ImpressionsPersonas.impression_id AND Impressions.timestamp >= "2016-06-01 00:00:00" AND Impressions.timestamp <= "2016-07-31 00:00:00")
解释
+----+-------------+---------------------+--------+-----------------------------------------------------------------------+-------------+---------+---------------------------------------------+------+----------+-----------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+---------------------+--------+-----------------------------------------------------------------------+-------------+---------+---------------------------------------------+------+----------+-----------------------+
| 1 | SIMPLE | Personas | ALL | PRIMARY | NULL | NULL | NULL | 159 | 100.00 | Using temporary |
| 1 | SIMPLE | ImpressionsPersonas | ref | impression_idx,persona_idx,comp_imp_persona,comp_imp_pri,comp_per_pri | persona_idx | 8 | gen1_d2go.Personas.id | 396 | 100.00 | Distinct |
| 1 | SIMPLE | Impressions | eq_ref | PRIMARY,timestamp,timestamp_id | PRIMARY | 8 | gen1_d2go.ImpressionsPersonas.impression_id | 1 | 100.00 | Using where; Distinct |
+----+-------------+---------------------+--------+-----------------------------------------------------------------------+-------------+---------+---------------------------------------------+------+----------+-----------------------+
3 rows in set, 1 warning (0.00 sec)
为角色创建声明
CREATE TABLE `personas` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`parent_id` bigint(20) unsigned DEFAULT NULL,
`persona` varchar(150) NOT NULL,
`subpersonas_count` int(10) unsigned DEFAULT '0',
`is_subpersona` tinyint(1) unsigned DEFAULT '0',
`impressions_count` bigint(20) unsigned DEFAULT '0',
`created` datetime DEFAULT NULL,
`modified` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `lookup` (`parent_id`,`persona`),
KEY `parent_index` (`parent_id`),
KEY `persona` (`persona`),
KEY `persona_a_id` (`id`,`persona`),
CONSTRAINT `self_referential_join_to_self` FOREIGN KEY (`parent_id`) REFERENCES `personas` (`id`) ON DELETE CASCADE ON UPDATE NO ACTION
) ENGINE=InnoDB AUTO_INCREMENT=1049 DEFAULT CHARSET=utf8;
为 IMPRESSIONS_PERSONAS 创建声明
CREATE TABLE `impressions_personas` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`impression_id` bigint(20) unsigned NOT NULL,
`persona_id` bigint(20) unsigned NOT NULL,
`created` datetime DEFAULT NULL,
`modified` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `impression_idx` (`impression_id`),
KEY `persona_idx` (`persona_id`),
KEY `comp_imp_persona` (`impression_id`,`persona_id`),
KEY `comp_imp_pri` (`impression_id`,`id`),
KEY `comp_per_pri` (`persona_id`,`id`),
CONSTRAINT `impression` FOREIGN KEY (`impression_id`) REFERENCES `impressions` (`id`) ON DELETE CASCADE ON UPDATE NO ACTION,
CONSTRAINT `persona` FOREIGN KEY (`persona_id`) REFERENCES `personas` (`id`) ON DELETE CASCADE ON UPDATE NO ACTION
) ENGINE=InnoDB AUTO_INCREMENT=19387839 DEFAULT CHARSET=utf8;
创建展示声明
CREATE TABLE `impressions` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`device_id` bigint(20) unsigned NOT NULL,
`beacon_id` bigint(20) unsigned NOT NULL,
`zone_id` bigint(20) unsigned NOT NULL,
`application_id` bigint(20) unsigned DEFAULT NULL,
`timestamp` datetime NOT NULL,
`google_place_id` bigint(20) unsigned DEFAULT NULL,
`name` varchar(60) DEFAULT NULL,
`lat` decimal(15,10) DEFAULT NULL,
`lng` decimal(15,10) DEFAULT NULL,
`personas_count` int(10) unsigned DEFAULT '0',
`created` datetime DEFAULT NULL,
`modified` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `device_idx` (`device_id`),
KEY `zone_idx` (`zone_id`),
KEY `beacon_id_idx2` (`beacon_id`),
KEY `timestamp` (`timestamp`),
KEY `appid_fk_idx_idx` (`application_id`),
KEY `comp_lookup` (`device_id`,`beacon_id`,`timestamp`),
KEY `timestamp_id` (`timestamp`,`id`),
CONSTRAINT `appid_fk_idx` FOREIGN KEY (`application_id`) REFERENCES `applications` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT `beacon_id` FOREIGN KEY (`beacon_id`) REFERENCES `beacons` (`id`) ON DELETE CASCADE ON UPDATE NO ACTION,
CONSTRAINT `device2` FOREIGN KEY (`device_id`) REFERENCES `devices` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT `zone_FK` FOREIGN KEY (`zone_id`) REFERENCES `zones` (`id`) ON DELETE CASCADE ON UPDATE NO ACTION
) ENGINE=InnoDB AUTO_INCREMENT=1582724 DEFAULT CHARSET=utf8;
现在 - 当我在没有 DISTINCT 并使用 COUNT(*) 的情况下运行查询时，它会提取大约 17,000,000 条记录。使用 DISTINCT 运行它会产生 112 条记录。我不知道为什么当解释只显示 159 和 396 时会显示这么多记录。
关于表格的一些信息：
角色表包含 159 条记录。 ImpressionsPersonas 表包含大约 1260 万条记录，Impressions 包含大约 920,000 条记录。
我们正在做的是选择Personas 表并通过连接表ImpressionsPersonas 连接到印象数。 Impressions 表（在本例中为日期）应用了过滤器。
注意：删除日期过滤器对执行时间的影响可以忽略不计 - 徘徊在 120 秒左右。有没有办法过滤这些记录以减少此查询的执行时间？
【问题讨论】：
将整个加入更改为IN 或有什么帮助？不确定引擎如何处理事情，但是当您实际上只是找到唯一 ID 然后从角色表中获取这几条记录时，加入和区分似乎有点毫无意义。
select * from personas where id in (select distinct(persona_id) from impressions_personas inner JOIN impressions Impressions ON Impressions.id = ImpressionsPersonas.impression_id AND Impressions.timestamp >= "2016-06-01 00:00:00" AND Impressions.timestamp <= "2016-07-31 00:00:00")) 中的内容。也不确定 group by 是否与 DISTINCT 不同
@SamiKuhmonen 很好 - 因为有一个连接表，它会导致返回重复的记录，因此需要不同的选择。
@BarryChapman 是的，但是当您按 first 进行不同/分组然后选择您想要的实际行时，它可能会非常不同。您不需要连接三个表，只需连接两个表并获得几个 ID 来选择最终结果。
DISTINCT 不是函数。它不需要任何论据。你的括号没有用（除了混淆）
【参考方案1】：
在impressions 表的timestamp 列上创建一个INDEX。并查看是否改进 else 尝试在查询中使用创建的索引（强制索引）。
更新
在JOIN 中使用INDEX
SELECT
DISTINCT (Personas.id),
Personas.parent_id,
Personas.persona,
Personas.subpersonas_count,
Personas.is_subpersona,
Personas.impressions_count,
Personas.created,
Personas.modified
FROM
personas as Personas
INNER JOIN
impressions_personas ImpressionsPersonas ON (
Personas.id = ImpressionsPersonas.persona_id
)
INNER JOIN
impressions Impressions WITH(INDEX(timestamp)) ON
(Impressions.id = ImpressionsPersonas.impression_id AND
Impressions.timestamp >= "2016-06-01 00:00:00" AND
Impressions.timestamp <= "2016-07-31 11:59:59")
【讨论】：
KEYtimestamp (timestamp),【参考方案2】：
我假设您想获取在指定时间段内至少有 1 次展示的人员列表。为此，您可以使用这样的相关子查询：
SELECT
Personas.id,
Personas.parent_id,
Personas.persona,
Personas.subpersonas_count,
Personas.is_subpersona,
Personas.impressions_count,
Personas.created,
Personas.modified
FROM personas as Personas
WHERE EXISTS(SELECT 1 FROM impressions_personas
LEFT JOIN impressions Impressions ON
Impressions.id = ImpressionsPersonas.impression_id
WHERE Personas.id = ImpressionsPersonas.persona_id
AND Impressions.timestamp >= "2016-06-01 00:00:00"
AND Impressions.timestamp <= "2016-07-31 00:00:00"
)
【讨论】：
【参考方案3】：目前，您首先连接三个具有数百万行的表，然后使用DISTINCT 仅从中取出几行。更好的方法是首先只获取所需的 ID，然后使用它们来选择实际的结果数据。
例如：
SELECT column, other FROM personas
WHERE id IN
(SELECT distinct persona_id
FROM impressions_personas
INNER JOIN impressions Impressions
ON Impressions.id = ImpressionsPersonas.impression_id
AND Impressions.timestamp >= "2016-06-01 00:00:00"
AND Impressions.timestamp <= "2016-07-31 00:00:00"))
这样引擎将只处理整个过程的一列，直到得到结果。
【讨论】：
以上是关于如何提高跨三个连接表的 DISTINCT 选择的性能？的主要内容，如果未能解决你的问题，请参考以下文章