TF-IDF算法介绍，简单模拟，以及在图数据中应用

Posted 2021-06-28 PoetryAndTheDistance

 

一、百度百科关于TF-IDF的算法介绍

https://baike.baidu.com/item/tf-idf/8816134?fr=aladdin

 

二、简单模拟

既然是简单模拟，我们就用人们最常用的一种工具mysql去模拟一下这个算法可以实现的效果

 

2.1、创建表以及数据

article_keywords.sql

CREATE TABLE `article_keywords` (
  `id` int NOT NULL AUTO_INCREMENT,
  `article` varchar(255) NOT NULL,
  `keyword` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=126 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;


INSERT INTO `article_keywords` VALUES ('113', '悟空悟空芭蕉扇牛魔王', '悟空');
INSERT INTO `article_keywords` VALUES ('114', '悟空悟空芭蕉扇牛魔王', '悟空');
INSERT INTO `article_keywords` VALUES ('115', '悟空悟空芭蕉扇牛魔王', '芭蕉扇');
INSERT INTO `article_keywords` VALUES ('116', '悟空悟空芭蕉扇牛魔王', '牛魔王');
INSERT INTO `article_keywords` VALUES ('117', '悟空红孩儿红孩儿三昧真火三昧真火', '悟空');
INSERT INTO `article_keywords` VALUES ('118', '悟空红孩儿红孩儿三昧真火三昧真火', '红孩儿');
INSERT INTO `article_keywords` VALUES ('119', '悟空红孩儿红孩儿三昧真火三昧真火', '红孩儿');
INSERT INTO `article_keywords` VALUES ('120', '悟空红孩儿红孩儿三昧真火三昧真火', '三昧真火');
INSERT INTO `article_keywords` VALUES ('121', '悟空红孩儿红孩儿三昧真火三昧真火', '三昧真火');
INSERT INTO `article_keywords` VALUES ('122', '悟空三昧真火火眼金睛', '三昧真火');
INSERT INTO `article_keywords` VALUES ('123', '悟空三昧真火火眼金睛', '火眼金睛');
INSERT INTO `article_keywords` VALUES ('124', '悟空三昧真火火眼金睛', '悟空');

2.2、模拟实现TF-IDF的统计算分

实现sql

#实现TF-IDF的统计算分值
SELECT * FROM (
	#计算TF-IDF的值
	SELECT *,(TF * IDF)as 'TF-IDF' FROM (
		#分别计算TF和IDF的值
		SELECT *,(keyword_in_current_article_count / current_article_keyword_count) as TF,(article_count / keyword_in_all_article_count) as IDF FROM(
			#查询表中数据以及联查计算TF-IDF所需的各种聚合值以及直接聚合文章总数量的值
			SELECT article_keywords.*,keyword_in_current_article_count.keyword_in_current_article_count,current_article_keyword_count.current_article_keyword_count,keyword_in_all_article_count.keyword_in_all_article_count,(SELECT COUNT(1) FROM (SELECT article FROM article_keywords GROUP BY article) as a)as article_count FROM article_keywords as article_keywords 
			#联查关键词在当前文章中出现的频率的聚合表
			LEFT JOIN (SELECT article,keyword,COUNT(1) as keyword_in_current_article_count FROM article_keywords GROUP BY article,keyword)as keyword_in_current_article_count
			ON article_keywords.article = keyword_in_current_article_count.article AND article_keywords.keyword = keyword_in_current_article_count.keyword
			#联查当前文章中所有关键词的个数的聚合表
			LEFT JOIN (SELECT article,COUNT(keyword) as current_article_keyword_count FROM article_keywords GROUP BY article) as current_article_keyword_count
			ON article_keywords.article = current_article_keyword_count.article
			#联查出现此关键词的文章个数的聚合表
			LEFT JOIN (SELECT a.keyword as keyword,COUNT(article) keyword_in_all_article_count FROM (
				SELECT article,keyword FROM article_keywords GROUP BY article,keyword
			)as a GROUP BY a.keyword)as keyword_in_all_article_count
			ON article_keywords.keyword = keyword_in_all_article_count.keyword
		) as article_keywords_tf_idf
	) as article_keywords_tfidf
)as article_keywords_tfidf_order 
#WHERE keyword = '三昧真火'
GROUP BY article,keyword ORDER BY article_keywords_tfidf_order.`TF-IDF` DESC

结果样例

这只是用大家都熟悉的一个MySQL做一个简单的模拟实现的效果

当然，在实际应用中的场景比这要复杂的多，而且实际应用也不会使用MySQL来实现

下面我们来看看在图数据中的一个实际应用场景

三、图数据中应用