featuretools入门

Posted 2020-12-08 syukaketu

featuretools入门尝试

简介

特征工程中最重要的一步，是对baseline提升最大的一个步骤，对数据的EDA以及构建特征，是不可缺少一部分。python的特征工程常用agg与groupby的进行聚合统计。
首先，我们得先了解一下featuretools的3个基本组成

• 实体集（EntitySet）：把一个二维表看作一个实体，实体集是一个或多个二维表的集合
• 特征基元（Feature Primitives）：分为聚合和转换两类，相当于构造新特征的方法
• 深度特征合成（DFS, Deep Feature Synthesis）：根据实体集里的实体和特征基元创造新特征

入门尝试

featuretools是一种自动化特征生成的特征工程框架，可以快速生成大量的，基于聚合统计类的特征。
并且对于时间类的特征，featuretools也可以自动切片。
首先从官方文档的入门教程开始尝试：

import featuretools as ft
 data = ft.demo.load_mock_customer()
customers_df = data["customers"]
customers_df

	customer_id	zip_code	join_date	date_of_birth
0	1	60091	2011-04-17 10:48:33	1994-07-18
1	2	13244	2012-04-15 23:31:04	1986-08-18
2	3	13244	2011-08-13 15:42:34	2003-11-21
3	4	60091	2011-04-08 20:08:14	2006-08-15
4	5	60091	2010-07-17 05:27:50	1984-07-28

sessions_df = data["sessions"]

sessions_df.head()

	session_id	customer_id	device	session_start
0	1	2	desktop	2014-01-01 00:00:00
1	2	5	mobile	2014-01-01 00:17:20
2	3	4	mobile	2014-01-01 00:28:10
3	4	1	mobile	2014-01-01 00:44:25
4	5	4	mobile	2014-01-01 01:11:30

transactions_df = data["transactions"]
transactions_df.head()

	transaction_id	session_id	transaction_time	product_id	amount
0	298	1	2014-01-01 00:00:00	5	127.64
1	2	1	2014-01-01 00:01:05	2	109.48
2	308	1	2014-01-01 00:02:10	3	95.06
3	116	1	2014-01-01 00:03:15	4	78.92
4	371	1	2014-01-01 00:04:20	3	31.54

这里有三张featuretools自带的数据表，三张数据表分别可以采用外键进行关联。 一般常用的特征工程方式是采用merge将session_id,customer_id,进行关联。然后在一张大表上进行特征工程。
当表中数据量较大时，在merge时可能需要耗费较多的时间，且当表数目增加以后，多表关联需要耗费较多时间。

接下来采用featuretools将三张数据表构成实体集，创建字典映射，实体名:{表名，索引}。

entities = {
    "customers" : (customers_df, "customer_id"),
   "sessions" : (sessions_df, "session_id", "session_start"),
  "transactions" : (transactions_df, "transaction_id", "transaction_time")
}

在entities实体集中的目前并没有相关联，因此将三张表创建关联。
创建relationships关联。

relationships = [("sessions", "session_id", "transactions", "session_id"),("customers", "customer_id", "sessions", "customer_id")]

创建关联后即可用采用ft.dfs进行特征生成。
设置主实体为‘customers’。

feature_matrix_customers, features_defs = ft.dfs(entities=entities,relationships=relationships,target_entity="customers")

可用通过查看生成的表格

feature_matrix_customers.head()

查看生成的都有什么特征

features_defs

[<Feature: zip_code>,
 <Feature: COUNT(sessions)>,
 <Feature: NUM_UNIQUE(sessions.device)>,
 <Feature: MODE(sessions.device)>,
 <Feature: SUM(transactions.amount)>,
 <Feature: STD(transactions.amount)>,
 <Feature: MAX(transactions.amount)>,
 <Feature: SKEW(transactions.amount)>,
 <Feature: MIN(transactions.amount)>,
 <Feature: MEAN(transactions.amount)>,
 <Feature: COUNT(transactions)>,
 <Feature: NUM_UNIQUE(transactions.product_id)>,
 <Feature: MODE(transactions.product_id)>,
 <Feature: DAY(date_of_birth)>,
 <Feature: DAY(join_date)>,
 <Feature: YEAR(date_of_birth)>,
 <Feature: YEAR(join_date)>,
 <Feature: MONTH(date_of_birth)>,
 <Feature: MONTH(join_date)>,
 <Feature: WEEKDAY(date_of_birth)>,
 <Feature: WEEKDAY(join_date)>,
 <Feature: SUM(sessions.NUM_UNIQUE(transactions.product_id))>,
 <Feature: SUM(sessions.MIN(transactions.amount))>,
 <Feature: SUM(sessions.MEAN(transactions.amount))>,
 <Feature: SUM(sessions.SKEW(transactions.amount))>,
 <Feature: SUM(sessions.MAX(transactions.amount))>,
 <Feature: SUM(sessions.STD(transactions.amount))>,
 <Feature: STD(sessions.NUM_UNIQUE(transactions.product_id))>,
 <Feature: STD(sessions.COUNT(transactions))>,
 <Feature: STD(sessions.MIN(transactions.amount))>,
 <Feature: STD(sessions.MEAN(transactions.amount))>,
 <Feature: STD(sessions.SKEW(transactions.amount))>,
 <Feature: STD(sessions.MAX(transactions.amount))>,
 <Feature: STD(sessions.SUM(transactions.amount))>,
 <Feature: MAX(sessions.NUM_UNIQUE(transactions.product_id))>,
 <Feature: MAX(sessions.COUNT(transactions))>,
 <Feature: MAX(sessions.MIN(transactions.amount))>,
 <Feature: MAX(sessions.MEAN(transactions.amount))>,
 <Feature: MAX(sessions.SKEW(transactions.amount))>,
 <Feature: MAX(sessions.SUM(transactions.amount))>,
 <Feature: MAX(sessions.STD(transactions.amount))>,
 <Feature: SKEW(sessions.NUM_UNIQUE(transactions.product_id))>,
 <Feature: SKEW(sessions.COUNT(transactions))>,
 <Feature: SKEW(sessions.MIN(transactions.amount))>,
 <Feature: SKEW(sessions.MEAN(transactions.amount))>,
 <Feature: SKEW(sessions.MAX(transactions.amount))>,
 <Feature: SKEW(sessions.SUM(transactions.amount))>,
 <Feature: SKEW(sessions.STD(transactions.amount))>,
 <Feature: MIN(sessions.NUM_UNIQUE(transactions.product_id))>,
 <Feature: MIN(sessions.COUNT(transactions))>,
 <Feature: MIN(sessions.MEAN(transactions.amount))>,
 <Feature: MIN(sessions.SKEW(transactions.amount))>,
 <Feature: MIN(sessions.MAX(transactions.amount))>,
 <Feature: MIN(sessions.SUM(transactions.amount))>,
 <Feature: MIN(sessions.STD(transactions.amount))>,
 <Feature: MEAN(sessions.NUM_UNIQUE(transactions.product_id))>,
 <Feature: MEAN(sessions.COUNT(transactions))>,
 <Feature: MEAN(sessions.MIN(transactions.amount))>,
 <Feature: MEAN(sessions.MEAN(transactions.amount))>,
 <Feature: MEAN(sessions.SKEW(transactions.amount))>,
 <Feature: MEAN(sessions.MAX(transactions.amount))>,
 <Feature: MEAN(sessions.SUM(transactions.amount))>,
 <Feature: MEAN(sessions.STD(transactions.amount))>,
 <Feature: NUM_UNIQUE(sessions.MONTH(session_start))>,
 <Feature: NUM_UNIQUE(sessions.WEEKDAY(session_start))>,
 <Feature: NUM_UNIQUE(sessions.YEAR(session_start))>,
 <Feature: NUM_UNIQUE(sessions.MODE(transactions.product_id))>,
 <Feature: NUM_UNIQUE(sessions.DAY(session_start))>,
 <Feature: MODE(sessions.MONTH(session_start))>,
 <Feature: MODE(sessions.WEEKDAY(session_start))>,
 <Feature: MODE(sessions.YEAR(session_start))>,
 <Feature: MODE(sessions.MODE(transactions.product_id))>,
 <Feature: MODE(sessions.DAY(session_start))>,
 <Feature: NUM_UNIQUE(transactions.sessions.customer_id)>,
 <Feature: NUM_UNIQUE(transactions.sessions.device)>,
 <Feature: MODE(transactions.sessions.customer_id)>,
 <Feature: MODE(transactions.sessions.device)>]

总结：

• 其中发现基本的聚合统计特征都可以生成，std，mean等常用的统计方法，而且最后做到了二阶的特征交叉，省去了特征构造的麻烦，而且对于时间特征，都可以自动生成，例如周几，第几天的特征，省去了平时简单却耗时的特征工程。
• 且当有新的数据表想加入时，只需要继续往齐总加入新的实体与新的关联。
• 但是其同时存在一定问题，当设置的max_depth>2时，可能会导致数据维度急速升高，且对于生成的特征列，难以解释，且可能同样生成了大量无用的数据列，对模型生成一定量的噪声，影响模型结果。
• 常用的数据特征降维方法如PCA，SVD，随机森林选择，神经网络生成等方式对数据进行降维。