lapply 在大型 data.table 上使用大型列表

Posted 2023-04-18

【中文标题】lapply 在大型 data.table 上使用大型列表

【英文标题】：lapply with a large list over a large data.table

【发布时间】：2018-03-30 16:47:09

【问题描述】：

我正在处理一个相当大的数据集“mutualhold”（约 170 行），其中包含 16881 个独特投资基金的月度信息，每个基金都有许多不同的持股。

dput(head(mutualhold,5))
structure(list(crsp_portno = c(1003678L, 1003678L, 1003678L, 
1003678L, 1003678L), report_dt = c("31/12/2001", "31/12/2001", 
"31/12/2001", "31/12/2001", "31/12/2001"), eff_dt = c("31/12/2001", 
"31/12/2001", "31/12/2001", "31/12/2001", "31/12/2001"), percent_tna = c(3.88, 
3.47, 2.64, 2.5, 2.48), cusip = c("36960410", "59491810", "30231G10", 
"93114210", "17296710"), permno = c(12060L, 10107L, 11850L, 55976L, 
70519L)), .Names = c("crsp_portno", "report_dt", "eff_dt", "percent_tna", 
"cusip", "permno"), class = c("data.table", "data.frame"), row.names = c(NA, 
-5L), .internal.selfref = <pointer: 0x00000000047d0788>)

dput(tail(mutualhold,5))
structure(list(crsp_portno = c(1050207L, 1050207L, 1050207L, 
1050207L, 1050207L), report_dt = c("30/11/2017", "30/11/2017", 
"30/11/2017", "30/11/2017", "30/11/2017"), eff_dt = c("21/12/2017", 
"21/12/2017", "21/12/2017", "21/12/2017", "21/12/2017"), percent_tna = c(0.03, 
0.03, 0.03, 0.03, 0.02), cusip = c("92553P20", "65122910", "90187B40", 
"05722G100", "G5785G10"), permno = c(91063L, 60986L, 93070L, 
NA, 14011L)), .Names = c("crsp_portno", "report_dt", "eff_dt", 
"percent_tna", "cusip", "permno"), class = c("data.table", "data.frame"
), row.names = c(NA, -5L), .internal.selfref = <pointer: 0x00000000047d0788>)

我的目标是提取列“report_dt”、“cusip”和“percent_tna”“crsp_portno”并将它们存储在一个列表中。最终列表的长度应为 16881，并包含 data.tables，其中包含每个“crsp_portno”的提取值。我的第一个预感是用 lapply 做到这一点，这当然是可能的：

require(data.table)
sample <- list(1003678L, 1050207L)
tnas <- lapply(sample, function(x) mutualhold[crsp_portno %in% x, .(report_dt, percent_tna, cusip)])

这是可行的，但速度很慢，我不知道我是否能够使用结果列表有效地执行进一步的操作。我非常感谢有关执行此类操作的更有效方法的建议，如有必要，我可以提供更大的样本，但此数据不公开，因此很遗憾我无法在此处分享。

【参考方案1】：

编辑：根据@Frank 的建议，您可以在拆分数据表时使用by 而不是f 作为参数名称。

按crsp_portno拆分数据

split(x = setDT(mutualhold)[, .(report_dt, cusip, percent_tna, crsp_portno)], by = 'crsp_portno' )

数据：

mutualhold <- structure(list(crsp_portno = c(1003678L, 1003678L, 1003678L, 1003678L, 1003678L), report_dt = c("31/12/2001", "31/12/2001", "31/12/2001", "31/12/2001", "31/12/2001"), eff_dt = c("31/12/2001", "31/12/2001", "31/12/2001", "31/12/2001", "31/12/2001"), percent_tna = c(3.88, 3.47, 2.64, 2.5, 2.48), cusip = c("36960410", "59491810", "30231G10", "93114210", "17296710"), permno = c(12060L, 10107L, 11850L, 55976L, 70519L)), .Names = c("crsp_portno", "report_dt", "eff_dt", "percent_tna", "cusip", "permno"), class = c("data.table", "data.frame"), row.names = c(NA, -5L))
mutualhold <- rbind(mutualhold, structure(list(crsp_portno = c(1050207L, 1050207L, 1050207L, 1050207L, 1050207L), report_dt = c("30/11/2017", "30/11/2017", "30/11/2017", "30/11/2017", "30/11/2017"), eff_dt = c("21/12/2017", "21/12/2017", "21/12/2017", "21/12/2017", "21/12/2017"), percent_tna = c(0.03, 0.03, 0.03, 0.03, 0.02), cusip = c("92553P20", "65122910", "90187B40", "05722G100", "G5785G10"), permno = c(91063L, 60986L, 93070L, NA, 14011L)), .Names = c("crsp_portno", "report_dt", "eff_dt", "percent_tna", "cusip", "permno"), class = c("data.table", "data.frame"), row.names = c(NA, -5L)))

【讨论】：

这太棒了！作为比较：使用 lapply，仅使用 50-100 个 ID 处理完全相同的事情需要更长的时间。使用 split 时，这只是几秒钟的事情。我对 data.table 包的效率感到惊讶，感谢您的快速解决方案:)