识别和标记重复字符

Posted 2023-03-29

【中文标题】识别和标记重复字符

【英文标题】：Identify and mark character duplicates

【发布时间】：2022-01-16 19:49:56

【问题描述】：

我有一个大型数据集，包含几百万行和几列。两列应该是唯一的 ID。但是有多个重复项。 这是我数据集的一个小样本（前 10 行，目前在 N 或 C 中没有任何重复）：

   inschrijf_last                     Pseudoniem_N                     Pseudoniem_C Verwijder
10     2020-10-01 UG2N006a768jvr18l2i1jahj8lba0gkp UG2C00780vgkrb6ducv16n4hm0jcvb7g         0
14     2020-10-01 UG2N005htkh5ptahdqidvptb85h9c3sc UG2C006mc1mjciccnvtdoke65lu75c71         0
19           <NA> UG2N0035cscm05jojivdt85av0sh34vo UG2C000r2dhjudrt679v15t4a5a9htnl         1
25     2020-10-01 UG2N005evrtnube8939bojvnmeevkkkc UG2C0035j36uuosv0l4250ctj1m4rm2u         0
26           <NA> UG2N007352gn5be17ppemn19dhf7h1fj UG2C                                     1
27     2020-10-01 UG2N0023reb9qr1t8lr8k11hnolu49b3 UG2C00361k53ni7u3dfge25neob23lrk         0
30     2016-01-01 UG2N002tv1i0s76h4ljehu2tj03g61a8 UG2C004jvu3f3ig9pbmi2tga0uc7db30         0
33     2020-10-01 UG2N0073i8mi4rmq8dqlrse6vk6t81sg UG2C001g3tla577jnh3s617thjh97631         0
34     2018-01-01 UG2N005ut027j93cbmhoa70nmnadjp6g UG2C001tubiclm75f9nad25pads9eata         0
35           <NA> UG2N0076de059qhm989ge9e7agjosmek UG2C005r9p6arkr3foi4b4ma5nbl6s2p         1

列 Pseudoniem_N 和 Pseudoniem_C 是“字符”。 我想识别并标记 3 个可选的重复项：

1: If Pseudoniem_N = duplicate -> identify and mark
2: If pseudoniem_C = duplicate -> identify and mark 
3: If pseudoniem N AND C both duplicate -> identify and mark
Preferably in three different variables/columns

想要的结果 我想添加这些额外的变量来标记（字符）重复 N、C 和 N + C。下面我做了一个小例子。

 inschrijf_last   Pseudoniem_N      Pseudoniem_C        Duplicate N      Duplicate C Duplicate N+C
10   2020-10-01    aaaaaaaa          aaaabbbb              1                   1         1
14   2020-10-01    aaaaaaaa          aaaabbbb              1                   1         1
19      <NA>       bbbbbbbb          bbbbaaaa              2                   
25   2020-10-01    bbbbbbbb          aaaacccc              2                   2
26      <NA>       cccccccc          aaaacccc                                  2
etc.

非常感谢您！

【问题讨论】：

使用data.table 包（因为您的数据集很大）并查看search.r-project.org/CRAN/refmans/data.table/html/…

【参考方案1】：

根据你想要的，我认为这就是你正在寻找的：

my_df <- data.frame("ID" = seq.int(from = 1, to = 6, by = 1),
                "Date" = rep("01/01/2021", 6),
                "Pseudoniem_N" = c("aaa", "aaa", "bbb", "bbb", "ccc", "ddd"),
                "Pseudoniem_C" = c("aab", "aab", "bba", "aac", "aac", "ddd"),
                stringsAsFactors = FALSE)


my_df <- my_df %>% group_by(Pseudoniem_N) %>% mutate("Dup_N" = cur_group_id())
my_df <- my_df %>% group_by(Pseudoniem_C) %>% mutate("Dup_C" = cur_group_id())
my_df <- my_df %>% group_by(Pseudoniem_N, Pseudoniem_C) %>% mutate("Dup_NC" = cur_group_id())

my_var <- which(!(duplicated(my_df$Pseudoniem_N) | duplicated(my_df$Pseudoniem_N, fromLast = TRUE)))
if (length(my_var)) 
 my_df[my_var, "Dup_N"] <- NA

my_var <- which(!(duplicated(my_df$Pseudoniem_C) | duplicated(my_df$Pseudoniem_C, fromLast = TRUE)))
if (length(my_var)) 
my_df[my_var, "Dup_C"] <- NA

my_var <- which(!(duplicated(my_df[, c("Pseudoniem_N", "Pseudoniem_C")]) | duplicated(my_df[, c("Pseudoniem_N", "Pseudoniem_C")], fromLast = TRUE)))
if (length(my_var)) 
my_df[my_var, "Dup_NC"] <- NA