如何在 tidytext 中包含选择的 2 词短语作为标记？

Posted 2023-04-18

【中文标题】如何在 tidytext 中包含选择的 2 词短语作为标记？

【英文标题】：How to include select 2-word phrases as tokens in tidytext?

【发布时间】：2019-08-01 07:28:32

【问题描述】：

我正在预处理一些文本数据以供进一步分析。我使用 unnest_tokens() [成单数词] 将文本标记化，但希望保留某些常见的 2 词短语，例如“美国”或“社会保障”。如何使用 tidytext 做到这一点？

tidy_data <- data %>%
                unnest_tokens(word, text) %>%
                anti_join(stop_words)

dput(data[1:6, 1:6])

structure(list(race = c("US House", "US House", "US House", "US House", 
"", "US House"), district = c(8L, 3L, 6L, 17L, 2L, 1L), party = c("Republican", 
"Republican", "Republican", "Republican", "", "Republican"), 
    state = c("AZ", "AZ", "KY", "TX", "IL", "NH"), sponsor = c(4, 
    4, 4, 1, NA, 4), approve = structure(c(1L, 1L, 1L, 4L, NA, 
    1L), .Label = c("no oral statement of approval, authorization", 
    "beginning of the spot", "middle of the spot", "end of the spot"
    ), class = "factor")), row.names = c(NA, 6L), class = "data.frame")

【参考方案1】：

如果我在这种情况下，我只需要在分析中保留一个简短的两词短语列表，我会在标记化之前和之后进行一些谨慎的替换。

首先，我会将两个单词的短语替换为可以粘在一起且不会被我正在使用的标记化过程分开的内容，例如 "united states" 到 "united_states"。

library(tidyverse)
library(tidytext)


df <- tibble(text = c("I live in the United States",
                      "United we stand, divided we fall",
                      "Information security is important!",
                      "I work at the Social Security Administration"))


df_parsed <- df %>%
  mutate(text = str_to_lower(text),
         text = str_replace_all(text, "united states", "united_states"),
         text = str_replace_all(text, "social security", "social_security"))

df_parsed
#> # A tibble: 4 x 1
#>   text                                        
#>   <chr>                                       
#> 1 i live in the united_states                 
#> 2 united we stand, divided we fall            
#> 3 information security is important!          
#> 4 i work at the social_security administration

然后你可以像往常一样进行标记化，然后再次用两个词的短语替换你刚刚制作的东西，所以"united_states" 回到"united states"。

df_parsed %>%
  unnest_tokens(word, text) %>%
  mutate(word = case_when(word == "united_states" ~ "united states",
                          word == "social_security" ~ "social security",
                          TRUE ~ word))
#> # A tibble: 21 x 1
#>    word         
#>    <chr>        
#>  1 i            
#>  2 live         
#>  3 in           
#>  4 the          
#>  5 united states
#>  6 united       
#>  7 we           
#>  8 stand        
#>  9 divided      
#> 10 we           
#> # … with 11 more rows

^{由reprex package (v0.3.0) 于 2019 年 8 月 3 日创建}

如果您的列表很长，那么这将变得困难和繁重，那么研究使用二元组和一元组标记化的方法可能是有意义的。您可以查看here 的示例。