tidytext：unnest_tokens 和 token = 'ngrams' 的问题

Posted 2023-04-12

【中文标题】tidytext：unnest_tokens 和 token = \'ngrams\' 的问题

【英文标题】：tidytext: Issue with unnest_tokens and token = 'ngrams'tidytext：unnest_tokens 和 token = 'ngrams' 的问题

【发布时间】：2020-01-31 17:57:57

【问题描述】：

我正在运行以下代码

library(rwhatsapp)
library(tidytext)

chat <- rwa_read(x = c(
  "31/1/15 04:10:59 - Menganito: Was it good?",
  "31/1/15 14:10:59 - Fulanito: Yes, it was"
))

chat %>% as_tibble() %>% 
  unnest_tokens(output = bigram, input = text, token = "ngrams", n = 2)

但我收到以下错误：

Error in unnest_tokens.data.frame(., output = bigram, input = text, token = "ngrams",  : 
  If collapse = TRUE (such as for unnesting by sentence or paragraph), unnest_tokens needs all input columns to be atomic vectors (not lists)

我尝试在 Google 上进行一些研究，但找不到答案。列 text 是一个字符向量，所以我不明白为什么我会收到错误消息说它不是。

【参考方案1】：

问题是因为有一些list 列是NULL

str(chat)
#tibble [2 × 6] (S3: tbl_df/tbl/data.frame)
# $ time      : POSIXct[1:2], format: "2015-01-31 04:10:59" "2015-01-31 14:10:59"
# $ author    : Factor w/ 2 levels "Fulanito","Menganito": 2 1
# $ text      : chr [1:2] "Was it good?" "Yes, it was"
# $ source    : chr [1:2] "text input" "text input"
# $ emoji     :List of 2   ###
#  ..$ : NULL
#  ..$ : NULL
# $ emoji_name:List of 2    ###
#  ..$ : NULL
#  ..$ : NULL

我们可以删除它，它现在可以工作了

library(rwhatsapp)
library(tidytext)
chat %>% 
   select_if(~ !is.list(.)) %>%
   unnest_tokens(output = bigram, input = text, token = "ngrams", n = 2)
# A tibble: 4 x 4
#  time                author    source     bigram 
#  <dttm>              <fct>     <chr>      <chr>  
#1 2015-01-31 04:10:59 Menganito text input was it 
#2 2015-01-31 04:10:59 Menganito text input it good
#3 2015-01-31 14:10:59 Fulanito  text input yes it 
#4 2015-01-31 14:10:59 Fulanito  text input it was 

---

此外，默认情况下collapse=TRUE，当有NULL 元素时，这会产生问题，因为当它是collapsed 时长度会有所不同。一种选择是指定collapse = FALSE

chat %>% 
   unnest_tokens(output = bigram, input = text, token = "ngrams",
        n = 2, collapse= FALSE)
# A tibble: 4 x 6
#  time                author    source     emoji  emoji_name bigram 
#  <dttm>              <fct>     <chr>      <list> <list>     <chr>  
#1 2015-01-31 04:10:59 Menganito text input <NULL> <NULL>     was it 
#2 2015-01-31 04:10:59 Menganito text input <NULL> <NULL>     it good
#3 2015-01-31 14:10:59 Fulanito  text input <NULL> <NULL>     yes it 
#4 2015-01-31 14:10:59 Fulanito  text input <NULL> <NULL>     it was