如何从文本中删除非UTF-8字符

Posted 2021-05-07

我需要帮助从词云中删除非UTF-8字符。到目前为止，这是我的代码。我已经尝试了gsub和removeWords，它们仍然存在于我的词云中，但我不知道该怎么做才能摆脱它们。任何帮助，将不胜感激。谢谢您的时间。

txt <- readLines("11-0.txt")
corpus = VCorpus(VectorSource(txt))
gsub("’","‘","",txt)

corpus = tm_map(corpus, content_transformer(tolower))
corpus = tm_map(corpus, removeWords, stopwords("english"))
corpus = tm_map(corpus, removePunctuation)
corpus = tm_map(corpus, stripWhitespace) 
corpus = tm_map(corpus, removeWords, c("gutenberg","gutenbergtm","â€","project"))

tdm = TermDocumentMatrix(corpus)
m = as.matrix(tdm)
v = sort(rowSums(m),decreasing = TRUE)
d = data.frame(word=names(v),freq=v)

wordcloud(d$word,d$freq,max.words = 20, random.order=FALSE, rot.per=0.2, colors=brewer.pal(8, "Dark2"))

编辑：这是我的inconv版本

txt <- readLines("11-0.txt")
Encoding(txt) <- "latin1"
iconv(txt, "latin1", "ASCII", sub="")

corpus = VCorpus(VectorSource(txt))
corpus = tm_map(corpus, content_transformer(tolower))
corpus = tm_map(corpus, removeWords, stopwords("english"))
corpus = tm_map(corpus, removePunctuation)
corpus = tm_map(corpus, stripWhitespace) 
corpus = tm_map(corpus, removeWords, c("gutenberg","gutenbergtm","project"))

tdm = TermDocumentMatrix(corpus)
m = as.matrix(tdm)
v = sort(rowSums(m),decreasing = TRUE)
d = data.frame(word=names(v),freq=v)

wordcloud(d$word,d$freq,max.words = 20, random.order=FALSE, rot.per=0.2, colors=brewer.pal(8, "Dark2"))
title(main="Alice in Wonderland word cloud",font.main=1,cex.main =1.5)

答案

gsub的签名是：

gsub（模式，替换，x，ignore.case = FALSE，perl = FALSE，固定= FALSE，useBytes = FALSE）

不确定您想做什么

gsub（“’”，“—，“”，txt）

但是该行可能没有按照您想要的去做...

关于gsub和非ascii符号的先前SO问题，请参阅here。>>

编辑：

使用iconv的建议解决方案：

正在删除

所有非ASCII字符：

txt <- "’xxx‘"

iconv(txt, "latin1", "ASCII", sub="")
返回：

[1] "xxx"