tm：读入数据框，保留文本ID，构建DTM并加入其他数据集
Posted
技术标签:
【中文标题】tm：读入数据框，保留文本ID，构建DTM并加入其他数据集【英文标题】：tm: read in data frame, keep text id's, construct DTM and join to other dataset 【发布时间】：2013-11-19 23:11:02 【问题描述】：我正在使用包 tm。
假设我有一个 2 列 500 行的数据框。 第一列是随机生成的 ID，其中包含字符和数字：“txF87uyK” 第二列是实际文本：“今天天气很好。约翰去慢跑了。等等，等等，......”
现在我想从这个数据框创建一个文档术语矩阵。
我的问题是我想保留 ID 信息，以便在获得文档术语矩阵后，我可以将此矩阵与另一个矩阵连接起来，该矩阵的每一行都是每个文档的其他信息（日期、主题、情绪），并且每行都由文档 ID 标识。
我该怎么做？
问题一：如何将这个数据框转化为语料库，并保留ID信息？
问题2：得到一个dtm后，如何通过ID加入另一个数据集？
【问题讨论】：
保持 id 列分开。建立 dtm。转换为data.frame。 cbind id 列重新打开。合并 一个可重复的小例子很有帮助。 Q1 有一个答案 here 虽然我已经在下面拼写出来完整。 【参考方案1】：2017 年 12 月对 tm 包进行了更新，readTabular 不见了
"Changes in tm version 0.7-2
SIGNIFICANT USER-VISIBLE CHANGES
DataframeSource now only processes data frames with the two mandatory columns "doc_id" and "text". Additional columns are used as document level metadata. This implements compatibility with Text Interchange Formats corpora (https://github.com/ropensci/tif)."
这使得将每个文档的 ID（以及您需要的任何其他元数据）更容易地放入语料库中，如 https://cran.r-project.org/web/packages/tm/news.html 中所述
【讨论】：
【参考方案2】：首先，来自https://***.com/a/15506875/1036500的一些示例数据
examp1 <- "When discussing performance with colleagues, teaching, sending a bug report or searching for guidance on mailing lists and here on SO, a reproducible example is often asked and always helpful. What are your tips for creating an excellent example? How do you paste data structures from r in a text format? What other information should you include? Are there other tricks in addition to using dput(), dump() or structure()? When should you include library() or require() statements? Which reserved words should one avoid, in addition to c, df, data, etc? How does one make a great r reproducible example?"
examp2 <- "Sometimes the problem really isn't reproducible with a smaller piece of data, no matter how hard you try, and doesn't happen with synthetic data (although it's useful to show how you produced synthetic data sets that did not reproduce the problem, because it rules out some hypotheses). Posting the data to the web somewhere and providing a URL may be necessary. If the data can't be released to the public at large but could be shared at all, then you may be able to offer to e-mail it to interested parties (although this will cut down the number of people who will bother to work on it). I haven't actually seen this done, because people who can't release their data are sensitive about releasing it any form, but it would seem plausible that in some cases one could still post data if it were sufficiently anonymized/scrambled/corrupted slightly in some way. If you can't do either of these then you probably need to hire a consultant to solve your problem"
examp3 <- "You are most likely to get good help with your R problem if you provide a reproducible example. A reproducible example allows someone else to recreate your problem by just copying and pasting R code. There are four things you need to include to make your example reproducible: required packages, data, code, and a description of your R environment. Packages should be loaded at the top of the script, so it's easy to see which ones the example needs. The easiest way to include data in an email is to use dput() to generate the R code to recreate it. For example, to recreate the mtcars dataset in R, I'd perform the following steps: Run dput(mtcars) in R Copy the output In my reproducible script, type mtcars <- then paste. Spend a little bit of time ensuring that your code is easy for others to read: make sure you've used spaces and your variable names are concise, but informative, use comments to indicate where your problem lies, do your best to remove everything that is not related to the problem. The shorter your code is, the easier it is to understand. Include the output of sessionInfo() as a comment. This summarises your R environment and makes it easy to check if you're using an out-of-date package. You can check you have actually made a reproducible example by starting up a fresh R session and pasting your script in. Before putting all of your code in an email, consider putting it on http://gist.github.com/. It will give your code nice syntax highlighting, and you don't have to worry about anything getting mangled by the email system."
examp4 <- "Do your homework before posting: If it is clear that you have done basic background research, you are far more likely to get an informative response. See also Further Resources further down this page. Do help.search(keyword) and apropos(keyword) with different keywords (type this at the R prompt). Do RSiteSearch(keyword) with different keywords (at the R prompt) to search R functions, contributed packages and R-Help postings. See ?RSiteSearch for further options and to restrict searches. Read the online help for relevant functions (type ?functionname, e.g., ?prod, at the R prompt) If something seems to have changed in R, look in the latest NEWS file on CRAN for information about it. Search the R-faq and the R-windows-faq if it might be relevant (http://cran.r-project.org/faqs.html) Read at least the relevant section in An Introduction to R If the function is from a package accompanying a book, e.g., the MASS package, consult the book before posting. The R Wiki has a section on finding functions and documentation"
examp5 <- "Before asking a technical question by e-mail, or in a newsgroup, or on a website chat board, do the following: Try to find an answer by searching the archives of the forum you plan to post to. Try to find an answer by searching the Web. Try to find an answer by reading the manual. Try to find an answer by reading a FAQ. Try to find an answer by inspection or experimentation. Try to find an answer by asking a skilled friend. If you're a programmer, try to find an answer by reading the source code. When you ask your question, display the fact that you have done these things first; this will help establish that you're not being a lazy sponge and wasting people's time. Better yet, display what you have learned from doing these things. We like answering questions for people who have demonstrated they can learn from the answers. Use tactics like doing a Google search on the text of whatever error message you get (searching Google groups as well as Web pages). This might well take you straight to fix documentation or a mailing list thread answering your question. Even if it doesn't, saying “I googled on the following phrase but didn't get anything that looked promising” is a good thing to do in e-mail or news postings requesting help, if only because it records what searches won't help. It will also help to direct other people with similar problems to your thread by linking the search terms to what will hopefully be your problem and resolution thread. Take your time. Do not expect to be able to solve a complicated problem with a few seconds of Googling. Read and understand the FAQs, sit back, relax and give the problem some thought before approaching experts. Trust us, they will be able to tell from your questions how much reading and thinking you did, and will be more willing to help if you come prepared. Don't instantly fire your whole arsenal of questions just because your first search turned up no answers (or too many). Prepare your question. Think it through. Hasty-sounding questions get hasty answers, or none at all. The more you do to demonstrate that having put thought and effort into solving your problem before seeking help, the more likely you are to actually get help. Beware of asking the wrong question. If you ask one that is based on faulty assumptions, J. Random Hacker is quite likely to reply with a uselessly literal answer while thinking Stupid question..., and hoping the experience of getting what you asked for rather than what you needed will teach you a lesson."
将示例数据放入数据框中...
df <- data.frame(ID = sapply(1:5, function(i) paste0(sample(letters, 5), collapse = "")),
txt = sapply(1:5, function(i) eval(parse(text=paste0("examp",i))))
)
这是“问题 1：如何将此数据框转换为语料库并保留 ID 信息？”的答案？
使用DataframeSource 和readerControl 将数据帧转换为语料库（来自https://***.com/a/15693766/1036500）...
require(tm)
m <- list(ID = "ID", Content = "txt")
myReader <- readTabular(mapping = m)
mycorpus <- Corpus(DataframeSource(df), readerControl = list(reader = myReader))
# Manually keep ID information from https://***.com/a/14852502/1036500
for (i in 1:length(mycorpus))
attr(mycorpus[[i]], "ID") <- df$ID[i]
现在为您的第二个问题提供一些示例数据...
从https://***.com/a/15506875/1036500...制作文档术语矩阵
skipWords <- function(x) removeWords(x, stopwords("english"))
funcs <- list(content_transformer(tolower), removePunctuation, removeNumbers, stripWhitespace, skipWords)
a <- tm_map(mycorpus, FUN = tm_reduce, tmFuns = funcs)
mydtm <- DocumentTermMatrix(a, control = list(wordLengths = c(3,10)))
inspect(mydtm)
制作另一个示例数据集以加入...
df2 <- data.frame(ID = df$ID,
date = seq(Sys.Date(), length.out=5, by="1 week"),
topic = sapply(1:5, function(i) paste0(sample(LETTERS, 3), collapse = "")) ,
sentiment = sample(c("+ve", "-ve"), 5, replace = TRUE)
)
这里是“问题 2：得到一个 dtm 后，如何通过 ID 将其与另一个数据集连接起来？”的答案？
使用 merge 将 dtm 加入日期、主题、情绪的示例数据集...
mydtm_df <- data.frame(as.matrix(mydtm))
# merge by row.names from https://***.com/a/7739757/1036500
merged <- merge(df2, mydtm_df, by.x = "ID", by.y = "row.names" )
head(merged)
ID date.x topic sentiment able actually addition allows also although
1 cpjmn 2013-11-07 XRT -ve 0 0 2 0 0 0
2 jkdaf 2013-11-28 TYJ -ve 0 0 0 0 1 0
3 jstpa 2013-12-05 SVB -ve 2 1 0 0 1 0
4 sfywr 2013-11-14 OMG -ve 1 1 0 0 0 2
5 ylaqr 2013-11-21 KDY +ve 0 1 0 1 0 0
always answer answering answers anything archives are arsenal ask asked asking
1 1 0 0 0 0 0 1 0 0 1 0
2 0 0 0 0 0 0 0 0 0 0 0
3 0 8 2 3 1 1 0 1 2 1 3
4 0 0 0 0 0 0 0 0 0 0 0
5 0 0 0 0 1 0 0 0 0 0 0
现在你有：
-
回答您的两个问题（通常这个网站每个...问题只有一个问题）
您可以在提出下一个问题时使用多种示例数据（让您的问题对可能想要回答的人更具吸引力）
希望您的问题的答案已经可以在 *** 的 r 标签的其他地方找到，如果您能想到如何将您的问题分解成更小的步骤。
如果这个没有回答您的问题，请提出另一个问题并包含代码以尽可能准确地重现您的用例。如果它确实回答了您的问题，那么您应该mark it as accepted（至少在出现更好的问题之前，例如，Tyler 可能会从他令人印象深刻的qdap 包中弹出一条线。 .)
【讨论】：
我刚刚了解到您将 MWE 上的信息用作 MWE。太棒了:) 在尝试复制您的解决方案时，我注意到您创建的语料库虽然在元数据中有文本，但不包含由 inspect(mycorpus[1] 显示的传统方式的文本)。这是最近对 tm 进行更改的结果吗？谢谢！ @MichaelDavidson 是的，从 tm 0.5 到 0.6 的更改引入了一些破坏此代码的更改。你可以在这里获得 v0.5：cran.r-project.org/src/contrib/Archive/tm @Ben，为什么它像 Micheal 指出的那样在元数据中设置内容，而不是在 PlainTextDocument 的内容属性中？有没有办法让这个解决方案与最新的 tm 包一起工作并拥有文档的内容集，或者有没有办法在内容元数据上使用 TDM？ （尝试创建 TDM 时解决方案失败，错误为“inherits(doc, "TextDocument") is not TRUE"） 这个答案不再积极维护，对不起?【参考方案3】：我也遇到了这个问题，为了改变每个内容的id，建议使用这段代码
for(k in 1:length(mycorpus))
mycorpus[[k]]$meta$id <- mycorpus$ID[k]
【讨论】：
【参考方案4】：在下面的代码中，“内容”应该是小写的，而不是下面例子中的大写。此更改将正确填充语料库的内容字段。
require(tm)
m <- list(ID = "ID", content = "txt")
myReader <- readTabular(mapping = m)
mycorpus <- Corpus(DataframeSource(df), readerControl = list(reader = myReader))
# Manually keep ID information from http://***.com/a/14852502/1036500
for (i in 1:length(mycorpus))
attr(mycorpus[[i]], "ID") <- df$ID[i]
现在试试
mycorpus[[3]]$content
【讨论】：
【参考方案5】：qdap 1.2.0 可以用很少的编码完成这两个任务，虽然不是一个单行 ;-)，并且不一定比 Ben 的快（因为 key_merge 是 merge 的便捷包装器）。从上面使用 Ben 的所有数据（这使得我的答案看起来更小，而它并没有那么小。
## The code
library(qdap)
mycorpus <- with(df, as.Corpus(txt, ID))
mydtm <- as.dtm(Filter(as.wfm(mycorpus,
col1 = "docs", col2 = "text",
stopwords = tm::stopwords("english")), 3, 10))
key_merge(matrix2df(mydtm, "ID"), df2, "ID")
【讨论】：
以上是关于tm：读入数据框，保留文本ID，构建DTM并加入其他数据集的主要内容，如果未能解决你的问题，请参考以下文章