短语的无监督语义聚类
Posted
技术标签:
【中文标题】短语的无监督语义聚类【英文标题】：unsupervised semantic clustering of phrases 【发布时间】：2014-08-06 04:51:29 【问题描述】：我有大约一千个潜在的调查项目作为字符串向量，我想减少到几百个。通常，当我们谈论数据缩减时，我们有实际数据。我将项目管理给参与者，并使用因子分析、PCA 或其他一些降维方法。
就我而言，我没有任何数据。只是项目（即文本字符串）。我想通过消除具有相似含义的项目来减少集合。据推测，如果实际对参与者进行管理，它们将高度相关。
我一直在阅读有关文本分析的聚类方法。这个SO question 演示了我在不同示例中看到的一种方法。 OP 指出集群解决方案并不能完全回答他/她的问题。以下是在我的案例中（不令人满意）的应用方式：
# get data (2 columns, 152 rows)
link to text.R 包含样本项 dput() 的文件
# clustering
library(tm)
library(Matrix)
x <- TermDocumentMatrix( Corpus( VectorSource(text$item) ) )
y <- sparseMatrix( i=x$i, j=x$j, x=x$v, dimnames = dimnames(x) )
plot( hclust(dist(t(y))) )
该图显示项目 145 和 149 是聚类的：
145 “让你知道你不想要”
149 “让你知道他爱你”
这些项目共享相同的词干，“让你知道”，这可能是集群的原因。从语义上讲，它们是对立的。
OP 在他/她的示例中遇到了类似的挑战。一位评论者指出wordnet 包是一种可能的解决方案。
问题（根据反馈编辑）
如何防止像 145 和 149 这样的项目因为共享词干而聚集？
一个不太注重程序化的次要问题：有人在这里看到更好的解决方案吗？我遇到的许多方法都涉及监督学习、测试/训练数据集和分类。我相信我正在寻找的是更多的语义相似性/聚类（例如，FAC pdf）。
【问题讨论】：
您可以删除一些停用词。 Mathew Jockers 使用了一种方法，即删除除可能有用的名词之外的所有内容。 是什么促使人们投票关闭？我提供了一个最小的样本数据集，包括我尝试过的代码，解释了为什么代码没有产生我正在寻找的结果，并询问有关替代方案的想法。这里肯定有一个概念性的文章，但我认为以前遇到过这个问题的人可以在 R 中提供一个实现目标的编程解决方案。人群一定最了解关闭，但我有点困惑。 我没有投票结束，但问题可能更多是关于内容，而不是关于编码。也许可以进行编辑以使问题更接近于编码。 谢谢，@TylerRinker。我编辑了这个问题，专门针对编码挑战。 【参考方案1】：+1 @TylerRinker 对
的建议 删除停用词和 仅使用名词使用 Jockers 的聚类方法（我有一个 here 的工作示例。您应该尝试的另一种选择是使用二元组而不是一元组来制作术语文档矩阵。如果您对短语感兴趣，二元组是一个好的开始。我有一个 here 的工作示例。
这是一个将停用词删除与二元组结合的工作示例。通过这个示例，您可以使用不同的参数值进行迭代，以获得对您来说最合理的聚类。
获取数据...
dat <- text <- structure(list(id = c("GHQ1", "GHQ2", "GHQ3", "GHQ4", "GHQ5",
"GHQ6", "GHQ7", "GHQ8", "GHQ9", "GHQ10", "GHQ11", "GHQ12", "GHQ13",
"GHQ14", "GHQ15", "GHQ16", "GHQ17", "GHQ18", "GHQ19", "GHQ20",
"GHQ21", "GHQ22", "GHQ23", "GHQ24", "CGMH9", "GHQ25", "GHQ26",
"GHQ27", "GHQ28", "GHQ29", "GHQ30", "GHQ31", "PARQ01A-P", "PARQ02A-P",
"PARQ03A-P", "PARQ04A-P", "PARQ05A-P", "PARQ06A-P", "PARQ07A-P",
"PARQ08A-P", "PARQ09A-P", "PARQ10A-P", "PARQ11A-P", "PARQ12A-P",
"PARQ13A-P", "PARQ14A-P", "PARQ15A-P", "PARQ16A-P", "PARQ17A-P",
"PARQ18A-P", "PARQ19A-P", "PARQ20A-P", "PARQ21A-P", "PARQ22A-P",
"PARQ23A-P", "PARQ24A-P", "PARQ25A-P", "PARQ26A-P", "PARQ27A-P",
"PARQ28A-P", "PARQ29A-P", "PARQ30A-P", "PARQ31A-P", "PARQ32A-P",
"PARQ33A-P", "PARQ34A-P", "PARQ35A-P", "PARQ36A-P", "PARQ37A-P",
"PARQ38A-P", "PARQ39A-P", "PARQ40A-P", "PARQ41A-P", "PARQ42A-P",
"PARQ43A-P", "PARQ44A-P", "PARQ45A-P", "PARQ46A-P", "PARQ47A-P",
"PARQ48A-P", "PARQ49A-P", "PARQ50A-P", "PARQ51A-P", "PARQ52A-P",
"PARQ53A-P", "PARQ54A-P", "PARQ55A-P", "PARQ56A-P", "PARQ57A-P",
"PARQ58A-P", "PARQ59A-P", "PARQ60A-P", "PARQ01A-C", "PARQ02A-C",
"PARQ03A-C", "PARQ04A-C", "PARQ05A-C", "PARQ06A-C", "PARQ07A-C",
"PARQ08A-C", "PARQ09A-C", "PARQ10A-C", "PARQ11A-C", "PARQ12A-C",
"PARQ13A-C", "PARQ14A-C", "PARQ15A-C", "PARQ16A-C", "PARQ17A-C",
"PARQ18A-C", "PARQ19A-C", "PARQ20A-C", "PARQ21A-C", "PARQ22A-C",
"PARQ23A-C", "PARQ24A-C", "PARQ25A-C", "PARQ26A-C", "PARQ27A-C",
"PARQ28A-C", "PARQ29A-C", "PARQ30A-C", "PARQ31A-C", "PARQ32A-C",
"PARQ33A-C", "PARQ34A-C", "PARQ35A-C", "PARQ36A-C", "PARQ37A-C",
"PARQ38A-C", "PARQ39A-C", "PARQ40A-C", "PARQ41A-C", "PARQ42A-C",
"PARQ43A-C", "PARQ44A-C", "PARQ45A-C", "PARQ46A-C", "PARQ47A-C",
"PARQ48A-C", "PARQ49A-C", "PARQ50A-C", "PARQ51A-C", "PARQ52A-C",
"PARQ53A-C", "PARQ54A-C", "PARQ55A-C", "PARQ56A-C", "PARQ57A-C",
"PARQ58A-C", "PARQ59A-C", "PARQ60A-C"), item = c("Been feeling unhappy or depressed",
"Been feeling reasonably happy, all things considered", "Feeling edgy and bad-tempered",
"Feel constantly under strain", "Found everything getting on top of you",
"Been feeling nervous and strung-up all the time", "found at times you couldn't do anything because your nerves were too bad",
"found everything getting on top of you", "thought of the possibility that you might make away with yourself",
"found that the idea of taking your own life kept coming into your mind?",
"found yourself withing you were dead and away from it all?",
"felt that life isn't worth living", "felt that life was entirely hopeless?",
"been able to enjoy your normal day-to-day activities", "been satisfied with the way you've carried out your task",
"felt that you are playing a useful part in things", "felt on the whole you were doing things well?",
"been feeling perfectly well and in good health", "been feeling in need of a good tonic",
"been feeling run down and out of sorts?", "felt that you are ill",
"been getting any pains in your head", "been getting a feeling of tightness or pressure in your head",
"been having hot or cold spells", "Do you feel you have physical problems because of stress?",
"Lost sleep over worry", "Had difficulty in staying asleep once you are off",
"felt capable of making decisions about things", "been taking longer over the things that you do",
"been managing to keep yourself busy and occupied", "been thinking of yourself as a worthless person",
"been getting scared or panicky for no good reason ", "You say nice things about your child",
"You nag or scold your child when (s)he is bad", "You ignore your child",
"You wonder if you really love your child", "You talk to your child about daily routines and plans, and listen to what (s)he has to say",
"You complain about your child to others when (s)he does not listen to you",
"You take an interest in your child", "You want your child to bring friends home, and you try to make things pleasant for them",
"You call your child names and make fun of him/her", "You ignore your child as long as (s)he does nothing to bother you",
"You yell at your child when you are angry", "You sit close with your child so that (s)he feels free to talk about important things",
"You are harsh with your child", "You enjoy having your child around you",
"You make your child feel proud when (s)he does well", "Your hit your child even when (s)he may not deserve it, like for small mistakes",
"You forget things you are supposed to do for your child", "You see your child as an annoyance",
"You praise your child to others", "You punish your child when you are angry",
"You make sure your child has the right kind of food to eat",
"You talk to your child in a warm and loving way", "You get angry easily at your child",
"You are too busy to answer your child's questions", "You hate/despise your child",
"You say nice things to your child when (s)he deserves it, such as when (s)he does well in school",
"You are irritable with your child", "You care about who your child's friends are",
"You are really interested in what your child does", "You say many unkind things to your child",
"You pay no attention to your child when (s)he asks for help",
"You think it is your child's own fault when (s)he is having trouble",
"You make your child feel wanted and needed", "You tell your child (s)he annoys you",
"You pay a lot of attention to your child", "You tell your child how proud you are of him/her when (s)he is good",
"You hurt your child's feelings", "You forget important things your child thinks you should remember",
"When your child misbehaves, you make him/her feel unloved",
"You make your child feel what (s)he does is important", "When your child does something wrong, you frighten or threaten him/her",
"You like to spend time with your child, for example you sit and laugh together",
"You try to help your child when (s)he is scared or upset", "When your child misbehaves, you shame him/her in front of his/her friends",
"You avoid your child's company", "You complain about your child",
"You care about what your child thinks, and encourage him/her to talk about it",
"You feel other children are better than your own child", "When you make plans, you take your child's thoughts into consideration",
"You let your child do things (s)he thinks are important, even if it is hard for you",
"When your child misbehaves, you compare him/her unfavorably with other children",
"You want to leave your child in someone else's care (for example, a neighbor or relative)",
"You let your child know (s)he is not wanted", "You are interested in the things your child does",
"You try to make your child feel better when (s)he is hurt or sick",
"You tell your child you are ashamed of him/her when (s)he misbehaves",
"You let your child know you love him/her", "You treat your child gently and with kindness",
"When your child misbehaves, you make him/her feel ashamed or guilty",
"You try to make your child happy", "Says nice things about you",
"Nags or scolds you when you are bad", "Ignores you", "Does not really love you",
"Talks to you about your plans and listens to what you have to say",
"Complains about you to others when you do not listen to him",
"Takes an interest in you", "Wants you to bring your friends home, and tries to make things pleasant for them",
"Calls you names, ridicules you, and makes fun of you", "Ignores you as long as you do nothing to bother him",
"Yells at you when he is angry", "Sits close with you so that you feel free to talk about important things",
"Treats you harshly", "Enjoys having you around him", "Make you feel proud when you do well",
"Hits you even when you do not deserve it, like for small mistakes",
"Forgets things he is supposed to do for you", "Sees you as an annoyance",
"Praises you to others", "Punishes you severely when he is angry",
"Makes sure you have the right kind of food to eat", "Talks to you in a warm and loving way",
"Gets angry at you easily", "Is too busy to answer your questions",
"Seems to hate / despise you", "Says nice things to you when you deserve them, such as when you do well in school",
"Gets mad quickly and picks on you", "Wants to know who your friends are",
"Is really interested in what you do", "Says many unkind things to you",
"Pays no attention when you ask for help", "Thinks it is your own fault when you are having trouble",
"Makes you feel wanted and needed", "Tells you that you annoy him",
"Pays a lot of attention to you", "Tells you how proud he is of you when you are good",
"Goes out of his way to hurt your feelings", "Forgets important things you think he should remember",
"Makes you feel unloved if you misbehave", "Makes you feel what you do is important",
"Frightens or threatens you when you do something wrong", "Likes to spend time with you, for example you sit and laugh together",
"Tries to help you when you are scared or upset", "Shames you in front of your friends when you misbehave",
"Tries to stay away from you", "Complains about you and talks about you behind your back",
"Cares about what you think, and likes you to talk about it",
"Feels other children are better than you are no matter what you do",
"Cares about what you would like when he makes plans", "Lets you do things you think are important, even if it is hard for him",
"Thinks other children behave better than you do", "Wants other people to take care of you (for example, a neighbor or relative)",
"Lets you know you are not wanted", "Is interested in the things you do",
"Shows concern and tries to make you feel better when you are hurt or sick",
"Tells you how ashamed he is when you misbehave", "Lets you know he loves you",
"Treats you gently and with kindness", "Makes you feel ashamed or guilty when you misbehave",
"Tries to make you happy")), .Names = c("id", "item"), row.names = c(NA,
152L), class = "data.frame")
现在制作一个二元组的 tdm，然后删除包含停用词的二元组...
library("RWeka")
library("tm")
library("Matrix")
BigramTokenizer <- function(x) NGramTokenizer(x, Weka_control(min = 2, max = 2))
x <- TermDocumentMatrix(Corpus(VectorSource(text$item)), control = list(tokenize = BigramTokenizer))
# little bit of regex to remove bigrams with stopwords in them, cf. https://***.com/a/6947724/1036500
stpwrds <- paste(stopwords("en"), collapse = "|")
x$dimnames$Terms[!grepl(stpwrds, x$dimnames$Terms)]
[1] "cold spells" "else s" "feel free" "feel proud" [5] "feel unloved" "feels free" "lost sleep"
使用tm 软件包附带的停用词库存列表快速测试删除二元组，结果表明它只给我们留下了 8 个二元组！显然我们需要一个更小的停用词列表，所以让我们通过在这个特定语料库中查找最常见的词并删除它们来创建一个自定义列表。
# find freq words in corpus
x <- TermDocumentMatrix(Corpus(VectorSource(text$item)))
# arbitrary choice of 10 occurances = hi freq
mystopwords <- findFreqTerms(x, 10, Inf)
您应该尝试使用 lowfreq 值，我尝试了几次后将其设置为 10，但其他值可能会更好。
# try to filter the bigrams again with custom stopword list
x <- TermDocumentMatrix(Corpus(VectorSource(text$item)), control = list(tokenize = BigramTokenizer))
# little bit of regex to remove bigrams with mystopwords in them, cf. https://***.com/a/6947724/1036500
mystpwrds <- paste(mystopwords, collapse = "|")
# subset tdm to keep only bigrams remaining after mystopwords removed
x <- x[x$dimnames$Terms[!grepl(mystpwrds, x$dimnames$Terms)],]
y <- sparseMatrix( i=x$i, j=x$j, x=x$v, dimnames = dimnames(x) )
plot( hclust(dist(t(y))) )
但这有点难读，所以让我们像这样打印组成员
hc <- hclust(dist(t(y)))
cutree(hc, k = 100)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
1 2 3 4 5 3 6 5 3 7 8 9 10 11 12 13
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
14 15 16 17 3 18 19 20 21 22 23 24 25 26 27 28
33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
3 29 3 30 31 32 33 34 35 36 3 37 3 3 38 39
49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64
40 41 3 3 42 43 44 45 3 46 3 3 47 48 49 50
65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80
3 51 52 53 3 3 3 54 55 56 57 58 3 3 3 59
81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96
60 61 3 62 63 47 64 65 3 3 66 3 3 67 3 68
97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112
69 70 33 71 35 72 73 74 3 75 3 76 40 41 3 73
113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128
42 43 77 45 78 79 80 81 47 48 82 83 3 3 84 85
129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144
86 87 3 88 89 90 91 92 93 3 3 59 3 94 95 96
145 146 147 148 149 150 151 152
3 97 98 99 100 3 66 3
我们看到我们在不同的组中有第 145 行和第 149 行。这是否是一个好的答案很难说，因为你没有指定你想要的输出应该是什么。这就是为什么您获得接近投票的原因，很难从您的问题中推断出一个好的答案是什么（更具体地说，问题是您要求：“推荐或查找工具、库或最喜欢的场外资源”，这会导致自以为是的答案），SO 人群似乎更喜欢所需输出的具体示例。您可以在新的data science stack exchange site 上尝试您的问题。
无论如何，希望您现在有了更多的想法和更多的技巧来探索数据。如果您遇到与此相关的特定编程问题，请随时提出另一个问题。
【讨论】：
感谢您花时间创建一个非常有用的答案。我现在正在经历它。在x <- TermDocumentMatrix(Corpus(VectorSource(text$item)), control = list(tokenize = BigramTokenizer)) 上出现错误：Error in rep(seq_along(x), sapply(tflist, length)) : invalid 'times' argument In addition: Warning message: In is.na(x) : is.na() applied to non-(list or vector) of type 'NULL'
可能与并行化或 Java 有关。这是一个常见的错误，毫无疑问你已经在 google 上搜索过了：***.com/q/17703553/1036500 & ***.com/q/19615181/1036500
是的，在调用 NGramTokenizer() 之前设置 options(mc.cores=1) 就可以了。
我认为以下错误不应特定于我的设置：x <- x[x$dimnames$Terms[!grepl(mystpwrds, x$dimnames$Terms)]] 给出Error in x$nrow : $ operator is invalid for atomic vectors。内部部分，x$dimnames$Terms[!grepl(mystpwrds, x$dimnames$Terms)]，有效。添加逗号使其运行：x <- x[x$dimnames$Terms[!grepl(mystpwrds, x$dimnames$Terms)],].
很好，我已经编辑了我的答案。这能回答你的问题吗？以上是关于短语的无监督语义聚类的主要内容，如果未能解决你的问题，请参考以下文章