使用聚类从文档列表中查找所有潜在的相似文档

Posted 2023-03-12

【中文标题】使用聚类从文档列表中查找所有潜在的相似文档

【英文标题】：Find all potential similar documents out of a list of documents using clustering

【发布时间】：2019-06-23 01:16:03

【问题描述】：

我正在使用我加载到 pd 数据框中的 quora 问题对 csv 文件并隔离 qid 和问题，因此我的问题采用这种形式：

0        What is the step by step guide to invest in sh...
1        What is the step by step guide to invest in sh...
2        What is the story of Kohinoor (Koh-i-Noor) Dia...
3        What would happen if the Indian government sto...
.....
19408    What are the steps to solve this equation: [ma...
19409                           Is IMS noida good for BCA?
19410              How good is IMS Noida for studying BCA?

我的数据集实际上更大（50 万个问题），但我将使用这些问题来展示我的问题。

我想识别出很有可能提出相同问题的成对问题。我想到了一种简单的方法，即使用 doc2vec 将每个句子转换为向量，然后为每个句子计算与其他句子的余弦相似度。然后，保留相似度最高的那个，最后打印所有具有足够高余弦相似度的那个。问题是这需要很长时间才能完成，所以我需要另一种方法。

然后我在另一个建议使用聚类解决类似问题的问题中找到了答案。因此，以下是我根据该答案实现的代码。

"Load and transform the dataframe to a new one with only question ids and questions"
train_df = pd.read_csv("test.csv", encoding='utf-8')

questions_df=pd.wide_to_long(train_df,['qid','question'],i=['id'],j='drop')
questions_df=questions_df.drop_duplicates(['qid','question'])[['qid','question']]
questions_df.sort_values("qid", inplace=True)
questions_df=questions_df.reset_index(drop=True)

print(questions_df['question'])

# vectorization of the texts
vectorizer = TfidfVectorizer(stop_words="english")
X = vectorizer.fit_transform(questions_df['question'].values.astype('U'))
# used words (axis in our multi-dimensional space)
words = vectorizer.get_feature_names()
print("words", words)


n_clusters=30
number_of_seeds_to_try=10
max_iter = 300
number_of_process=2 # seads are distributed
model = KMeans(n_clusters=n_clusters, max_iter=max_iter, n_init=number_of_seeds_to_try, n_jobs=number_of_process).fit(X)

labels = model.labels_
# indices of preferable words in each cluster
ordered_words = model.cluster_centers_.argsort()[:, ::-1]

print("centers:", model.cluster_centers_)
print("labels", labels)
print("intertia:", model.inertia_)

texts_per_cluster = numpy.zeros(n_clusters)
for i_cluster in range(n_clusters):
    for label in labels:
        if label==i_cluster:
            texts_per_cluster[i_cluster] +=1

print("Top words per cluster:")
for i_cluster in range(n_clusters):
    print("Cluster:", i_cluster, "texts:", int(texts_per_cluster[i_cluster])),
    for term in ordered_words[i_cluster, :10]:
        print("\t"+words[term])

print("\n")
print("Prediction")

text_to_predict = "Why did Donald Trump win the elections?"
Y = vectorizer.transform([text_to_predict])
predicted_cluster = model.predict(Y)[0]
texts_per_cluster[predicted_cluster]+=1

print(text_to_predict)
print("Cluster:", predicted_cluster, "texts:", int(texts_per_cluster[predicted_cluster])),
for term in ordered_words[predicted_cluster, :10]:
    print("\t"+words[term])

我认为这样我可以为每个句子找到它最可能属于的集群，然后计算该集群的所有其他问题之间的余弦相似度。这种方式不是在所有数据集上做，而是在更少的文档上做。 However using the code for an example sentence "Why did Donald Trump win the elections?"我有以下结果。

Prediction
Why did Donald Trump win the elections?
Cluster: 25 texts: 244
    trump
    donald
    clinton
    hillary
    president
    vote
    win
    election
    did
    think

我知道我的句子属于集群 25，并且我可以看到该集群的顶部单词。但是，我如何访问该集群中的句子。有什么办法吗？

【参考方案1】：

您可以使用predict 获取集群。然后使用numpy 获取特定集群中的所有文档

clusters = model.fit_predict(X_train)

clusterX = np.where(clusters==0) 

indices = X_train[clusterX]

所以现在indices 将拥有该集群中文档的所有索引

【讨论】：

我的示例中的 X_train 是什么？

questions_df['question'].values.astype('U') 或 X

@thelaw 如果您的问题得到解答，请接受答案