从词频创建 ARFF

Posted 2023-03-13

【中文标题】从词频创建 ARFF

【英文标题】：Creating ARFF from word frequencies

【发布时间】：2011-07-26 21:47:46

【问题描述】：

我有一些代码可以为我提供一个单词列表以及它们在文本中出现的频率，我正在寻找它，以便代码自动将前 10 个单词转换为带有

的 ARFF

@RELATION 词频

@ATTRIBUTE 字串 @ATTRIBUTE 频率数值

以及前 10 个数据及其频率。

我正在为如何使用我当前的代码做到这一点而苦苦挣扎

import re
import nltk

# Quran subset
filename = 'subsetQuran.txt'

# create list of lower case words
word_list = re.split('\s+', file(filename).read().lower())
print 'Words in text:', len(word_list)

word_list2 = [w.strip() for w in word_list if w.strip() not in nltk.corpus.stopwords.words('english')]



# create dictionary of word:frequency pairs
freq_dic = 
# punctuation and numbers to be removed
punctuation = re.compile(r'[-.?!,":;()|0-9]') 
for word in word_list2:
    # remove punctuation marks
    word = punctuation.sub("", word)
    # form dictionary
    try: 
        freq_dic[word] += 1
    except: 
        freq_dic[word] = 1


print '-'*30

print "sorted by highest frequency first:"
# create list of (val, key) tuple pairs
freq_list2 = [(val, key) for key, val in freq_dic.items()]
# sort by val or frequency
freq_list2.sort(reverse=True)
freq_list3 = list(freq_list2)
# display result
for freq, word in freq_list2:
    print word, freq
f = open("wordfreq.txt", "w")
f.write( str(freq_list3) )
f.close()

对此的任何帮助表示赞赏，这样做的方法真的让我绞尽脑汁！

【问题讨论】：

不确定这是否会有所帮助，但它可能会向您展示如何为所有单词制作一个 arff，然后对其进行编辑以仅获取前 10 个单词？ ***.com/questions/5230699/…

【参考方案1】：

我希望你不介意轻微的重写：

import re
import nltk
from collections import defaultdict

# Quran subset
filename = 'subsetQuran.txt'

# create list of lower case words
word_list = open(filename).read().lower().split()
print 'Words in text:', len(word_list)

# remove stopwords
word_list = [w for w in word_list if w not in nltk.corpus.stopwords.words('english')]

# create dictionary of word:frequency pairs
freq_dic = defaultdict(int)

# punctuation and numbers to be removed
punctuation = re.compile(r'[-.?!,":;()|0-9]') 
for word in word_list:
    # remove punctuation marks
    word = punctuation.sub("", word)
    # increment count for word
    freq_dic[word] += 1

print '-' * 30

print "sorted by highest frequency first:"
# create list of (frequency, word) tuple pairs
freq_list = [(freq, word) for word, freq in freq_dic.items()]

# sort by descending frequency
freq_list.sort(reverse=True)

# display result
for freq, word in freq_list:
    print word, freq

# write ARFF file for 10 most common words
f = open("wordfreq.txt", "w")
f.write("@RELATION wordfrequencies\n")
f.write("@ATTRIBUTE word string\n")
f.write("@ATTRIBUTE frequency numeric\n")
f.write("@DATA\n")
for freq, word in freq_list[ : 10]:
    f.write("'%s',%d\n" % (word, freq))
f.close()