综合练习：词频统计

Posted 2020-10-28 孙创锋

1.英文词频统

下载一首英文的歌词或文章

将所有,.？！’:等分隔符全部替换为空格

将所有大写转换为小写

生成单词列表

生成词频统计

排序

排除语法型词汇，代词、冠词、连词

输出词频最大TOP20

将分析对象存为utf-8编码的文件，通过文件读取的方式获得词频分析内容。

news = ‘‘‘《Faded》歌词‘‘‘
sep = ‘‘‘,.?!‘":‘‘‘
exclude = {‘you‘, ‘and‘, ‘to‘}

for c in sep:
    news = news.replace(c,‘‘)
wordList = news.lower().split()

wordDict = {}
# for w in wordList:
#     wordDict[w] = wordDict.get(w, 0)+1
# for w in exclude:
#     del (wordDict[w])

wordSet = set(wordList) - exclude
for w in wordSet:
    wordDict[w] = wordList.count(w)

dictList = list(wordDict.items())
dictList.sort(key = lambda x:x[1], reverse=True)

for i in range(20):
    print(dictList[i])

# print(dictList)
# for w in wordDict:
#     print(w, wordDict[w])

# for w in wordList:
#     print(w)

# f = open(‘w.txt‘, ‘r‘)
# news = f.read()
# f.close()
# print(news)

　　

2.中文词频统计

下载一长篇中文文章。

从文件读取待分析文本。

news = open(‘gzccnews.txt‘,‘r‘,encoding = ‘utf-8‘)

安装与使用jieba进行中文分词。

pip install jieba

import jieba

list(jieba.lcut(news))

生成词频统计

排序

排除语法型词汇，代词、冠词、连词

输出词频最大TOP20（或把结果存放到文件里）

import jieba
 
f = open(‘gzccnews.txt‘,‘r‘, encoding=‘utf-8‘)
text = f.read()
f.close()
 
 
sep = ‘‘‘，。‘’“”：；（）！？、 ‘‘‘
a= {
     ‘\n‘, ‘\u3000‘,‘的‘,
     ‘他‘, ‘之‘, ‘不‘, ‘人‘,  ‘一‘,   ‘我‘, ‘下‘, ‘大‘,
     }
for i in sep:
    text = text.replace(i, ‘‘)
print(list(jieba.cut(text)))
t = list(jieba.lcut(text))
print(t)
count = {}
wl = list(set(t) - a)
print(wl)
 
for i in range(0, len(wl)):
    count[wl[i]] = text.count(str(wl[i]))
 
cl = list(count.items())
cl.sort(key=lambda x: x[1], reverse=True)
print(cl)
 
f = open(‘cpCount.txt‘, ‘a‘)
for i in range(20):
    f.write(cl[i][0] + ‘:‘ + str(cl[i][1]) + ‘\n‘)
f.close()