中文词频统计

Posted 2020-10-28 BennyKuang

下载一长篇中文文章。

从文件读取待分析文本。

news = open(\'gzccnews.txt\',\'r\',encoding = \'utf-8\')

安装与使用jieba进行中文分词。

pip install jieba

import jieba

list(jieba.lcut(news))

生成词频统计

排序

排除语法型词汇，代词、冠词、连词

输出词频最大TOP20

 

import jieba

fo = open("xiyouji.txt", "r", encoding=\'utf-8\')
novel = fo.read()
fo.close()

delword = [\'我\', \'他\', \'你\', \'了\', \'那\', \'又\', \'-\', \'\\n\', \'，\', \'。\', \'？\', \'！\', \'“\', \'”\', \'：\', \'；\', \'、\', \'.\', \'‘\', \'’\']
wordDict = {}
wordList = list(jieba.cut(novel))
for i in wordList:
    wordDict[i] = wordList.count(i)

for i in delword:
    if i in wordDict:
        del wordDict[i]
sort = sorted(wordDict.items(), key=lambda item: item[1], reverse=True)
for i in range(20):
    print(sort[i])

运行结果：