利用搜狐新闻语料库训练100维的word2vec——使用python中的gensim模块

Posted 2021-02-09 米仓山下

　　关于word2vec的原理知识参考文章https://www.cnblogs.com/Micang/p/10235783.html

　　语料数据来自搜狐新闻2012年6月—7月期间国内，国际，体育，社会，娱乐等18个频道的新闻数据
　　数据处理参考这篇文章

　　模型训练：

# -*- coding: utf-8-*-
from gensim.models.word2vec import Word2Vec 
sentences = [[\'A1\',\'A2\'],[\'A1\',\'A3\',\'A2\']] 

num=0
with open(\'sohu_train.txt\') as trainText:  #, encoding=\'utf-8\'
    for line in trainText:
        id,catgre,body= line.split(\'^_^\')
        words=body.replace(\'\\n\',\'\').split(\'    \')
        sentences.append(words)
        # if num>1000:break
        num+=1
        # print(sentences)

model= Word2Vec(min_count=1)
print("start train ...")
model.build_vocab(sentences)
model.train(sentences,total_examples = model.corpus_count,epochs = model.iter)
print("train finished!",num)

model.save(\'./sohu_model/Model\')
#model.save_word2vec_format(\'/tmp/mymodel.txt\',binary = False)
#model.save_word2vec_format(\'/tmp/mymodel.bin.gz\',binary = True)
#前一组方法保存的文件不能利用文本编辑器查看但是保存了训练的全部信息，可以在读取后追加训练
#后一组方法保存为word2vec文本格式但是保存时丢失了词汇树等部分信息，不能追加训练
print("save finished!")

　　模型使用：

# #模型使用
model = Word2Vec.load(\'./sohu_model/Model\')
print("load model sesuess!")
# model.most_similar([\'北京\'])

print u\'most similar with 北京:\'
for i in model.most_similar("北京"): #计算余弦距离最接近“北京”的10个词
    print i[0].decode(\'utf-8\'),i[1]

print u\'皇帝+女性-男性：\'
for i in model.most_similar(positive = [\'皇帝\',\'女性\'],negative = [\'男性\'],topn = 3):print i[0].decode(\'utf-8\'),i[1]

print u\'手机+移动-智能：\'
for i in model.most_similar(positive = [\'手机\',\'移动\'],negative = [\'智能\'],topn = 3):print i[0].decode(\'utf-8\'),i[1]

print u\'电影+科幻-剧情：\'
for i in model.most_similar(positive = [\'电影\',\'科幻\'],negative = [\'剧情\'],topn = 3):print i[0].decode(\'utf-8\'),i[1]

print u\'北京 vector:\'
print model[\'北京\']

　　输出：

C:\\Python27\\lib\\site-packages\\gensim\\utils.py:1212: UserWarning: detected Windows; aliasing chunkize to chunkize_serial
  warnings.warn("detected Windows; aliasing chunkize to chunkize_serial")
load model sesuess!
most similar with 北京:
C:\\Python27\\lib\\site-packages\\gensim\\matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int32 == np.dtype(int).type`.
  if np.issubdtype(vec.dtype, np.int):
南京 0.670382142067
上海 0.661236405373
成都 0.639219224453
杭州 0.63784122467
广州 0.631313323975
深圳 0.624626278877
武汉 0.624594151974
昆明 0.620243370533
长春 0.61394149065
长沙 0.60389906168
皇帝+女性-男性：
哥 0.60431176424
魔术师 0.586149096489
魔女 0.581812143326
手机+移动-智能：
智能手机 [phone]
互联网 0.54615008831
苹果 0.539426982403
电影+科幻-剧情：
纪录片 0.648482918739
动画 0.639703273773
迪斯尼 0.61851131916
北京 vector:
[-0.08981118  0.18538047 -4.7453156  -1.7730242   2.0390635   2.6085184
  5.088326    2.8057106   2.6798103  -1.4660915   2.778077    2.4279277
  0.69682086 -3.0003173   2.1341784   0.32419717 -5.2817945   0.18809023
 -1.3016417   3.8344557  -0.87402123 -0.26100433  2.8857462  -2.725345
 -2.5024219  -0.70686543 -0.4838663  -2.2535524   0.23617841  3.329134
  3.9053504  -1.9609474  -3.4581995   1.2530506  -2.079397    1.6266809
  0.23296945  1.4600109  -1.9104419   0.80835503 -0.13650164  3.355157
  2.4561696   0.6016032  -1.0312346   1.6474588   1.320931    1.4579619
  1.8017172  -3.5526018   1.2293625   4.798621   -3.5554793   0.5800354
  3.7429204  -0.4906999  -1.3069346  -1.0603447  -0.95469594 -0.35445935
 -1.7658769  -3.2370284  -2.2224278  -0.56134427 -0.46095294  2.8492029
  2.7202766  -3.3692176   1.1739812  -1.9770668   0.37050596  1.1764477
 -0.27834406  5.033905    0.09570877 -0.5670941  -2.1803875  -0.9094422
  1.0485793   0.03497482 -2.07145    -0.8045679  -1.8192968   2.6160874
  0.5630188  -0.45463613 -0.22750562  2.2233796   3.4276621  -0.8689221
  1.5558586  -0.39026013 -1.1843458  -3.378433   -4.2200727   1.6359595
  2.27458    -1.6011585  -0.89109504  2.3993087 ]