embedding based logistic regression-神经网络逻辑回归tensorflow
Posted LarryGates
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了embedding based logistic regression-神经网络逻辑回归tensorflow相关的知识,希望对你有一定的参考价值。
--- 灵感 --- 因为最近一直在做rnn based NLP,其中无论是什么cell,lstm, GRU或者cnn都是基于单词的embedding表示;单词的embdding就是把每个单词表示成一个向量, 然后通过bp训练这些向量的值,这种想法很奇妙,于是我尝试性的把这种思想用在logistic regression上面;
--- 问题 --- 对于logistic regression的话,很多向量都是categorial,如果碰到有1000个category怎么做?转换成1000*1的one-hot向量吗? 方法:用embedding, 每个category给一个10维的向量,然后再用传统的回归或者神经网络的方法;
--- 实验 --- 1: 数据一览; 数据来自kaggle, 是redhat那个项目,感兴趣的自己去看看; 2:方法; 标题是逻辑回归,但是本质上还是神经网络做分类;但是这个问题传统上都是用逻辑回归解决的,因为包含了很多categorial的数据,然后label是0和1,要求做分类; 运行一个logistic regression是很简单的;但是这里的问题是数据里面有个group变量和一个people向量,group大概有3k+种类,people大概有180K+种类,显然转换成dummy变量再做逻辑回归的话不合适;这里我主要是参考word embedding的思想,在tensorflow里面建立两个个词典,一个people词典一个group词典,然后训练的时候分别去查这个词典返回两个10维的实数向量,这两个实数向量就分别是people和group的特征;之后再随便弄了一点full connected的层和一些激活函数,效果不错,很快收敛到90%以上了; 3:效果; 这个数据的话,我刚开始只是想用来实验在tf.Session()的情况下怎么样batch读取tfrecords数据的,因为tfrecords数据读取的话不需要把整个数据load进去内存;之前一直用estimator的方法读tfrecords,但是用session之后似乎没有很好的解决方法; 效果还不错,主要是感觉对于多种类的问题都可以用embedding的方法来做了以后;
#encoding=utf-8 import numpy as np import tensorflow as tf import pickle import random model_dir = ‘/home/yanjianfeng/kaggle/data/model_dir/‘ people_dic, group_dic, dic = pickle.load(open(‘/home/yanjianfeng/kaggle/data/data.dump‘, ‘r‘)) def create_train_op(loss): train_op = tf.contrib.layers.optimize_loss(loss = loss, global_step = tf.contrib.framework.get_global_step(), learning_rate = 0.1, clip_gradients = 10.0, optimizer = "Adam") return train_op def create_input(): random_id = random.randint(0, len(dic[‘outcome‘])-2049) keys = dic.keys() data = {} for k in keys: data[k] = dic[k][random_id: random_id+2048] return data # 主体部分还是最好不要放在函数里面,不太容易提取出某个特定的值 # 或者直接把主体部分放在tf.Session里面比较容, 大概就是这么一个模式; global_step = tf.Variable(0, name = ‘global_step‘, trainable=False) people_id = tf.placeholder("int64", [None]) group = tf.placeholder(‘int64‘, [None]) time = tf.placeholder(‘int64‘, [None]) peofea = tf.placeholder(‘int64‘, [None, 262]) rowfea = tf.placeholder(‘int64‘, [None, 174]) outcome = tf.placeholder("int64", [None]) name_embed = tf.get_variable(‘names‘, shape = [189120, 10]) group_embed = tf.get_variable(‘groups‘, shape = [35000, 10]) name_ = tf.nn.embedding_lookup(name_embed, people_id) group_ = tf.nn.embedding_lookup(group_embed, group) name_w = tf.get_variable(‘name_w‘, shape = [10, 2]) group_w = tf.get_variable(‘group_w‘, shape = [10, 5]) name_outcome = tf.matmul(name_, name_w) group_outcome = tf.matmul(group_, group_w) w_1 = tf.get_variable(‘w_1‘, shape = [262, 10]) w_2 = tf.get_variable(‘w_2‘, shape = [174, 10]) w_3 = tf.get_variable(‘w_3‘, shape = [1]) peofea_outcome = tf.matmul(tf.to_float(peofea), w_1) rowfea_outcome = tf.matmul(tf.to_float(rowfea), w_2) time_outcome = tf.mul(tf.to_float(time), w_3) time_outcome = tf.expand_dims(time_outcome, -1) name_outcome = tf.sigmoid(name_outcome) group_outcome = tf.sigmoid(group_outcome) peofea_outcome = tf.sigmoid(peofea_outcome) rowfea_outcome = tf.sigmoid(rowfea_outcome) time_outcome = tf.sigmoid(time_outcome) x = tf.concat(1, [name_outcome, group_outcome, peofea_outcome, rowfea_outcome, time_outcome]) w_f = tf.get_variable(‘w_f‘, shape = [28, 28]) b = tf.get_variable(‘b‘, shape = [1]) w_f_2 = tf.get_variable(‘w_f_2‘, shape = [28, 1]) pred = tf.sigmoid(tf.matmul(x, w_f)) + b pred = tf.matmul(pred, w_f_2) y = tf.expand_dims(tf.to_float(outcome), -1) prob = tf.sigmoid(pred) prob = tf.to_float(tf.greater(prob, 0.5)) c = tf.reduce_mean(tf.to_float(tf.equal(prob, y))) loss = tf.nn.sigmoid_cross_entropy_with_logits(pred, y) loss = tf.reduce_mean(loss) train_op = create_train_op(loss) # 这里的顺序很重要,要是在最前面用saver,则会save到最开始的情况? saver = tf.train.Saver() with tf.Session() as sess: sess.run(tf.initialize_all_variables()) ckpt = tf.train.get_checkpoint_state(model_dir) if ckpt and ckpt.model_checkpoint_path: print ‘the model being restored is ‘ print ckpt.model_checkpoint_path saver.restore(sess, ckpt.model_checkpoint_path) print ‘sucesssfully restored the session‘ count = global_step.eval() for i in range(0, 10000): data = create_input() l, _ , c_ = sess.run([loss, train_op, c], feed_dict = {people_id: data[‘people_id‘], group: data[‘group‘], time: data[‘time‘], peofea: data[‘people_features‘], rowfea: data[‘row_features‘], outcome: data[‘outcome‘]}) print ‘the loss\t‘ + str(l) + ‘\t\tthe count\t‘ + str(c_) global_step.assign(count).eval() saver.save(sess, model_dir + ‘model.ckpt‘, global_step = global_step) count += 1
以上是关于embedding based logistic regression-神经网络逻辑回归tensorflow的主要内容,如果未能解决你的问题,请参考以下文章
Embedding-based Retrieval in Facebook Search
Dependency-Based Word Embeddings(基于依存的词向量)
论文阅读|深度Role-based network embedding via structural features reconstruction with degree-regularized
文献总结RDAA:Role Discovery-Guided Network Embedding Based on Autoencoder and Attention Mechanism
文献总结RDAA:Role Discovery-Guided Network Embedding Based on Autoencoder and Attention Mechanism
论文阅读|深读RDAA:Role Discovery-Guided Network Embedding Based on Autoencoder and Attention Mechanism