deep learning-intuition

Posted 2020-09-07

线性可分视角：神经网络的学习就是学习如何利用矩阵的线性变换加激活函数的非线性变换，将原始输入空间投向线性可分/稀疏的空间去分类/回归。

增加节点数：增加维度，即增加线性转换能力。
增加层数：增加激活函数的次数，即增加非线性转换次数。

物质组成视角：神经网络的学习过程就是学习物质组成方式的过程。

增加节点数：增加同一层物质的种类，比如118个元素的原子层就有118个节点。
增加层数：增加更多层级，比如分子层，原子层，器官层，并通过判断更抽象的概念来识别物体。

Each layer stretches and squishes space, but it never cuts, breaks, or folds it. Intuitively, we can see that it preserves topological properties.For example, a set will be connected afterwards if it was before (and vice versa).

 

tanh (and sigmoid and softplus but not ReLU) are continuous functions with continuous inverses. They are bijections if we are careful about the domain and range we consider. Applying them pointwise is a homemorphism.

two hidden units 从拓扑学来讨论2-layer nn 若一层的W可逆，则前后同胚无法分开。若W不可逆，则会坍缩，降维，导致一些标签不同的点混在一起，分不开。

If we add a third hidden unit, the problem becomes trivial.

 

The more I think about standard neural network layers – that is, with an affine transformation followed by a point-wise activation function – the more disenchanted I feel. It’s hard to imagine that these are really very good for manipulating manifolds.

 

Topological properties of data, such as links, may make it impossible to linearly separate classes using low-dimensional networks, regardless of depth. Even in cases where it is technically possible, such as spirals, it can be very challenging to do so.

 

To accurately classify data with neural networks, wide layers are sometimes necessary. Further, traditional neural network layers do not seem to be very good at representing important manipulations of manifolds; even if we were to cleverly set weights by hand, it would be challenging to compactly represent the transformations we want. New layers, specifically motivated by the manifold perspective of machine learning, may be useful supplements.

 

 https://zhuanlan.zhihu.com/p/22888385

reference：http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/