deep learning

Neural Tangent Kernel(NTK) “In short, NTK represent the changes of the weights before and after the gradient descent update” Let’s start the journey of revealing the black-box neural networks. Setup a Neural Network First of all, we need to define a simple neural network with 2 hidden layers $$y(x,w)$$ where $y$ is the neural network with weights $w\in {\mathbb{R}}^m$ and, ${x,\overline{y}}_N$ is the dataset which is a set of the input data and the output data with $N$ data points....

Neural Network Gaussian Process(NNGP) Model the neural network as GP, aka neural network Gaussian Process(NNGP). Intuitively, the kernel of NNGP compute the distance between the output vectors of 2 input data points. We define the following functions as neural networks with fully-conntected layers: $$z_i^1(x)=b_i^1+\sum _{j=1}^{N_1}{W}_{ij}^1{x}_j^1(x),x_j^1(x)=\varphi (b_i^0+\sum _{k=1}^{d_{in}}{W}_{ik}^0{x}_k(x))$$ where $b_i^1$ is the $i$th-bias of the second layer(the same as first hidden layer), $W_{ij}^1$ is the $i$th-weights of the first layer(the same as input layer) , function $\varphi$ is the activation function, and $x$ is the input data of the neural network....