Tanh
A neural network activation function that maps inputs to the interval (-1, 1). Its output is zero-centered and converges more smoothly than Sigmoid, but it likewise suffers from the vanishing gradient problem.
Core Facts
Timeline (last 90 days)
The Tanh function has an output range of -1 to 1 and likewise suffers from the vanishing gradient problem.
Xavier initialization adjusts the variance of weights according to the number of input and output neurons, and is suitable for Sigmoid/Tanh-type activation functions.
Compared to Sigmoid or Tanh activation functions, ReLU has a constant gradient of 1 on the positive half-axis, effectively alleviating the vanishing gradient problem in deep network training, while also having extremely low computational cost.
All Facts (4)
The mathematical definition of the ReLU activation function is f(x) = max(0, x), which can effectively mitigate the vanishing gradient problem compared to Sigmoid and Tanh.
70%UnverifiedThe Tanh function has an output range of -1 to 1 and likewise suffers from the vanishing gradient problem.
50%UnverifiedXavier initialization adjusts the variance of weights according to the number of input and output neurons, and is suitable for Sigmoid/Tanh-type activation functions.
50%UnverifiedCompared to Sigmoid or Tanh activation functions, ReLU has a constant gradient of 1 on the positive half-axis, effectively alleviating the vanishing gradient problem in deep network training, while also having extremely low computational cost.
50%