From this video tutorial Vanishing Gradient Tutorial, the sigmoid function and the hyperbolic tangent can produce the vanishing gradient problem.
What other activation functions can lead to the vanishing gradient problem?
Asked
Active
Viewed 488 times
1
-
Possibly, the [saturating activation functions](http://proceedings.mlr.press/v48/gulcehre16.pdf) (i.e. activation functions that squash their input to a finite interval). Both sigmoid and hyperbolic tangents are saturating activation functions. – nbro Mar 16 '20 at 17:56