Neural Network Isn't Learning For A First Few Epochs On Keras

I'm testing simple networks on Keras with TensorFlow backend and I ran into an issue with using sigmoid activation function The network isn't learning for first 5-10 epochs, and th

Solution 1:

Activation function suggestion：

In practice, the sigmoid non-linearity has recently fallen out of favor and it is rarely ever used. ReLU is the most common choice, if there are a large fraction of “dead” units in network, try Leaky ReLU and tanh. Never use sigmoid.

Reasons for not using the sigmoid：

A very undesirable property of the sigmoid neuron is that when the neuron’s activation saturates at either tail of 0 or 1, the gradient at these regions is almost zero. In addition, Sigmoid outputs are not zero-centered.