PReLU
A representative implementation of learnable activation functions that extends ReLU by treating the negative-slope coefficient as a learnable parameter, enabling the network to adaptively adjust the nonlinear response in the negative domain. The additional parameter overhead is minimal.
Timeline (last 90 days)
PReLU (Parametric ReLU) treats the slope of the negative half-axis as a learnable parameter, rather than fixing it at 0 or some constant.
Using a channel-wise PReLU for a convolutional layer with 512 channels adds only 512 scalar parameters, which is almost negligible compared to the millions of weight parameters in that layer.
A pragmatic approach is to first train a stable baseline using standard activation functions, then gradually replace them with lightweight alternatives such as PReLU or trainable Swish, and verify through A/B comparison whether WER improves.
All Facts (3)
PReLU (Parametric ReLU) treats the slope of the negative half-axis as a learnable parameter, rather than fixing it at 0 or some constant.
50%UnverifiedUsing a channel-wise PReLU for a convolutional layer with 512 channels adds only 512 scalar parameters, which is almost negligible compared to the millions of weight parameters in that layer.
50%UnverifiedA pragmatic approach is to first train a stable baseline using standard activation functions, then gradually replace them with lightweight alternatives such as PReLU or trainable Swish, and verify through A/B comparison whether WER improves.
50%