optiml.ml.neural_network.activations module

class optiml.ml.neural_network.activations.Activation

Bases: ABC

Base abstract class for all activation functions. Subclasses must implement function and its element-wise derivative jacobian.

function(x)

jacobian(x)

class optiml.ml.neural_network.activations.Linear

Bases: Activation

Identity (linear) activation function \(f(x) = x\).

function(x)

jacobian(x)

class optiml.ml.neural_network.activations.ReLU

Bases: Activation

Rectified linear unit activation function \(f(x) = \max(0, x)\).

function(x)

jacobian(x)

class optiml.ml.neural_network.activations.Tanh

Bases: Activation

Hyperbolic tangent activation function \(f(x) = \tanh(x)\).

function(x)

jacobian(x)

class optiml.ml.neural_network.activations.Sigmoid

Bases: Activation

Logistic sigmoid activation function \(f(x) = \frac{1}{1 + e^{-x}}\).

function(x)

jacobian(x)

class optiml.ml.neural_network.activations.SoftMax

Bases: Activation

Softmax activation function \(f(x)_i = \frac{e^{x_i}}{\sum_j e^{x_j}}\).

function(x, axis=-1)

jacobian(x)