optiml.ml.neural_network.losses module

class optiml.ml.neural_network.losses.NeuralNetworkLoss(neural_net, X, y)

Bases: OptimizationFunction, ABC

Base abstract class for all neural network loss functions. It defines the objective minimized during training, i.e., the data loss averaged over the samples plus the layers regularization terms, together with its jacobian computed via back-propagation.

Subclasses must implement loss and, optionally, override delta.

Parameters:

• neural_net (NeuralNetwork instance) – The neural network estimator this loss is attached to. It provides the layers and the forward/backward passes used by the objective.

• X (ndarray of shape (n_samples, n_features)) – Training data over which the loss is evaluated.

• y (ndarray of shape (n_samples, n_outputs)) – Target values associated with X.

args()

loss(y_pred, y_true)

delta(y_pred, y_true)

function(packed_coef_inter, X_batch=None, y_batch=None)

jacobian(packed_coef_inter, X_batch=None, y_batch=None)

The Jacobian (i.e., the gradient) of the function.

Parameters:

x – 1D array of points at which the Jacobian is to be computed.

Returns:

the Jacobian of the function at x.

f_star()

function_jacobian(*args, **kwargs)

hessian(x)

The Hessian matrix of the function.

Parameters:

x – 1D array of points at which the Hessian is to be computed.

Returns:

the Hessian matrix of the function at x.

x_star()

class optiml.ml.neural_network.losses.MeanSquaredError(neural_net, X, y)

Bases: NeuralNetworkLoss

Compute the mean squared error loss for regression as:

\[L(y_{pred}, y_{true}) = \sum (y_{pred} - y_{true})^2\]

Parameters:

• neural_net (NeuralNetwork instance) – The neural network estimator this loss is attached to. It provides the layers and the forward/backward passes used by the objective.

• X (ndarray of shape (n_samples, n_features)) – Training data over which the loss is evaluated.

• y (ndarray of shape (n_samples, n_outputs)) – Target values associated with X.

x_star()

f_star()

loss(y_pred, y_true)

args()

delta(y_pred, y_true)

function(packed_coef_inter, X_batch=None, y_batch=None)

function_jacobian(*args, **kwargs)

hessian(x)

The Hessian matrix of the function.

Parameters:

x – 1D array of points at which the Hessian is to be computed.

Returns:

the Hessian matrix of the function at x.

jacobian(packed_coef_inter, X_batch=None, y_batch=None)

The Jacobian (i.e., the gradient) of the function.

Parameters:

x – 1D array of points at which the Jacobian is to be computed.

Returns:

the Jacobian of the function at x.

class optiml.ml.neural_network.losses.MeanAbsoluteError(neural_net, X, y)

Bases: NeuralNetworkLoss

Compute the mean absolute error loss for regression as:

\[L(y_{pred}, y_{true}) = \sum \lvert y_{pred} - y_{true} \rvert\]

Parameters:

• neural_net (NeuralNetwork instance) – The neural network estimator this loss is attached to. It provides the layers and the forward/backward passes used by the objective.

• X (ndarray of shape (n_samples, n_features)) – Training data over which the loss is evaluated.

• y (ndarray of shape (n_samples, n_outputs)) – Target values associated with X.

loss(y_pred, y_true)

delta(y_pred, y_true)

args()

f_star()

function(packed_coef_inter, X_batch=None, y_batch=None)

function_jacobian(*args, **kwargs)

hessian(x)

The Hessian matrix of the function.

Parameters:

x – 1D array of points at which the Hessian is to be computed.

Returns:

the Hessian matrix of the function at x.

jacobian(packed_coef_inter, X_batch=None, y_batch=None)

The Jacobian (i.e., the gradient) of the function.

Parameters:

x – 1D array of points at which the Jacobian is to be computed.

Returns:

the Jacobian of the function at x.

x_star()

class optiml.ml.neural_network.losses.BinaryCrossEntropy(neural_net, X, y)

Bases: NeuralNetworkLoss

Binary Cross-Entropy aka Sigmoid Cross-Entropy loss function for binary and multi-label classification or regression between 0 and 1 with sigmoid output layer:

\[L(y_{pred}, y_{true}) = -\sum \left[ y_{true} \log(y_{pred}) + (1 - y_{true}) \log(1 - y_{pred}) \right]\]

Parameters:

• neural_net (NeuralNetwork instance) – The neural network estimator this loss is attached to. It provides the layers and the forward/backward passes used by the objective.

• X (ndarray of shape (n_samples, n_features)) – Training data over which the loss is evaluated.

• y (ndarray of shape (n_samples, n_outputs)) – Target values associated with X.

loss(y_pred, y_true)

args()

delta(y_pred, y_true)

f_star()

function(packed_coef_inter, X_batch=None, y_batch=None)

function_jacobian(*args, **kwargs)

hessian(x)

The Hessian matrix of the function.

Parameters:

x – 1D array of points at which the Hessian is to be computed.

Returns:

the Hessian matrix of the function at x.

jacobian(packed_coef_inter, X_batch=None, y_batch=None)

The Jacobian (i.e., the gradient) of the function.

Parameters:

x – 1D array of points at which the Jacobian is to be computed.

Returns:

the Jacobian of the function at x.

x_star()

class optiml.ml.neural_network.losses.CategoricalCrossEntropy(neural_net, X, y)

Bases: NeuralNetworkLoss

Categorical Cross-Entropy loss function for multi-class (single-label) classification with softmax output layer and one-hot encoded target data:

\[L(y_{pred}, y_{true}) = -\sum y_{true} \log(y_{pred})\]

Parameters:

• neural_net (NeuralNetwork instance) – The neural network estimator this loss is attached to. It provides the layers and the forward/backward passes used by the objective.

• X (ndarray of shape (n_samples, n_features)) – Training data over which the loss is evaluated.

• y (ndarray of shape (n_samples, n_outputs)) – Target values associated with X.

loss(y_pred, y_true)

delta(y_pred, y_true)

args()

f_star()

function(packed_coef_inter, X_batch=None, y_batch=None)

function_jacobian(*args, **kwargs)

hessian(x)

The Hessian matrix of the function.

Parameters:

x – 1D array of points at which the Hessian is to be computed.

Returns:

the Hessian matrix of the function at x.

jacobian(packed_coef_inter, X_batch=None, y_batch=None)

The Jacobian (i.e., the gradient) of the function.

Parameters:

x – 1D array of points at which the Jacobian is to be computed.

Returns:

the Jacobian of the function at x.

x_star()

class optiml.ml.neural_network.losses.SparseCategoricalCrossEntropy(neural_net, X, y)

Bases: NeuralNetworkLoss

Sparse Categorical Cross-Entropy loss function for multi-class (single-label) classification with softmax output layer

Parameters:

• neural_net (NeuralNetwork instance) – The neural network estimator this loss is attached to. It provides the layers and the forward/backward passes used by the objective.

• X (ndarray of shape (n_samples, n_features)) – Training data over which the loss is evaluated.

• y (ndarray of shape (n_samples, n_outputs)) – Target values associated with X.

loss(y_pred, y_true)

delta(y_pred, y_true)

args()

f_star()

function(packed_coef_inter, X_batch=None, y_batch=None)

function_jacobian(*args, **kwargs)

hessian(x)

The Hessian matrix of the function.

Parameters:

x – 1D array of points at which the Hessian is to be computed.

Returns:

the Hessian matrix of the function at x.

jacobian(packed_coef_inter, X_batch=None, y_batch=None)

The Jacobian (i.e., the gradient) of the function.

Parameters:

x – 1D array of points at which the Jacobian is to be computed.

Returns:

the Jacobian of the function at x.

x_star()

optiml.ml.neural_network.losses.mean_squared_error

alias of MeanSquaredError

optiml.ml.neural_network.losses.mean_absolute_error

alias of MeanAbsoluteError

optiml.ml.neural_network.losses.binary_cross_entropy

alias of BinaryCrossEntropy

optiml.ml.neural_network.losses.categorical_cross_entropy

alias of CategoricalCrossEntropy

optiml.ml.neural_network.losses.sparse_categorical_cross_entropy

alias of SparseCategoricalCrossEntropy