optiml.ml.svm package

Submodules

Module contents

class optiml.ml.svm.SVM(loss=None, kernel=GaussianKernel(), C=1, rho=1, mu=1, fit_intercept=True, intercept_scaling=1, reg_intercept=False, dual=False, optimizer=<class 'optiml.opti.unconstrained.stochastic.gradient_descent.StochasticGradientDescent'>, master_solver='clarabel', learning_rate='auto', momentum_type='none', momentum=0.9, max_iter=1000, max_f_eval=15000, tol=0.0001, batch_size=None, shuffle=True, random_state=None, early_stopping=False, validation_split=0.0, patience=5, verbose=False, master_verbose=False)[source]

Bases: BaseEstimator, ABC

Base abstract class for all SVM-type estimator.

Parameters:

loss (SVMLoss instance, default=None) – Specifies the loss function.
kernel (Kernel instance like {linear, poly, gaussian, sigmoid}, default=gaussian) – Specifies the kernel type to be used in the algorithm.
C (float, default=1) – Regularization parameter. The strength of the regularization is inversely proportional to C. Must be strictly positive.
rho (int, default=1) – Rho parameter for the augmented term of the Lagrangian method. Must be strictly positive.
mu (float, default=1) – Mu parameter for the proximal bundle method. Only used when optimizer is ProximalBundle. Must be strictly positive.
fit_intercept (bool, default=True) – Whether to calculate the intercept for this model. If set to False, no intercept will be used in calculations (i.e., data is expected to be already centered).
intercept_scaling (float, default=1) – When fit_intercept is True, instance vector x becomes [x, intercept_scaling], i.e., a “synthetic” feature with constant value equals to intercept_scaling is appended to the instance vector. The intercept becomes intercept_scaling * synthetic feature weight Note: the synthetic feature weight is subject to L1/L2 regularization as all other features. To lessen the effect of regularization on synthetic feature weight (and therefore on the intercept) intercept_scaling has to be increased.
reg_intercept (bool, default=False) – Whether to include the intercept in the regularization term.
dual (bool, default=False) – Select the algorithm to either solve the dual or primal optimization problem. Prefer dual=False when n_samples > n_features and the instance vector is linearly separable in the given space or, if not, consider the possibly to apply a non-linear transformation of the instance vector using a low-rank kernel matrix approximation, i.e., Nystrom, before training. See more at: - https://scikit-learn.org/stable/modules/classes.html#module-sklearn.kernel_approximation - https://cdn.rawgit.com/mstrazar/mklaren/master/docs/build/html/projection.html
optimizer (LineSearchOptimizer or StochasticOptimizer subclass, default=StochasticGradientDescent) – The solver for optimization. It can be a subclass of the LineSearchOptimizer which can converge faster and perform better for small datasets, e.g., the BFGS quasi-Newton method or, alternatively, a subclass of the StochasticOptimizer e.g., the StochasticGradientDescent or Adam, which works well on relatively large datasets (with thousands of training samples or more) in terms of both training time and validation score.
master_solver (string, default='clarabel') – Master solver for the proximal bundle method for the CVXPY interface. Only used when optimizer is ProximalBundle.
learning_rate ('auto' or double, default='auto') – The initial learning rate used for weight update. It controls the step-size in updating the weights. Only used when optimizer is a subclass of StochasticOptimizer. If ‘auto’, 1/L is used where L is the Lipschitz constant.
momentum_type ({'none', 'polyak', 'nesterov'}, default='none') – Momentum type used for weight update. Only used when optimizer is a subclass of StochasticOptimizer.
momentum (float, default=0.9) – Momentum for weight update. Should be between 0 and 1. Only used when optimizer is a subclass of StochasticOptimizer.
max_iter (int, default=1000) – Maximum number of iterations. The solver iterates until convergence (determined by tol) or this number of iterations. If the optimizer is a subclass of StochasticOptimizer, this value determines the number of epochs (how many times each data point will be used), not the number of gradient steps.
max_f_eval (int, default=15000) – Only used when optimizer is a subclass of LineSearchOptimizer. Maximum number of loss function calls. The solver iterates until convergence (determined by tol), number of iterations reaches max_iter, or this number of loss function calls. Note that number of loss function calls will be greater than or equal to the number of iterations.
tol (float, default=1e-4) – Tolerance for stopping criterion.
batch_size (int, default=None) – Size of mini batches for stochastic optimizers. Only used when optimizer is a subclass of StochasticOptimizer.
shuffle (bool, default=True) – Whether to shuffle samples for batch sampling in each iteration. Only used when the optimizer is a subclass of StochasticOptimizer.
random_state (int, RandomState instance or None, default=None) – Controls the pseudo random number generation for train-test split if early_stopping is True and shuffling the data for batch sampling when an instance of StochasticOptimizer class is used as optimizer value. Pass an int for reproducible output across multiple function calls.
early_stopping (bool, default=False) – Whether to use early stopping to terminate training. If set to True and validation_split is greater than 0, it will automatically set aside validation_split``% of training data as validation and terminate training when validation score is not improving by at least ``tol for patience consecutive epochs, otherwise terminate training when train loss does not improve by more than tol for patience consecutive passes over the training set. Only used when optimizer is a subclass of StochasticOptimizer.
validation_split (float, default=0.) – The proportion of training data to set aside as validation set for early stopping. Must be between 0 and 1. Only used when optimizer is a subclass of StochasticOptimizer and early_stopping is True.
patience (int, default=5) – Maximum number of epochs to not meet tol improvement. Only used when optimizer is a subclass of StochasticOptimizer.
verbose (bool or int, default=False) – Controls the verbosity of progress messages to stdout. Use a boolean value to switch on/off or an int value to show progress each verbose time optimization steps.
master_verbose (bool or int, default=False) – Controls the verbosity of the CVXPY interface. Only used when optimizer is ProximalBundle.

coef_

Weights assigned to the features (coefficients in the primal problem). This is only available in the case of a linear kernel.

Type:: ndarray of shape (n_features,)

dual_coef_

Coefficients of the support vector in the decision function.

Type:: ndarray of shape (n_SV,)

intercept_

Constants in decision function.

Type:: float

support_

Indices of support vectors.

Type:: ndarray of shape (n_SV,)

support_vectors_

Support vectors.

Type:: ndarray of shape (n_SV, n_features)

fit(X, y)[source]

decision_function(X)[source]

get_metadata_routing()

Get metadata routing of this object.

Please check User Guide on how the routing mechanism works.

Returns:: routing – A MetadataRequest encapsulating routing information.
Return type:: MetadataRequest

get_params(deep=True)

Get parameters for this estimator.

Parameters:: deep (bool, default=True) – If True, will return the parameters for this estimator and contained subobjects that are estimators.
Returns:: params – Parameter names mapped to their values.
Return type:: dict

set_params(**params)

Set the parameters of this estimator.

The method works on simple estimators as well as on nested objects (such as Pipeline). The latter have parameters of the form <component>__<parameter> so that it’s possible to update each component of a nested object.

Parameters:: **params (dict) – Estimator parameters.
Returns:: self – Estimator instance.
Return type:: estimator instance

class optiml.ml.svm.SVC(loss=<class 'optiml.ml.svm.losses.SquaredHinge'>, kernel=GaussianKernel(), C=1, rho=1, mu=1, fit_intercept=True, intercept_scaling=1, reg_intercept=False, dual=False, optimizer=<class 'optiml.opti.unconstrained.stochastic.gradient_descent.StochasticGradientDescent'>, master_solver='clarabel', learning_rate='auto', momentum_type='none', momentum=0.9, max_iter=1000, max_f_eval=15000, tol=0.0001, batch_size=None, shuffle=True, random_state=None, early_stopping=False, validation_split=0.0, patience=5, verbose=False, master_verbose=False)[source]

Bases: ClassifierMixin, SVM

C-Support Vector Classification.

Parameters:: loss (SVMLoss instance like {hinge, squared_hinge}, default=’squared_hinge’) – Specifies the loss function. The hinge loss is the L1 loss, while the squared hinge loss is the L2 loss.

fit(X, y)[source]

predict(X)[source]

decision_function(X)

get_metadata_routing()

Get metadata routing of this object.

Please check User Guide on how the routing mechanism works.

Returns:: routing – A MetadataRequest encapsulating routing information.
Return type:: MetadataRequest

get_params(deep=True)

Get parameters for this estimator.

Parameters:: deep (bool, default=True) – If True, will return the parameters for this estimator and contained subobjects that are estimators.
Returns:: params – Parameter names mapped to their values.
Return type:: dict

score(X, y, sample_weight=None)

Return accuracy on provided data and labels.

In multi-label classification, this is the subset accuracy which is a harsh metric since you require for each sample that each label set be correctly predicted.

Parameters:

X (array-like of shape (n_samples, n_features)) – Test samples.
y (array-like of shape (n_samples,) or (n_samples, n_outputs)) – True labels for X.
sample_weight (array-like of shape (n_samples,), default=None) – Sample weights.

Returns:

score – Mean accuracy of self.predict(X) w.r.t. y.

Return type:

float

set_params(**params)

Set the parameters of this estimator.

The method works on simple estimators as well as on nested objects (such as Pipeline). The latter have parameters of the form <component>__<parameter> so that it’s possible to update each component of a nested object.

Parameters:: **params (dict) – Estimator parameters.
Returns:: self – Estimator instance.
Return type:: estimator instance

set_score_request(*, sample_weight: bool | None | str = '$UNCHANGED$') → SVC

Configure whether metadata should be requested to be passed to the score method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.
False: metadata is not requested and the meta-estimator will not pass it to score.
None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.
str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:: sample_weight (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for sample_weight parameter in score.
Returns:: self – The updated object.
Return type:: object

class optiml.ml.svm.SVR(loss=<class 'optiml.ml.svm.losses.SquaredEpsilonInsensitive'>, epsilon=0.1, kernel=GaussianKernel(), C=1, rho=1, mu=1, fit_intercept=True, intercept_scaling=1, reg_intercept=False, dual=False, optimizer=<class 'optiml.opti.unconstrained.stochastic.gradient_descent.StochasticGradientDescent'>, master_solver='clarabel', learning_rate='auto', momentum_type='none', momentum=0.9, max_iter=1000, max_f_eval=15000, tol=0.0001, batch_size=None, shuffle=True, random_state=None, early_stopping=False, validation_split=0.0, patience=5, verbose=False, master_verbose=False)[source]

Bases: RegressorMixin, SVM

Epsilon-Support Vector Regression.

Parameters:

loss (SVMLoss instance like {epsilon_insensitive, squared_epsilon_insensitive}, default=’squared_epsilon_insensitive’) – Specifies the loss function. The epsilon-insensitive loss is the L1 loss, while the squared epsilon-insensitive loss is the L2 loss.
epsilon (float, default=0.1) – Epsilon parameter in the (squared) epsilon-insensitive loss function. It specifies the epsilon-tube within which no penalty is associated in the training loss function with points predicted within a distance epsilon from the actual value.

fit(X, y)[source]

predict(X)[source]

decision_function(X)

get_metadata_routing()

Get metadata routing of this object.

Please check User Guide on how the routing mechanism works.

Returns:: routing – A MetadataRequest encapsulating routing information.
Return type:: MetadataRequest

get_params(deep=True)

Get parameters for this estimator.

Parameters:: deep (bool, default=True) – If True, will return the parameters for this estimator and contained subobjects that are estimators.
Returns:: params – Parameter names mapped to their values.
Return type:: dict

score(X, y, sample_weight=None)

Return coefficient of determination on test data.

The coefficient of determination, $R^2$, is defined as $(1 - \frac{u}{v})$, where $u$ is the residual sum of squares ((y_true - y_pred)** 2).sum() and $v$ is the total sum of squares ((y_true - y_true.mean()) ** 2).sum(). The best possible score is 1.0 and it can be negative (because the model can be arbitrarily worse). A constant model that always predicts the expected value of y, disregarding the input features, would get a $R^2$ score of 0.0.

Parameters:

X (array-like of shape (n_samples, n_features)) – Test samples. For some estimators this may be a precomputed kernel matrix or a list of generic objects instead with shape (n_samples, n_samples_fitted), where n_samples_fitted is the number of samples used in the fitting for the estimator.
y (array-like of shape (n_samples,) or (n_samples, n_outputs)) – True values for X.
sample_weight (array-like of shape (n_samples,), default=None) – Sample weights.

Returns:

score – $R^2$ of self.predict(X) w.r.t. y.

Return type:

float

Notes

The $R^2$ score used when calling score on a regressor uses multioutput='uniform_average' from version 0.23 to keep consistent with default value of r2_score(). This influences the score method of all the multioutput regressors (except for MultiOutputRegressor).

set_params(**params)

Set the parameters of this estimator.

The method works on simple estimators as well as on nested objects (such as Pipeline). The latter have parameters of the form <component>__<parameter> so that it’s possible to update each component of a nested object.

Parameters:: **params (dict) – Estimator parameters.
Returns:: self – Estimator instance.
Return type:: estimator instance

set_score_request(*, sample_weight: bool | None | str = '$UNCHANGED$') → SVR

Configure whether metadata should be requested to be passed to the score method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.
False: metadata is not requested and the meta-estimator will not pass it to score.
None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.
str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:: sample_weight (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for sample_weight parameter in score.
Returns:: self – The updated object.
Return type:: object