GCL.eval¶

Available Evaluation Models¶

`BaseTrainableEvaluator`	Base class for trainable (e.g., logistic regression) evaluation.
`BaseSKLearnEvaluator`	Base class for sklearn-based evaluation.
`LRTrainableEvaluator`	Evaluate using a trainable logistic regression model.
`LRSklearnEvaluator`	Evaluate using the sklearn logistic regression classifier.
`SVMEvaluator`	Evaluate using the sklearn SVM classifier.
`RFEvaluator`	Evaluate using the sklearn random forest classifier.
`random_split`	Generate split indices for training, test, and validation sets.
`from_PyG_split`	Convert from PyG split indices of training, test, and validation sets.

Evaluation Models¶

class BaseSKLearnEvaluator(evaluator, metrics, split, params=None, param_grid=None, grid_search_scoring=None, grid_search_params=None)¶

Base class for sklearn-based evaluation.

Parameters

evaluator (BaseEstimator) – The sklearn evaluator.
metrics (Dict[str, Callable]) – The metrics to evaluate in a dictionary with metric names as keys and callables as values.
split (Union[List[Dict], BaseCrossValidator]) – A list of split indices (for multiple folds), or a sklearn cross-validator. If a list of indices is given, the validation set will be ignored.
params (Dict, optional) – Other parameters for the evaluator. (default: None)
param_grid (List[Dict], optional) – The parameter grid for the grid search. (default: None)
grid_search_scoring (Dict[str, Callable], optional) – If param_grid is given, provide metrics in grid search. If multiple metrics are given, the first one will be used to retrain the best model. (default: None)
grid_search_params (Dict, optional) – If param_grid is given, further pass the parameters for the sklearn grid search cross-validator. See sklearn GridSearchCV for details. (default: None)

evaluate(x, y)¶

Evaluate the model on the given data using sklearn evaluator.

Parameters

x (np.ndarray) – The data.
y (np.ndarray) – The targets (labels).

Returns

Evaluation results with metrics as keys and mean and standard deviation as values.

Return type

Dict[str, Dict]

class BaseTrainableEvaluator(model, optimizer, optimizer_params, objective, split, metrics, device='cpu', num_epochs=1000, test_interval=20, test_metric=None)¶

Base class for trainable (e.g., logistic regression) evaluation.

Parameters

model (torch.nn.Module) – The evaluation model to train. It should implement a predict function to convert logits to predictions.
optimizer (Optimizer) – The optimizer class for training.
optimizer_params (Dict) – The parameters for the optimizer.
objective (Callable) – The objective function to use for training. Its signature should be like f(logits, y).
split (Union[List[Dict], BaseCrossValidator]) – A list of split indices (for multiple folds), or a sklearn cross-validator.
metrics (Dict[str, Callable]) – The metrics to evaluate in a dictionary with metric names as keys and callables as values.
device (Union[str, torch.device]) – The device to use for training. (default: 'cpu')
num_epochs (int) – The number of epochs to train the model. (default: 1000)
test_interval (int) – The number of epochs between each test. (default: 20)
test_metric (Union[Callable, str], optional) – The metric to test on the validation set during training. It could be a callable function, or a string specifying the key in metrics. If set to None, the test metric will be the first in metrics. If split is a sklearn cross-validator, this parameter is ignored as no validation set is used. (default: None)

evaluate(x, y)¶

Evaluate the model on the given data by training another evaluator model.

Parameters

x (torch.Tensor) – The data.
y (torch.Tensor) – The targets (labels).

Returns

Evaluation results with metrics as keys and mean and standard deviation as values.

Return type

Dict[str, Dict]

class LRSklearnEvaluator(metrics, split, params=None, param_grid=None, grid_search_scoring=None, grid_search_params=None)¶

Evaluate using the sklearn logistic regression classifier.

Parameters

metrics (Dict[str, Callable]) – The metrics to evaluate in a dictionary with metric names as keys and callables as values.
split (BaseCrossValidator) – The sklearn cross-validator to split the data.
params (Dict, optional) – Other parameters for the logistic regression model. See sklearn LogisticRegression for details. (default: None)
param_grid (List[Dict], optional) – The parameter grid for the grid search. (default: None)
grid_search_scoring (Dict[str, Callable], optional) – If param_grid is given, provide metrics in grid search. If multiple metrics are given, the first one will be used to retrain the best model. (default: None)
grid_search_params (Dict, optional) –
If param_grid is given, further pass the parameters for the sklearn grid search cross-validator. See sklearn GridSearchCV for details. (default: None)

class LRTrainableEvaluator(input_dim, num_classes, split, metrics, device='cpu', num_epochs=5000, learning_rate=0.01, weight_decay=0.0, test_interval=20, test_metric=None)¶

Evaluate using a trainable logistic regression model.

Parameters

input_dim (int) – The dimension of the input data.
num_classes (int) – The number of classes.
split (Union[List[Dict], BaseCrossValidator]) – A list of split indices (for multiple folds), or a sklearn cross-validator.
metrics (Dict[str, Callable]) – The metrics to evaluate in a dictionary with metric names as keys and callables as values.
device (Union[str, torch.device]) – The device to use for training. (default: 'cpu')
num_epochs (int) – The number of epochs to train. (default: 5000)
learning_rate (float) – The learning rate for the optimizer. (default: 0.01)
weight_decay (float) – The weight decay for the optimizer. (default: 0.0)
test_interval (int) – The number of epochs between each test. (default: 20)
test_metric (Union[Callable, str], optional) – The metric to test on the validation set during training. It could be a callable function, or a string specifying the key in metrics. If set to None, the test metric will be the first in metrics. If split is a sklearn cross-validator, this parameter is ignored as no validation set is used. (default: None)

class RFEvaluator(metrics, split, params=None, param_grid=None, grid_search_scoring=None, grid_search_params=None)¶

Evaluate using the sklearn random forest classifier.

Parameters

metrics (Dict[str, Callable]) – The metrics to evaluate in a dictionary with metric names as keys and callables as values.
split (BaseCrossValidator) – The sklearn cross-validator to split the data.
params (Dict, optional) – Other parameters for the random forest classifier. See sklearn RandomForestClassifier for details. (default: None)
param_grid (List[Dict], optional) – The parameter grid for the grid search. (default: None)
grid_search_scoring (Dict[str, Callable], optional) – If param_grid is given, provide metrics in grid search. If multiple metrics are given, the first one will be used to retrain the best model. (default: None)
grid_search_params (Dict, optional) –
If param_grid is given, further pass the parameters for the sklearn grid search cross-validator. See sklearn GridSearchCV for details. (default: None)

class SVMEvaluator(metrics, split, linear=True, params=None, param_grid=None, grid_search_scoring=None, grid_search_params=None)¶

Evaluate using the sklearn SVM classifier.

Parameters

metrics (Dict[str, Callable]) – The metrics to evaluate in a dictionary with metric names as keys and callables as values.
split (BaseCrossValidator) – The sklearn cross-validator to split the data.
linear (bool) – Whether to use linear SVM. (default: True)
params (Dict, optional) – Other parameters for the SVM model. See sklearn SVC for details. (default: None)
param_grid (List[Dict], optional) – The parameter grid for the grid search. (default: None)
grid_search_scoring (Dict[str, Callable], optional) – If param_grid is given, provide metrics in grid search. If multiple metrics are given, the first one will be used to retrain the best model. (default: None)
grid_search_params (Dict, optional) –
If param_grid is given, further pass the parameters for the sklearn grid search cross-validator. See sklearn GridSearchCV for details. (default: None)

from_PyG_split(data)¶

Convert from PyG split indices of training, test, and validation sets.

Parameters: data (Data) – A PyG data object.
Returns: A list of dictionaries of split indices.
Return type: List[Dict]
Raises: ValueError – If the data object does not have the split indices.

random_split(num_samples, num_splits=1, train_ratio=0.1, test_ratio=0.8)¶

Generate split indices for training, test, and validation sets.

Parameters

num_samples (int) – The size of the dataset.
num_splits (int, optional) – The number of splits to generate. (default: 1)
train_ratio (float, optional) – The ratio of the training set. (default: 0.1)
test_ratio (float, optional) – The ratio of the test set. (default: 0.8)

Returns

A list of dictionaries of split indices.

Return type

List[Dict]

Examples

>>> random_split(10, num_splits=1, train_ratio=0.5, test_ratio=0.4)
[{'train': [3, 4, 0, 1, 2], 'test': [5, 7, 6, 8], 'valid': [9]}]