Sklearn Model Evaluation and Tuning
Evaluating and tuning machine learning models are key steps to ensure model generalization ability and prediction accuracy.
Through appropriate evaluation metrics and tuning methods, model performance can be effectively improved, avoiding the risk of overfitting or underfitting.
This chapter will introduce in detail cross-validation, grid search, random search, model evaluation methods, and other content.
1. Cross-Validation
Introduce the concept of cross-validation
Cross-validation is a technique used to evaluate model performance. It divides the dataset into multiple subsets (folds) and trains and tests the model multiple times to obtain more stable and reliable evaluation results.
Cross-validation helps detect whether the model is overfitting, and can more accurately evaluate the model's generalization ability.
Common cross-validation methods include:
- K-fold cross-validation: Divide the data into K folds, select one fold as the test set in turn, use the other K-1 folds as the training set, repeat K times, and finally calculate the average of the K results.
- Leave-One-Out Cross-Validation (LOOCV): Keep only one data point as the test set each time, and the rest as the training set. This method is very time-consuming but can be used for small datasets.
- Stratified K-fold cross-validation: In K-fold, ensure that the class distribution in each fold is similar to the entire dataset, suitable for class imbalance situations.
scikit-learn provides multiple cross-validation methods, such as cross_val_score and cross_val_predict, which can help us perform cross-validation efficiently.
Use cross_val_score to perform K-fold cross-validation
The cross_val_score function is used to perform K-fold cross-validation, returning the score of each fold to help us evaluate the model's stability and performance.
Use cross_val_score to perform K-fold cross-validation:
Example
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris
# Load data
data = load_iris()
X, y = data.data, data.target
# Create model
model = RandomForestClassifier()
# Perform K-fold cross-validation
scores = cross_val_score(model, X, y, cv=5) # 5-fold cross-validation
print(f"Cross-validation scores: {scores}")
print(f"Mean accuracy: {scores.mean()}")
cv=5: Indicates performing 5-fold cross-validation.scores: Returns the score of each fold, and the final result is the average of these scores, representing the model's performance.
The output is as follows:
Cross-validation scores: [0.96666667 0.96666667 0.93333333 0.96666667 1. ] Mean accuracy: 0.9666666666666668
2. Grid Search and Random Search
Use GridSearchCV for hyperparameter tuning
GridSearchCV is a technique that finds the best hyperparameters by exhaustively searching all hyperparameter combinations.
GridSearchCV calculates the performance of each combination by providing a set of candidate values for parameters, and finally selects the best parameter combination.
Common parameters of GridSearchCV:
param_grid: The hyperparameter grid to be tuned, usually a dictionary, where keys are parameter names and values are candidate parameter values.cv: The number of folds for cross-validation, usually set to 5 or 10.
scikit-learn example of using GridSearchCV:
Example
from sklearn.svm import SVC
from sklearn.datasets import load_iris
# Load data
data = load_iris()
X, y = data.data, data.target
# Create model
model = SVC()
# Define hyperparameter grid
param_grid = {'kernel': ['linear', 'rbf'], 'C': [1, 10, 100]}
# Perform grid search
grid_search = GridSearchCV(model, param_grid, cv=5)
grid_search.fit(X, y)
# Output best parameters and best score
print(f"Best parameters: {grid_search.best_params_}")
print(f"Best score: {grid_search.best_score_}")
grid_search.best_params_: Returns the best-performing hyperparameter combination in the grid search.grid_search.best_score_: Returns the cross-validation score under the best parameter combination.
The output is as follows:
Best parameters: {'C': 1, 'kernel': 'linear'}
Best score: 0.9800000000000001
Use RandomizedSearchCV to speed up the tuning process
RandomizedSearchCV is a more efficient hyperparameter tuning method that speeds up the tuning process by randomly selecting a certain number of combinations from the hyperparameter space for evaluation.
RandomizedSearchCV is suitable for large hyperparameter spaces and can save computation time.
Common parameters of RandomizedSearchCV:
param_distributions: The hyperparameter distribution to be tuned, usually a dictionary, where values can be distribution objects (such asscipy.statsdistributions in) or a list of discrete values.n_iter: The number of iterations for random search, i.e., the number of randomly selected hyperparameter combinations.
scikit-learn example of using RandomizedSearchCV:
Example
from sklearn.svm import SVC
from sklearn.datasets import load_iris
from scipy.stats import uniform
# Load data
data = load_iris()
X, y = data.data, data.target
# Create model
model = SVC()
# Define hyperparameter distribution
param_distributions = {'C': uniform(0, 10), 'kernel': ['linear', 'rbf']}
# Perform random search
random_search = RandomizedSearchCV(model, param_distributions, n_iter=10, cv=5)
random_search.fit(X, y)
# Output best parameters and best score
print(f"Best parameters: {random_search.best_params_}")
print(f"Best score: {random_search.best_score_}")
random_search.best_params_: Returns the best-performing hyperparameter combination in the random search.random_search.best_score_: Returns the cross-validation score under the best parameter combination.
The output is as follows:
Best parameters: {'C': 8.355688344706016, 'kernel': 'rbf'}
Best score: 0.98666666666666673. Model Evaluation
Use classification_report, confusion_matrix, roc_auc_score
For classification models, we usually use metrics such as precision, recall, and F1 score to evaluate model performance.
scikit-learn provides many evaluation tools to help us deeply understand the model's performance.
classification_report- Provides information such as precision, recall, F1 score, and support (the number of samples in each class).
Example
# Assume y_test is the true labels, y_pred is the model prediction results
print(classification_report(y_test, y_pred))
confusion_matrix- The confusion matrix is used to display the performance of a classification model on each class, especially how positive classes are predicted as negative classes, and vice versa.
Example
# Assume y_test is the true labels, y_pred is the model prediction results
print(confusion_matrix(y_test, y_pred))
roc_auc_score- ROC AUC (Area Under the Receiver Operating Characteristic Curve) is a metric for evaluating classification model performance, especially suitable for imbalanced datasets. The higher the AUC value, the better the model performance.
Example
# Assume y_test is the true labels, y_pred_proba is the model predicted probability values
print(f"ROC AUC Score: {roc_auc_score(y_test, y_pred_proba)}")
Regression model evaluation: mean_squared_error, r2_score
For regression problems, commonly used evaluation metrics include Mean Squared Error (MSE) and Coefficient of Determination (R²).
mean_squared_error- Mean Squared Error is a common evaluation standard for regression models, calculating the mean of squared errors between predicted values and true values.
Example
# Assume y_test is the true values, y_pred is the predicted values
print(f"Mean Squared Error: {mean_squared_error(y_test, y_pred)}")
r2_score- The coefficient of determination R² is used to measure how well the model fits the data. The closer the value is to 1, the better the model fit.
Example
# Assume y_test is the true values, y_pred is the predicted values
print(f"R² Score: {r2_score(y_test, y_pred)}")