Hyperparameter tuning

Posted 2021-03-10 gaowenxingxing

超参数

定义

在拟合模型之前需要定义好的参数

适用

• Linear regression: Choosing parameters
• Ridge/lasso regression: Choosing alpha
• k-Nearest Neighbors: Choosing n_neighbors
• Parameters like alpha and k: Hyperparameters
• Hyperparameters cannot be learned by tting the model

Grid

• 超参数自动搜索模块
• 网格搜索+交叉验证
• 指定的参数范围内，按步长依次调整参数，利用调整的参数训练学习器，从所有的参数中找到在验证集上精度最高的参数，这其实是一个训练和比较的过程
  

# Import necessary modules
from sklearn.model_selection import GridSearchCV

from sklearn.linear_model import LogisticRegression
# Setup the hyperparameter grid
# 创建一个参数集
c_space = np.logspace(-5, 8, 15)
# 这里是创建一个字典保存参数集
param_grid = {'C': c_space}

# Instantiate a logistic regression classifier: logreg
# 针对回归模型进行的超参数调整
logreg = LogisticRegression()

# Instantiate the GridSearchCV object: logreg_cv
logreg_cv = GridSearchCV(logreg, param_grid, cv=5)

# Fit it to the data
logreg_cv.fit(X,y)

# Print the tuned parameters and score
# 得到最好的模型
print("Tuned Logistic Regression Parameters: {}".format(logreg_cv.best_params_)) 
# 得到最好的模型的最好的结果
print("Best score is {}".format(logreg_cv.best_score_))

<script.py> output:
    Tuned Logistic Regression Parameters: {'C': 3.727593720314938}
    Best score is 0.7708333333333334

GridSearchCV can be computationally expensive, especially if you are searching over a large hyperparameter space and dealing with multiple hyperparameters. A solution to this is to use RandomizedSearchCV, in which not all hyperparameter values are tried out. Instead, a fixed number of hyperparameter settings is sampled from specified probability distributions.
grid相当于一个for循环，会遍历每一个参数，因此，当调参很多的时候，会导致计算量非常的大，因此，使用随机抽样的随机搜索会好一些

RandomizedSearchCV的使用方法其实是和GridSearchCV一致的，但它以随机在参数空间中采样的方式代替了GridSearchCV对于参数的网格搜索，在对于有连续变量的参数时，RandomizedSearchCV会将其当作一个分布进行采样这是网格搜索做不到的，它的搜索能力取决于设定的n_iter参数，同样的给出代码
csdn

RandomizedSearchCV

• 随机搜索法
• 不是每一个参数都被选取，而是从指定概率分布的参数中，抽取一定量的参数
  我还是没太能明白？

复现

# Import necessary modules
from scipy.stats import randint
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import RandomizedSearchCV

# Setup the parameters and distributions to sample from: param_dist
# 以决策树为例，注意定一个字典的形式哦
param_dist = {"max_depth": [3, None],
              "max_features": randint(1, 9),
              "min_samples_leaf": randint(1, 9),
              "criterion": ["gini", "entropy"]}

# Instantiate a Decision Tree classifier: tree
tree = DecisionTreeClassifier()

# Instantiate the RandomizedSearchCV object: tree_cv
tree_cv = RandomizedSearchCV(tree, param_dist, cv=5)

# Fit it to the data
tree_cv.fit(X,y)

# Print the tuned parameters and score
print("Tuned Decision Tree Parameters: {}".format(tree_cv.best_params_))
print("Best score is {}".format(tree_cv.best_score_))

<script.py> output:
    Tuned Decision Tree Parameters: {'criterion': 'gini', 'max_depth': 3, 'max_features': 5, 'min_samples_leaf': 2}
    Best score is 0.7395833333333334

reference