SKlearn：高斯过程回归在学习期间没有改变

Posted 2023-03-12

【中文标题】SKlearn：高斯过程回归在学习期间没有改变

【英文标题】：SKlearn: Gaussian Process Regression not changed during learning

【发布时间】：2018-09-16 05:34:44

【问题描述】：

我正在尝试使用 GaussianProcessRegressor 拟合 GP，但我注意到我的超参数仍处于初始值。我在 gpr.py 中做了一些步骤，但无法确定确切的原因。使用初始值进行预测会产生一条零线。

我的数据包含 5400 个样本，每个样本有 12 个特征，映射到单个输出变量。尽管设计可能不是那么好，但我仍然希望能学到一些东西。

所需文件：

features.txt

output.txt

import pandas as pd
import numpy as np
import time
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import RBF, ConstantKernel,WhiteKernel

designmatrix = pd.read_csv('features.txt', index_col = 0)
y = pd.read_csv('output.txt', header=None, index_col = 0)

# The RBF kernel is a stationary kernel. It is also known as the “squared exponential” kernel. 
# It is parameterized by a length-scale parameter length_scale>0, which can either be a scalar (isotropic variant of the kernel) 
# or a vector with the same number of dimensions as the inputs X (anisotropic variant of the kernel). 
# 
# The ConstantKernel can be used as part of a product-kernel where it scales the magnitude of the other factor (kernel) or as 
# part of a sum-kernel, where it modifies the mean of the Gaussian process.
#
# The main use-case of the White kernel is as part of a sum-kernel where it explains the noise-component of the signal. 
# Tuning its parameter corresponds to estimating the noise-level: k(x_1, x_2) = noise_level if x_1 == x_2 else 0

kernel = ConstantKernel(0.1, (1e-23, 1e5)) * 
RBF(0.1*np.ones(designmatrix.shape[1]), (1e-23, 1e10) ) + WhiteKernel(0.1, (1e-23, 1e5))

gp = GaussianProcessRegressor(kernel=kernel, n_restarts_optimizer=0)

print('Training')
t = time.time()
gp = gp.fit(designmatrix, y)
elapsed = time.time() - t
print(elapsed)

score = gp.score(designmatrix, y)
print(score)

print("initial params")
params = gp.get_params()
print(params)
print("learned kernel params")
print(gp.kernel_.get_params())

结果如下：

initial params

'alpha': 1e-10, 'copy_X_train': True, 'kernel__k1': 1**2, 'kernel__k2': RBF(len
gth_scale=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]), 'kernel__k1__constant_value': 1
.0, 'kernel__k1__constant_value_bounds': (1e-05, 100000.0), 'kernel__k2__length_
scale': array([ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.]), 'ke
rnel__k2__length_scale_bounds': (1e-05, 100000.0), 'kernel': 1**2 * RBF(length_s
cale=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]), 'n_restarts_optimizer': 0, 'normaliz
e_y': False, 'optimizer': 'fmin_l_bfgs_b', 'random_state': None

learned kernel params

'k1': 1**2, 'k2': RBF(length_scale=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]), 'k1__
constant_value': 1.0, 'k1__constant_value_bounds': (1e-05, 100000.0), 'k2__lengt
h_scale': array([ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.]), '
k2__length_scale_bounds': (1e-05, 100000.0)

所以，内核参数不变...

有没有办法检查警告？

是我做错了什么，还是有什么可以检查的？

任何帮助将不胜感激......

本

【问题讨论】：

这看起来很奇怪，你能检查一下你得到了奇怪的结果吗：print("initial params\n") print(gp.kernel.get_params()) gp.fit(X, y) print("\nlearned kernel params") print(gp.kernel_.get_params()) 抱歉格式很糟糕，但我还没有答案！

get_params() 只返回分类的超参数，而不是学习到的属性。超参数没有改变。请参阅 attributes 部分下的 the documentation 以了解将要更改的内容。

@Vivek Kumar kernel_（注意下划线）属性确实包含标准示例的优化超参数

但是你又在内核上调用 get_params() ，它再次只会返回初始化时使用的参数。内核对象的优化参数是theta 和bounds。见this for example

@Vivek Kumar 我打电话给gp.kernel_.get_params() :)

【参考方案1】：

不是答案（还）

开始笔记

数据对于 SO 问题来说太大了，我们需要很长时间才能测试您的问题。我已将您的代码更改为仅包含每个文件的前 600 行。您粘贴在此处的代码也无法运行，我已修复。

尾注

使用python 3.6.4、scikit-learn==0.19.1 和numpy==1.14.2。

正如您在n_restarts_optimizer 的文档中看到的，如果要优化内核超参数，则需要使其大于 0。\

n_restarts_optimizer : int, optional (default: 0)
    The number of restarts of the optimizer for finding the kernel's
    parameters which maximize the log-marginal likelihood. The first run
    of the optimizer is performed from the kernel's initial parameters,
    the remaining ones (if any) from thetas sampled log-uniform randomly
    from the space of allowed theta-values. If greater than 0, all bounds
    must be finite. Note that n_restarts_optimizer == 0 implies that one
    run is performed.

因此，将代码中的值从 0 更改为 2，会产生以下输出：

import pandas as pd
import numpy as np
import time
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import RBF, ConstantKernel,WhiteKernel

designmatrix = pd.read_csv('features.txt', index_col = 0).iloc[0:600,]
y = pd.read_csv('output.txt', header=None, index_col = 0).iloc[0:600,]

# The RBF kernel is a stationary kernel. It is also known as the “squared exponential” kernel. 
# It is parameterized by a length-scale parameter length_scale>0, which can either be a scalar (isotropic variant of the kernel) 
# or a vector with the same number of dimensions as the inputs X (anisotropic variant of the kernel). 
# 
# The ConstantKernel can be used as part of a product-kernel where it scales the magnitude of the other factor (kernel) or as 
# part of a sum-kernel, where it modifies the mean of the Gaussian process.
#
# The main use-case of the White kernel is as part of a sum-kernel where it explains the noise-component of the signal. 
# Tuning its parameter corresponds to estimating the noise-level: k(x_1, x_2) = noise_level if x_1 == x_2 else 0

kernel = ConstantKernel(0.1, (1e-23, 1e5)) * \
         RBF(0.1*np.ones(designmatrix.shape[1]), (1e-23, 1e10) ) + \
         WhiteKernel(0.1, (1e-23, 1e5))

gp = GaussianProcessRegressor(kernel=kernel, n_restarts_optimizer=2)

print("initial params")
params = gp.get_params()
print(params)

print('Training')
t = time.time()
gp = gp.fit(designmatrix, y)
elapsed = time.time() - t
print(elapsed)

score = gp.score(designmatrix, y)
print(score)

print("learned kernel params")
print(gp.kernel_.get_params())

还有输出：

initial params
'alpha': 1e-10, 'copy_X_train': True, 'kernel__k1': 0.316**2 * RBF(length_scale=[0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1]), 'kernel__k2': WhiteKernel(noise_level=0.1), 'kernel__k1__k1': 0.316**2, 'kernel__k1__k2': RBF(length_scale=[0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1]), 'kernel__k1__k1__constant_value': 0.1, 'kernel__k1__k1__constant_value_bounds': (1e-23, 100000.0), 'kernel__k1__k2__length_scale': array([0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1]), 'kernel__k1__k2__length_scale_bounds': (1e-23, 10000000000.0), 'kernel__k2__noise_level': 0.1, 'kernel__k2__noise_level_bounds': (1e-23, 100000.0), 'kernel': 0.316**2 * RBF(length_scale=[0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1]) + WhiteKernel(noise_level=0.1), 'n_restarts_optimizer': 2, 'normalize_y': False, 'optimizer': 'fmin_l_bfgs_b', 'random_state': None
Training
3.9108407497406006
1.0
learned kernel params
'k1': 20.3**2 * RBF(length_scale=[0.00289, 9.29e-15, 8.81e-20, 0.00165, 2.7e+08, 3.2e+06, 0.233, 5.62e+07, 8.78e+07, 0.0169, 4.88e-21, 3.23e-20]), 'k2': WhiteKernel(noise_level=2.17e-13), 'k1__k1': 20.3**2, 'k1__k2': RBF(length_scale=[0.00289, 9.29e-15, 8.81e-20, 0.00165, 2.7e+08, 3.2e+06, 0.233, 5.62e+07, 8.78e+07, 0.0169, 4.88e-21, 3.23e-20]), 'k1__k1__constant_value': 411.28699807005, 'k1__k1__constant_value_bounds': (1e-23, 100000.0), 'k1__k2__length_scale': array([2.88935323e-03, 9.29401433e-15, 8.81112330e-20, 1.64832813e-03,
       2.70454686e+08, 3.20194179e+06, 2.32646715e-01, 5.62487948e+07,
       8.77636837e+07, 1.68642019e-02, 4.88384874e-21, 3.22536538e-20]), 'k1__k2__length_scale_bounds': (1e-23, 10000000000.0), 'k2__noise_level': 2.171274720012903e-13, 'k2__noise_level_bounds': (1e-23, 100000.0)

您能否编辑您的问题，以便您的观察可以被复制？

【讨论】：

感谢您的努力，我会更新我的sklearn版本，看看我是否可以发布我的数据

我已经添加了相关的示例文件，也许你可以再看一下。任何帮助将不胜感激@adrin

但在您的结果中，参数仍然保持不变。奇数。

@Ben，你的意思是length_scale？这是用户设置的超参数（或默认值），而不是要从数据中训练的参数 (AFAIK) (scikit-learn.org/stable/auto_examples/gaussian_process/…)

学习各向异性 RBF 的长度尺度。它定义了在多大程度上考虑附近的样本。检查你的例子，他们也改变了..