elastic net and multicolinearity

Elastic Net and Multicollinearity

弹性网络（Elastic Net)实质上是也是一种回归，简单的说是LASSO和Ridge回归的结合。LASSO回归使用的是一阶范数的正则化，倾向于将回归系数（回归中的beta）收缩（shrinkage)为0，这样起到了变量选择的作用。岭回归的解是在线性回归基础上，加了二阶范数的正则化．当特征中某些特征存在线性相关时，

可降低了估计值（回归中的beta）的方差， 提高了估计精度。 岭回归结果表明，岭回归虽然一定程度上可以拟合模型，但容易导致回归结果失真；lasso回归虽然能刻画模型代表的现实情况，但是模型过于简单，不符合实际。弹性网回归结果表明，一方面达到了岭回归对重要特征选择的目的，另一方面又像Lasso回归那样，删除了对因变量影响较小的特征，取得了很好的效果

为什么 使用一阶范数正则化的LASSO 有变量选择功能而使用二阶范数的RIDGE 没有？详细的解释见这一篇博文：http://blog.peachdata.org/2017/02/07/Lasso-Ridge.html

共线性 不错的课程 https://onlinecourses.science.psu.edu/stat501/node/347/

When Can You Safely Ignore Multicollinearity? https://statisticalhorizons.com/multicollinearity

A good idea is to assess this using neuroimaging data:

The impact of multicollinearity on the variation of coefficient estimation when using logistic regression https://www.business-school.ed.ac.uk/crc/wp-content/uploads/sites/55/2017/02/The-Impact-of-Multicollinearity-on-the-Variation-of-Coefficient-Estimation-When-Using-Logistic-Regression-de-Jongh-and-Webste.pdf

共线性的后果和是否可以强制把coefficent规定为正

https://stats.stackexchange.com/questions/242142/positive-coefficients-in-regression-analysis

https://stats.stackexchange.com/questions/198271/regression-coefficients-seem-to-have-the-wrong-sign-can-i-force-them-to-have-a

https://stats.stackexchange.com/questions/104890/is-it-possible-in-r-or-in-general-to-force-regression-coefficients-to-be-a-cer

https://stats.stackexchange.com/questions/1580/regression-coefficients-that-flip-sign-after-including-other-predictors

Simpson's paradox

https://en.wikipedia.org/wiki/Simpson's_paradox

Ridge Regression with VIF

def vif_ridge(corr_x, pen_factors, is_corr=True):
    """variance inflation factor for Ridge regression

    assumes penalization is on standardized variables
    data should not include a constant

    Parameters
    ----------
    corr_x : array_like
        correlation matrix if is_corr=True or original data if is_corr is False.
    pen_factors : iterable
        iterable of Ridge penalization factors
    is_corr : bool
        Boolean to indicate how corr_x is interpreted, see corr_x

    Returns
    -------
    vif : ndarray
        variance inflation factors for parameters in columns and ridge
        penalization factors in rows

    could be optimized for repeated calculations
    """
    corr_x = np.asarray(corr_x)
    if not is_corr:
        corr = np.corrcoef(corr_x, rowvar=0, bias=True)
    else:
        corr = corr_x

    eye = np.eye(corr.shape[1])
    res = []
    for k in pen_factors:
        minv = np.linalg.inv(corr + k * eye)
        vif = minv.dot(corr).dot(minv)
        res.append(np.diag(vif))
    return np.asarray(res)