Gradient descent is one of the well-known optimisation algorithms. However, are the regression algorithms in scikit-learn implemented with gradient descent or some other techniques?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
There are multiple approaches to optimization in scikit-learn. I will focus on generalized linear models, where a vector of coefficients needs to be estimated:
- LinearRegression and Ridge use closed-form solution $\beta=(X^TX+I\lambda)^{-1}X^TY$, but Ridge can also use stochastic gradient descent or method of conjugate gradients
- Lasso and ElasticNet use coordinate descent
- OrthogonalMatchingPursuit uses a greedy algorithm with the same name, that has $L_0$ penalty on coefficients
- ARDRegression and BayesianRidge use something like EM algorithm
- SGDRegressor and PassiveAggressiveRegressor use guess what! Stochastic gradient descent.
- HuberRegressor uses BFGS (a second-order optimization method)
