Efficient penalized regression methods
for genetic prediction

Florian Privé

Prediction Modelling Presentations -- October 14, 2020

Slides: https://privefl.github.io/thesis-docs/PLR-genetics.html

1 / 25

Context of application2 / 25

Application to building polygenic scores

Polygenic scores (PGS) = (predictive) scores that combine many genetic variants (linearly here)

Polygenic risk scores (PRS) = PGS for disease risk (e.g. probability of being diagnosed during lifetime)

3 / 25

Application to building polygenic scores

Polygenic scores (PGS) = (predictive) scores that combine many genetic variants (linearly here)

Polygenic risk scores (PRS) = PGS for disease risk (e.g. probability of being diagnosed during lifetime)

Major benefit of PRS: genetic variants mostly do not change during lifetime, meaning you can derive such risk from birth.

3 / 25

Application to building polygenic scores

Polygenic scores (PGS) = (predictive) scores that combine many genetic variants (linearly here)

Polygenic risk scores (PRS) = PGS for disease risk (e.g. probability of being diagnosed during lifetime)

Major benefit of PRS: genetic variants mostly do not change during lifetime, meaning you can derive such risk from birth.

What is hard: most common traits or diseases have genetic variants associated with them, usually with very small effect sizes.

3 / 25

Application to building polygenic scores

Polygenic scores (PGS) = (predictive) scores that combine many genetic variants (linearly here)

Polygenic risk scores (PRS) = PGS for disease risk (e.g. probability of being diagnosed during lifetime)

Major benefit of PRS: genetic variants mostly do not change during lifetime, meaning you can derive such risk from birth.

What is hard: most common traits or diseases have genetic variants associated with them, usually with very small effect sizes.

Limitation: prediction from genetic variants is limited by heritability.

3 / 25

Utility of polygenic scores

4 / 25

Application to predicting height

Height is up to 80% heritable. Here we can predict ~40% variance using genetics (400K individuals $\times$ 560K genetic variants).

5 / 25

Application to predicting celiac disease

Gluten intolerance, autoimmune disease with large effects on chromosome 6.

6 / 25

Penalized regression modelsA reminder7 / 25

Multiple linear regression

We want to solve

$y = \beta_0 + \beta_1 G_1 + \cdots + \beta_p G_p + \gamma_1 COV_1 + \cdots + \gamma_q COV_q + \epsilon~.$

8 / 25

Multiple linear regression

We want to solve

$y = \beta_0 + \beta_1 G_1 + \cdots + \beta_p G_p + \gamma_1 COV_1 + \cdots + \gamma_q COV_q + \epsilon~.$ Let $\beta = (\beta_0, \beta_1, \dots, \beta_p, \gamma_1, \dots, \gamma_q)$ and $X = [1; G_1; \dots;G_p; COV_1; \dots; COV_q]$ , then

$y = X \beta + \epsilon~.$

8 / 25

Multiple linear regression

We want to solve

$y = X \beta + \epsilon~.$

This is equivalent to minimizing

$||y - X \beta||_2^2 = ||\epsilon||_2^2~,$

8 / 25

Multiple linear regression

We want to solve

$y = X \beta + \epsilon~.$

This is equivalent to minimizing

$||y - X \beta||_2^2 = ||\epsilon||_2^2~,$ whose solution is

$\beta = (X^T X)^{-1} X^T y~.$

8 / 25

Multiple linear regression

We want to solve

$y = X \beta + \epsilon~.$

This is equivalent to minimizing

$||y - X \beta||_2^2 = ||\epsilon||_2^2~,$ whose solution is

$\beta = (X^T X)^{-1} X^T y~.$

What is the problem when analyzing genotype data?

8 / 25

Multiple linear regression

We want to solve

$y = X \beta + \epsilon~.$

This is equivalent to minimizing

$||y - X \beta||_2^2 = ||\epsilon||_2^2~,$ whose solution is

$\beta = (X^T X)^{-1} X^T y~.$

What is the problem when analyzing genotype data?

$n < p$

8 / 25

Penalization term -- $L_2$ regularization

Instead, we can minimize

$||y - X \beta||_2^2 + \lambda ||\beta||_2^2~,$

9 / 25

Penalization term -- $L_2$ regularization

Instead, we can minimize

$||y - X \beta||_2^2 + \lambda ||\beta||_2^2~,$ whose solution is

$\beta = (X^T X + \lambda I)^{-1} X^T y~.$

9 / 25

Penalization term -- $L_2$ regularization

Instead, we can minimize

$||y - X \beta||_2^2 + \lambda ||\beta||_2^2~,$ whose solution is

$\beta = (X^T X + \lambda I)^{-1} X^T y~.$

This is the L2-regularization ("ridge", Hoerl and Kennard, 1970); it shrinks coefficients $\beta$ towards 0.

https://doi.org/10.1080/00401706.1970.10488634

9 / 25

Penalization term -- $L_1$ regularization

Instead, we can minimize

$||y - X \beta||_2^2 + \lambda ||\beta||_1~,$

10 / 25

Penalization term -- $L_1$ regularization

Instead, we can minimize

$||y - X \beta||_2^2 + \lambda ||\beta||_1~,$ which does not have any closed form but can be solved using iterative algorithms.

10 / 25

Penalization term -- $L_1$ regularization

Instead, we can minimize

$||y - X \beta||_2^2 + \lambda ||\beta||_1~,$ which does not have any closed form but can be solved using iterative algorithms.

This is the L1-regularization ("lasso", Tibshirani, 1996); it forces some of the coefficients to be equal to 0 and can be used as a means of variable selection, leading to sparse models.

https://doi.org/10.1111/j.2517-6161.1996.tb02080.x

10 / 25

Penalization term -- $L_1$ and $L_2$ regularization

Instead, we can minimize

$||y - X \beta||_2^2 + \lambda (\alpha ||\beta||_1 + (1 - \alpha) ||\beta||_2^2)~,$

11 / 25

Penalization term -- $L_1$ and $L_2$ regularization

Instead, we can minimize

$||y - X \beta||_2^2 + \lambda (\alpha ||\beta||_1 + (1 - \alpha) ||\beta||_2^2)~,$ which does not have any closed form but can be solved using iterative algorithms ( $0 \le \alpha \le 1$ ).

11 / 25

Penalization term -- $L_1$ and $L_2$ regularization

Instead, we can minimize

$||y - X \beta||_2^2 + \lambda (\alpha ||\beta||_1 + (1 - \alpha) ||\beta||_2^2)~,$ which does not have any closed form but can be solved using iterative algorithms ( $0 \le \alpha \le 1$ ).

This is the L1- and L2-regularization ("elastic-net", Zou and Hastie, 2005); it is a compromise between the two previous penalties.

https://doi.org/10.1111/j.1467-9868.2005.00503.x

11 / 25

Advantages and drawbacks of penalization12 / 25

Advantages and drawbacks of penalization

Advantages

Makes it possible to solve linear problems when $n < p$
Generally prevents overfitting (because of smaller effects)

12 / 25

Advantages and drawbacks of penalization

Advantages

Makes it possible to solve linear problems when $n < p$
Generally prevents overfitting (because of smaller effects)

Drawback

Add at least one hyper-parameter ( $\lambda$ ) that needs to be chosen and another one if using the elastic-net regularization ( $\alpha$ )

12 / 25

Advantages and drawbacks of penalization

Advantages

Makes it possible to solve linear problems when $n < p$
Generally prevents overfitting (because of smaller effects)

Drawback

Add at least one hyper-parameter ( $\lambda$ ) that needs to be chosen and another one if using the elastic-net regularization ( $\alpha$ )

Alternative

Select a few variables before fitting the linear model (e.g. using marginal significance/p-values); heuristic: $p = n / 10$

12 / 25

Cross-validation for choosing hyper-parameters

However, if using e.g. R package {glmnet}, it can take a long time to run for genetic data.

13 / 25

A slightly different approach in {bigstatsr}14 / 25

Cross-Model Selection and Averaging (CMSA)

15 / 25

Advantages of using {bigstatsr}

automatic choice for the two hyper-parameters $\lambda$ and $\alpha$
faster (mainly because of early-stopping criterion and no need of refitting)
memory efficient (because data is stored on disk)
parallelization of fitting (easy because data on disk)

So, can be easily applied to huge data.

16 / 25

Advantages of using {bigstatsr}

automatic choice for the two hyper-parameters $\lambda$ and $\alpha$
faster (mainly because of early-stopping criterion and no need of refitting)
memory efficient (because data is stored on disk)
parallelization of fitting (easy because data on disk)

So, can be easily applied to huge data.

also two new options:
- use of different scaling (default is dividing variants by SD)
- adaptive lasso (larger marginal effects are penalized less)

16 / 25

Application to 240 phenotypes
within the UK Biobank17 / 25

Genetic data

1,117,182 variants (HapMap3)
Training based on 434,868 individuals of European ancestry +
testing in 8636 South Asians, 1803 East Asians and 6983 Africans

18 / 25

Phenotypic data

111 continuous + 129 binary = 240 phenotypes

Some examples:

multiple diseases: cancers, diabetes, autoimmune, etc.
body measures: height, BMI, BMD, etc.
blood biochemistry: cholesterol, vitamin D, etc.
ECG measures
misc

19 / 25

Results for binary phenotypes

20 / 25

Results for continuous phenotypes

21 / 25

Limitation of polygenic scores: prediction in different ancestries

Robust slope of pcor_other ~ pcor_eur, squared:

62.3% for South Asians
45.5% for East Asians
18.8% for Africans

22 / 25

How fast is the LASSO implementation?

Running time is quadratic with the number of non-zero variables.

23 / 25

Conclusion

Using R package {bigstatsr}, you can fit penalized regressions on 100s of GBs of data (any matrix-like data).
E.g. to build polygenic scores for 240 traits in the UK Biobank
However, more work is needed (e.g. to improve PGS in other ancestries)

24 / 25

Conclusion

Using R package {bigstatsr}, you can fit penalized regressions on 100s of GBs of data (any matrix-like data).
E.g. to build polygenic scores for 240 traits in the UK Biobank
However, more work is needed (e.g. to improve PGS in other ancestries)

Paper describing R packages {bigstatsr} and {bigsnpr}:
https://doi.org/10.1093/bioinformatics/bty185
Paper specifically describing the penalized regression implementation:
https://doi.org/10.1534/genetics.119.302019
Tutorial:
https://privefl.github.io/bigstatsr/articles/penalized-regressions.html

24 / 25

Thanks!

Presentation available at
https://privefl.github.io/thesis-docs/PLR-genetics.html

privefl privefl F. Privé

Slides created via the R package xaringan.

25 / 25

↑, ←, Pg Up, k	Go to previous slide
↓, →, Pg Dn, Space, j	Go to next slide
Home	Go to first slide
End	Go to last slide
Number + Return	Go to specific slide
b / m / f	Toggle blackout / mirrored / fullscreen mode
c	Clone slideshow
p	Toggle presenter mode
t	Restart the presentation timer
?, h	Toggle this help

Efficient penalized regression methodsfor genetic prediction

Florian Privé

Prediction Modelling Presentations -- October 14, 2020

Context of application

Application to building polygenic scores

Application to building polygenic scores

Application to building polygenic scores

Application to building polygenic scores

Utility of polygenic scores

Application to predicting height

Application to predicting celiac disease

Penalized regression models

A reminder

Multiple linear regression

Multiple linear regression

Multiple linear regression

Multiple linear regression

Multiple linear regression

Multiple linear regression

Penalization term -- L2L_2 regularization

Penalization term -- L2L_2 regularization

Penalization term -- L2L_2 regularization

Penalization term -- L1L_1 regularization

Penalization term -- L1L_1 regularization

Penalization term -- L1L_1 regularization

Penalization term -- L1L_1 and L2L_2 regularization

Penalization term -- L1L_1 and L2L_2 regularization

Penalization term -- L1L_1 and L2L_2 regularization

Advantages and drawbacks of penalization

Advantages and drawbacks of penalization

Advantages

Advantages and drawbacks of penalization

Advantages

Drawback

Advantages and drawbacks of penalization

Advantages

Drawback

Alternative

Cross-validation for choosing hyper-parameters

A slightly different approach in {bigstatsr}

Cross-Model Selection and Averaging (CMSA)

Advantages of using {bigstatsr}

Advantages of using {bigstatsr}

Application to 240 phenotypeswithin the UK Biobank

Genetic data

Phenotypic data

Results for binary phenotypes

Results for continuous phenotypes

Limitation of polygenic scores: prediction in different ancestries

How fast is the LASSO implementation?

Conclusion

Conclusion

Thanks!

Context of application

Help

Efficient penalized regression methods
for genetic prediction

Penalization term -- $L_2$ regularization

Penalization term -- $L_2$ regularization

Penalization term -- $L_2$ regularization

Penalization term -- $L_1$ regularization

Penalization term -- $L_1$ regularization

Penalization term -- $L_1$ regularization

Penalization term -- $L_1$ and $L_2$ regularization

Penalization term -- $L_1$ and $L_2$ regularization

Penalization term -- $L_1$ and $L_2$ regularization

Application to 240 phenotypes
within the UK Biobank