• Fix issue with snp_subset() when either $fam or $map are missing.
  • Add function snp_asGeneticPos2() (and download_genetic_map()) where you can provide any reference genetic map as a data frame. This function uses linear interpolation to transform physical positions (in bp) to genetic positions (in cM).
  • Add parameter p_bounds in LDpred2-auto to provide bounds for the estimate of the polygenicity p.
  • Implement matrix accessors [,] for bed objects.
  • Use a safer detection of strong divergences in LDpred2 and lassosum2.
  • Fix C++ code that used integers to store the positions for clumping.
  • Add other architectures (AMD / ARM) as options for PLINK2.
  • Add option use_MLE in LDpred2-auto to allow, when using FALSE, for running LDpred2-auto as in previous versions (e.g. v1.10.8), which did not include alpha in the model. Default is TRUE.
  • Detect strong divergence in LDpred2-auto, and return missing values in that case.
  • Autocomplete PLINK builds to be downloaded (fix #383).
  • Extend and improve LDpred2-auto to allow for estimating h2, p, and α, a new third parameter that controls how expected effect sizes relate to minor allele frequencies.
  • Now use a sparse format for sampling betas returned in LDpred2-auto, instead of a dense matrix that could require quite some memory to store.
  • Better snp_ldsplit():
    • also return $cost2, the sum of squared sizes of the blocks,
    • for equivalent splits (with the same cost), now return the one that also minimizes cost2,
    • now return unique splits only (e.g. could get equivalent splits with different max_size).
  • Slightly change the default parameters of lassosum2:
    • delta from c(0.001, 0.005, 0.02, 0.1, 0.6, 3) to c(0.001, 0.01, 0.1, 1),
    • nlambda from 20 to 30,
    • maxiter from 500 to 1000.
  • Add a penalty multiplicative factor for delta and lambda to regularize variants with smaller GWAS sample sizes more (when they are different, as in meta-analyses with different sets of variants).
  • Now use the same updating strategy for residuals in LDpred2 as in lassosum2. This can make LDpred2-grid and LDpred2-auto an order of magnitude faster, especially for small p.
  • Add two new parameters to snp_ldsplit(): max_r2, the maximum squared correlation allowed outside blocks, and max_cost, the maximum cost of reported solutions (i.e. the sum of all squared correlations outside blocks). Using max_r2 offers an extra guarantee that the splitting is very good, and makes the function much faster by discarding lots of possible splits.
  • LDpred2-grid does not use OpenMP for parallelism anymore, it now simply uses multiple R processes.

  • LDpred2-grid and LDpred2-auto can now make use of set.seed() to get reproducible results. Note that LDpred2-inf and lassosum2 do not use any sampling.

  • Enforce scipen = 50 when writing files to turn off scientific format (e.g. for physical positions stored as double).
  • Use a better strategy for appending to an SFBM ($add_columns()).
  • Fix an issue in snp_readBGI() when using an outdated version of package {bit64}.
  • Remove parameter info from snp_cor() and bed_cor() because this correction is not useful after all.

  • snp_cor() and bed_cor() now return NaNs when e.g. the standard deviation is 0 (and warn about it). Before, these values were not reported (i.e. treated as 0).

  • You can now return information on all variants with snp_readBGI().
  • Enhance function snp_ancestry_summary() by allowing to estimate ancestry proportions after PCA projection (instead of directly using the allele frequencies).
  • Add function snp_ancestry_summary() to estimate ancestry proportions from a cohort using only its summary allele frequencies.
  • Add function snp_scaleAlpha(), which is similar to snp_scaleBinom(), but has a parameter alpha that controls the relation between the scaling and the allele frequencies.
  • Function snp_cor() now also uses the upper triangle (@uplo = "U") when the sparse correlation matrix is diagonal, so that it is easier to use with e.g. as_SFBM().
  • Add parameter return_flip_and_rev to snp_match() for whether to return internal boolean variables "_FLIP_" and "_REV_".
  • Add $perc_kept in the output of snp_ldsplit(), the percentage of initial non-zero values kept within the blocks defined.
  • Add function snp_prodBGEN() to compute a matrix product between BGEN files and a matrix (or a vector). This removes the need to read an intermediate FBM object with snp_readBGEN() to compute the product. Moreover, when using dosages, they are not rounded to two decimal places anymore.
  • Trade new parameter num_iter_change for a simpler allow_jump_sign.

  • Change defaults in LDpred2-auto to use 500 burn-in iterations (was 1000 before) followed by 200 iterations (500 before). Such a large number of iterations is usually not really needed.

  • New compact format for SFBMs which should be really useful for LDpred2 (should require about half of memory and be twice as fast). The only thing that you need to change is as_SFBM(corr0, compact = TRUE). Make sure to reinstall {bigsnpr} after updating to {bigsparser} v0.5.
  • Prepare for incoming paper on (among other things) improved robustness of LDpred2-auto:
    • add parameter shrink_corr to shrink off-diagonal elements of the LD matrix,
    • add parameter num_iter_change to control when starting to shrink the variants that change sign too much,
    • also return corr_est, the “imputed” correlations between variants and phenotypes, which can be used for post-QCing variants by comparing those to beta / sqrt(n_eff * beta_se^2 + beta^2).
  • Replace parameter s by delta in snp_lassosum2(). This new parameter delta better reflects that the lassosum model also uses L2-regularization (therefore, elastic-net regularization).

  • Now detect strong divergence in lassosum2 and LDpred2-grid, and return missing values for the corresponding effect sizes.

  • Now use a better formula for computing standard errors in snp_ldsc() when using blocks with different sizes.
  • Add parameter info to snp_cor() to correct correlations when they are computed from imputed dosage data.
  • Function snp_readBGEN() now also returns frequencies and imputation INFO scores.
  • Add function snp_lassosum2() to train the lassosum models using the exact same input data as LDpred2.
  • Add parameter report_step in snp_ldpred2_auto() to report some of the internal sampling betas.
  • Remove penalization in snp_ldsplit(). Instead, report the best splits for a range of numbers of blocks desired.
  • Penalization in snp_ldsplit() now makes more sense. Also fix a small bug that prevented splitting the last block in some cases.
  • Add function snp_ldsplit() for optimally splitting variants in nearly independent blocks of LD.
  • Add option file.type = "--gzvcf" for using gzipped VCF in snp_plinkQC().
  • Finally remove function snp_assocBGEN(); prefer reading small parts with snp_readBGEN() as a temporary bigSNP object and do the association test with e.g. big_univLinReg().
  • Add function snp_thr_correct() for correcting for winner’s curse in summary statistics when using p-value thresholding.
  • Use a better formula for the scale in LDpred2, useful when there are some variants with very large effects (e.g. explaining more than 10% phenotypic variance).

  • Simplify LDpred2; there was not really any need for initialization and ordering of the Gibbs sampler.

  • Faster cross-product with an SFBM, which should make all LDpred2 models faster.

  • Also return $postp_est, $h2_init and $p_init in LDpred2-auto.

  • Add multiple checks in snp_readBGEN() to make sure of the expected format.
  • Workaround for error could not find function "ldpred2_gibbs_auto".
  • Can now directly do as_SFBM(corr0) instead of bigsparser::as_SFBM(as(corr0, "dgCMatrix")). This should also use less memory and be faster.
  • Add option sparse to enable getting also a sparse solution in LDpred2-auto.
  • Faster as_SFBM().

  • Allow for format 01 or 1 for chromosomes in BGI files.

  • Fasten snp_match(). Also now remove duplicates by default.
  • Fix a bug when using very large correlation matrices in LDpred2 (although we do not recommend to do so).
  • All 3 LDpred2 functions now use an SFBM as input format for the correlation matrix.

  • Allow for multiple initial values for p in snp_ldpred2_auto().

  • Add function coef_to_liab() for e.g. converting heritability to the liability scale.

  • Change default of parameter alpha of function snp_cor() to 1.
  • Parallelization of clumping algorithms has been modified. Before, chromosomes were imputed in parallel. Now, chromosomes are processed sequentially, but computations within each chromosome are performed in parallel thanks to OpenMP. This should prevent major slowdowns for very large samples sizes (due to swapping).

  • Use OpenMP to parallelize other functions as well (possibly only sequential until now).

  • Can now run snp_cor() in parallel.

  • Parallelization of snp_fastImpute() has been modified. Before this version, chromosomes were imputed in parallel. Now, chromosomes are processed sequentially, but computation of correlation between variants and XGBoost models are performed using parallelization.

  • Use new class bed_light internally to make parallel algorithms faster because they have to transfer less data to clusters. Also define differently functions used in big_parallelize() for the same reason.
  • Use the new implementation of robust OGK Mahalanobis distance in {bigutilsr}.
  • Cope with new read-only option in {bigstatsr} version >= 1.1.
  • Fix a bug in snp_readBGEN() that could not handle duplicated variants or individuals.
  • When using snp_grid_PRS(), it now stores not only the FBM, but also the input parameters as attributes (the whole result basically).
  • Add 3 SCT functions snp_grid_*() to improve from Clumping and Thresholding (preprint coming soon).

  • Add snp_match() function to match between summary statistics and some SNP information.

  • Parameter is.size.in.bp is deprecated.
  • Add parameter read_as for snp_readBGEN(). It is now possible to sample BGEN probabilities as random hard calls using read_as = "random". Default remains reading probabilities as dosages.
  • When reading bed files, switch reading of Os and 2s to be consistent with other software.
  • Add function snp_assocBGEN() for computing quick association tests from BGEN files. Could be useful for quick screening of useful SNPs to read in bigSNP format. This function might be improved in the future.
  • Change url to download PLINK 1.9.
  • Add function snp_readBGEN() to read UK Biobank BGEN files in bigSNP format.
  • Add parameter is.size.in.bp to snp_autoSVD() for the clumping part.

  • Change the threshold of outlier detection in snp_autoSVD() (it now detects less outliers). See the documentation details if you don’t have any information about SNPs.

  • Keep up with {bigstatsr}.
  • Provide function snp_gene (as a gist) to get genes corresponding to ‘rs’ SNP IDs thanks to package {rsnps} from rOpenSci. See README.
  • Faster defaults + possibility to estimate correlations based on a subset of individuals for snp_fastImpute. Also store information in an FBM (instead of a data frame) so that imputation can be done by parts (you can stop the imputation by killing the R processes and come back to it later). Note that the defaults used in the Bioinformatics paper were alpha = 0.02 and size = 500 (instead of 1e-4 and 200 now, respectively). These new defaults are more stringent on the SNPs that are used, which makes the imputation faster (30 min instead of 42-48 min), without impacting accuracy (still 4.7-4.8% of errors).
  • This package won’t be on CRAN. (Okay, it has been back on CRAN since; I was just pissed at BR :D)
  • No longer download PLINK automatically (because it is a CRAN policy violation).