A Split-Apply-Combine strategy to apply common R functions to a Filebacked Big Matrix.
big_apply(
X,
a.FUN,
a.combine = NULL,
ind = cols_along(X),
ncores = 1,
block.size = block_size(nrow(X), ncores),
...
)An object of class FBM.
The function to be applied to each subset matrix.
It must take a Filebacked Big Matrix as first argument and
ind, a vector of indices, which are used to split the data.
For example, if you want to apply a function to X[ind.row, ind.col],
you may use X[ind.row, ind.col[ind]] in a.FUN.
Function to combine the results with do.call.
This function should accept multiple arguments (...). For example, you
can use c, cbind, rbind. This package also provides function plus
to add multiple arguments together. The default is NULL, in which case
the results are not combined and are returned as a list, each element being
the result of a block.
Initial vector of subsetting indices. Default is the vector of all column indices.
Number of cores used. Default doesn't use parallelism. You may use nb_cores.
Maximum number of columns (or rows, depending on how you
use ind for subsetting) read at once. Default uses block_size.
Extra arguments to be passed to a.FUN.
This function splits indices in parts, then apply a given function to each subset matrix and finally combine the results. If parallelization is used, this function splits indices in parts for parallelization, then split again them on each core, apply a given function to each part and finally combine the results (on each cluster and then from each cluster). See also the corresponding vignette.
X <- big_attachExtdata()
# get the means of each column
colMeans_sub <- function(X, ind) colMeans(X[, ind])
str(colmeans <- big_apply(X, a.FUN = colMeans_sub, a.combine = 'c'))
#> num [1:4542] 1.32 1.59 1.53 1.63 1.09 ...
# get the norms of each column
colNorms_sub <- function(X, ind) sqrt(colSums(X[, ind]^2))
str(colnorms <- big_apply(X, colNorms_sub, a.combine = 'c'))
#> num [1:4542] 33.6 38.4 37.5 39.2 29.6 ...
# get the sums of each row
# split along rows: need to change the "complete" `ind` parameter
str(rowsums <- big_apply(X, a.FUN = function(X, ind) rowSums(X[ind, ]),
ind = rows_along(X), a.combine = 'c',
block.size = 100))
#> num [1:517] 6243 6168 6242 6249 6212 ...
# it is usually preferred to split along columns
# because matrices are stored by column.
str(rowsums2 <- big_apply(X, a.FUN = function(X, ind) rowSums(X[, ind]),
a.combine = 'plus'))
#> num [1:517] 6243 6168 6242 6249 6212 ...