Read a FBM from a file

Florian Privé

July 2, 2018

In this vignette, you learn how to read a Filebacked Big Matrix from a text file. Package {bigreadr} is required.

Data

library(bigstatsr)
library(bigreadr)

## Warning: package 'bigreadr' was built under R version 4.2.3

## LONG CSV
df <- datasets::mtcars
csv <- fwrite2(df[rep(seq_len(nrow(df)), 500000), ], 
               tempfile(fileext = ".csv"), 
               row.names = TRUE)
format(file.size(csv), big.mark = ",")

## [1] "948,944,463"

Check file content

nlines(csv)

## [1] 1.6e+07

(first_rows <- fread2(csv, nrows = 5))

##                  V1  mpg cyl disp  hp drat    wt  qsec vs am gear carb
## 1         Mazda RX4 21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
## 2     Mazda RX4 Wag 21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
## 3        Datsun 710 22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
## 4    Hornet 4 Drive 21.4   6  258 110 3.08 3.215 19.44  1  0    3    1
## 5 Hornet Sportabout 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2

sapply(first_rows, typeof)

##          V1         mpg         cyl        disp          hp        drat 
## "character"    "double"   "integer"   "integer"   "integer"    "double" 
##          wt        qsec          vs          am        gear        carb 
##    "double"    "double"   "integer"   "integer"   "integer"   "integer"

ncol(first_rows)

## [1] 12

What you can see with these first lines:

• there are some column names (header),
• each line has 12 elements,
• the first element of each line stores some row names,
• and the other elements seems to be all numeric.

Read those data

• Read all numeric columns in an FBM

  (test <- big_read(csv, select = 2:12))

  ## A Filebacked Big Matrix of type 'double' with 16000000 rows and 11 columns.

  rbind(csv, test$backingfile)

  ##     [,1]                                                                         
  ## csv "C:\\Users\\au639593\\AppData\\Local\\Temp\\RtmpeuJKQN\\file30485001141c.csv"
  ##     "C:\\Users\\au639593\\AppData\\Local\\Temp\\RtmpeuJKQN\\file30485001141c.bk"

  attr(test, "fbm_names")

  ##  [1] "mpg"  "cyl"  "disp" "hp"   "drat" "wt"   "qsec" "vs"   "am"   "gear"
  ## [11] "carb"

• Read other non-numeric data afterwards

  meta <- fread2(csv, select = 1)
  head(meta)

  ##                  V1
  ## 1         Mazda RX4
  ## 2     Mazda RX4 Wag
  ## 3        Datsun 710
  ## 4    Hornet 4 Drive
  ## 5 Hornet Sportabout
  ## 6           Valiant

Read filtered data

## Get the filter data
filter <- fread2(csv, select = "cyl")[[1]] == 4
## Read only rows corresponding to 'filter'
(test2 <- big_read(csv, select = 2:12, filter = filter,
                   backingfile = tempfile()))

## A Filebacked Big Matrix of type 'double' with 5500000 rows and 11 columns.

test2$is_saved

## [1] TRUE

(rds <- test2$rds)

## [1] "C:\\Users\\au639593\\AppData\\Local\\Temp\\RtmpeuJKQN\\file3048541b5413.rds"

You need to read from the text file only once. To get the FBM object in another R session, just use big_attach():

(test3 <- big_attach(rds))

## A Filebacked Big Matrix of type 'double' with 5500000 rows and 11 columns.