Skip to contents

Memory Maps

Source: vignettes/memmap.Rmd
memmap.Rmd

Introduction

Note: you should have read vignette("altrepr") in order to understand this one

Memory mapping (memmap) refers to mapping a file on disk to an object in memory. Data is only loaded from disk as required, meaning that the entire object can be much larger than there is space for in memory.

Apparently, R contains a family of memmap ALTREP classes that provide this behaviour.

Creating a Memory Map

Seemingly, the only interface for the memmap classes provided by R is another internal function. In other words, memmap classes have no public constructor in R yet.

That doesn’t mean that altrepr doesn’t help, though. The mmap_make() function attempts to turn this internal constructor into something a bit more user friendly.

To test it, we first create a dataset to play with:

data <- rnorm(n=5)
data
#> [1] -1.0824438  1.8582515 -1.7921479 -3.0155199 -0.4661626

Then we write this dataset to a temporary file, and open it as a memory mapped vector that exists on disk!

library(altrepr)
filename <- tempfile()
mapped <- mmap_make(filename, data)
mapped
#> [1] -1.0824438  1.8582515 -1.7921479 -3.0155199 -0.4661626

Prying into the Memory Map

The classname is mmap_ + data type, which in this case is real:

alt_classname(mapped)
#> [1] "mmap_real"

The data1 slot is a pointer to the memory map itself, in memory (although it’s a kind of virtual pointer because the memory exists on disk):

alt_data1(mapped)
#> <pointer: 0x7f451612f000>

The data2 slot contains a list, whose entries are1:

  1. The name of the file we have memory mapped
  2. A length-2 vector:
    1. The size of the memory map in bytes
    2. The number of entries in the vector
  3. A length-4 vector:
    1. The SEXPTYPE of the memory map (either 13 for integer, or 14 for double)
    2. A boolean called ptrOK. It is unclear what this does or was planned to do.
    3. A boolean, which is 1 if the memory map is requested to be writable
    4. Another boolean called serOK. Also unclear what this means.
alt_data2(mapped)
#> [[1]]
#> [1] "/tmp/Rtmp4JS6iT/file145e2b0243f1"
#> 
#> [[2]]
#> [1] 40  5
#> 
#> [[3]]
#> [1] 14  1  0  0

As usual, altrepr provides a utility function for more easily accessing these fields:

mmap_details(mapped)
#> $file_name
#> [1] "/tmp/Rtmp4JS6iT/file145e2b0243f1"
#> 
#> $size_bytes
#> [1] 40
#> 
#> $length
#> [1] 5
#> 
#> $type
#> [1] "double"
#> 
#> $ptrOK
#> [1] TRUE
#> 
#> $wrtOK
#> [1] FALSE
#> 
#> $serOK
#> [1] FALSE