July 24, 2009

SBCL: Getting started with mmap()

The POSIX mmap() system call provides a convenient way to access a file’s content.

Basically you just tell the kernel that you want a specific file accessible in memory.

This shifts the responsibility for efficient and organized file access from your application to the kernel. In the meantime you simply work with an in-memory array of the file’s contents.

SBCL provides access to this system call via its SB-POSIX package.

Here’s a small hands-on example using SB-POSIX:MMAP to read a few eight-bit characters from the beginning of my system’s fstab, which starts with this line:

# This file is edited by fstab-sync - see 'man fstab-sync' for details

Let’s start with a basic convenience function in a file.

;;; mmap.lisp
(require :sb-posix)
 
(defun mmap-file (file-stream)
  "Use mmap() to map FILE-STREAM into memory."
  (sb-posix:mmap nil ; where to map (NIL if you don't care)?
                 (file-length file-stream) ; how much?
                 sb-posix:prot-read
                 sb-posix:map-shared
                 (sb-sys:fd-stream-fd file-stream)
                 0)) ; offset into the file

Now we load this file into a fresh SBCL session and work with it:

CL-USER(1): (load "mmap.lisp")
 
; loading system definition from
; /home/sky/projects/lisp/sbcl.git/contrib/sb-grovel/sb-grovel.asd into
; #<PACKAGE "ASDF1">
; registering #<SYSTEM SB-GROVEL {AAC1E19}> as SB-GROVEL
T
CL-USER(2): (mmap-file (open "/etc/fstab"))
 
#.(SB-SYS:INT-SAP #XB7FD3000)
;; This is just a 32-bit pointer to memory; keep it around.
;; Note the readable representation, it's just a wrapped integer.
CL-USER(3): (defparameter p *)
 
P
;; Let's convert this pointer to a so-called "alien".
;; See below for an explanation of what an alien is.
CL-USER(4): (sap-alien p (* char))
 
#<sb -ALIEN-INTERNALS:ALIEN-VALUE :SAP #XB7FD3000 :TYPE (* (SIGNED 8))>
CL-USER(5): (defparameter myalien (sap-alien p (* char)))
 
MYALIEN
;; An alien is just a typed pointer, let's use DESCRIBE to take a look:
CL-USER(6): (describe myalien)
 
#<SB-ALIEN-INTERNALS:ALIEN-VALUE :SAP #XB7FD3000 :TYPE (* (SIGNED 8))>
  [structure-object]
 
Slots with :INSTANCE allocation:
  SAP   = #.(SB-SYS:INT-SAP #XB7FD3000)
  TYPE  = #<SB-ALIEN-INTERNALS:ALIEN-POINTER-TYPE (* (SIGNED 8))>
;; Now we're ready to access our memory-mapped file. Get the first octet:
CL-USER(7): (deref myalien)
 
35
;; Well, I'm not *that* good at ASCII...
CL-USER(8): (code-char 35)
 
#\#
CL-USER(9): (code-char (deref myalien 1))
 
#\
CL-USER(10): (code-char (deref myalien 2))
 
#\T
CL-USER(11): (code-char (deref myalien 3))
 
#\h
CL-USER(12): (code-char (deref myalien 4))
 
#\i
CL-USER(13): (code-char (deref myalien 5))
 
#\s
;; Looks good.

There are many knobs to fiddle with when you’re working with memory-mapped files. Read mmap(3P) to find out more.

Clozure CL also supports memory-mapped files, see the manual for details.

I hope this post helps newbies to get started with SB-POSIX and SB-ALIEN and helps them to see that it’s not as hard as it might look at first sight.

Sorry for the dodgy escaping of opening braces, Wordpress sucks.

July 24th 2009 Posted to Lisp

* Comments(4)

Comments

  1. dasuxullebt
    July 24th, 2009 | 5:14 pm

    Does this work unter Mac OS and Windows, too?

  2. Leslie
    July 24th, 2009 | 6:38 pm

    The call itself is part of POSIX so it will work on most operating systems including most modern versions of Windows.

    See http://en.wikipedia.org/wiki/POSIX#POSIX-oriented_operating_systems

    It’s likely that SB-POSIX (and thus SB-POSIX:MMAP) is available on all systems that support POSIX properly.

  3. blah
    July 25th, 2009 | 3:35 pm

    AFAIK mmap() nowadays works on windows, but windows implementations of it do tend to be criminally slow compared to unix.

    The gratuitously-different-windows-native-analogue of course exists, MapViewOfFile and friends:
    http://msdn.microsoft.com/en-us/library/aa366781(VS.85).aspx#file_mapping_functions

  4. Ala'a (cmo-0)
    July 26th, 2009 | 10:04 pm

    IIRC ACL has a keyword parameter :mapped for /open/ call

    more over have a look at these discussion about io and mmap:

    http://groups.google.com/group/comp.lang.lisp/browse_frm/thread/6dac10215611f08?hl=en&q=mmap#

    http://groups.google.com/group/comp.lang.lisp/browse_frm/thread/ed5fdab0db1a80a5?hl=en&q=mmap#

Leave a reply