Space Station Lambda

About two weeks ago Franz, Inc. announced a new string utility function named STRING+ for their Common Lisp implementation.

It embodies the string concatenation paradigm of CONCATENATE, which is orthogonal to that of FORMAT. Some people don’t like FORMAT at all, but I think both have their uses.
I absolutely hate using the concatenation way for doing things like (here in JavaScript/Java/C++):

"You ate " + A + " apples, " + B + " bananas and " + C + " " + OTHER_THING + "."

The way the punctuation marks are placed here drives me totally nuts, and I’d rather prefer a lovely

"You ate ~A apples, ~A bananas and ~A ~A." A B C OTHER_THING

But the former method is useful for building file system paths and URLs, for example (MERGE-PATHNAMES aside).

So basically STRING+ is a shorthand for CONCATENATE ‘STRING. Franz claims speed improvements for small values, but I don’t need that. Short and sweet, here’s the functional equivalent:

(defun string+ (&rest parts)
  (with-output-to-string (out)
    (dolist (part parts)
      (write part :stream out :escape nil))))

Works as advertised, except for the example with the keyword symbol, where case may vary depending on your implementation’s default case conventions (CLISP prints it uppercase).

Comments(8)

Over the past few years quite a few solutions for object persistency in CL have emerged.

For my current project, I have chosen Elephant, after having looked at other popular alternatives. In this post, I’d like to talk about my reasons for choosing Elephant, taking a comparative approach. I didn’t take a look at the LW and ACL solutions since they are either not free or not portable.

Flexibility

In Elephant, I have flexibility, and in more than one way:

First, you choose your backend, and you don’t do it once and for virtual eternity. Elephant works best with Berkeley DB, but also has a performant (so I’m told) Postmodern backend, usable CL-SQL and SQLite3 backends and an experimental SEXP backend. Switching among those both in the development stage and in the production stage is (at least theoretically) easy. CL-PEREC and Submarine only work with PostgreSQL.

Second, you choose the storage model. I’m going to talk about this in a coming post, for now just accept the idea that you are in command and are able to choose the model which works best for you. Hey, doesn’t that resemble Common Lisp philosophy? :)

Third, Elephant is not an object-relational mapper. While a lot of people might that see as a disadvantage (it only stores key-value pairs at the database level). But this leads to a very flexible backend model and enables easier schema evolution.

Maturity

Elephant is used in production environments (you can read about that in the manual). The only other library being used in this way is, to my knowledge, CL-PEREC.
And in some areas of Submarine still has its still be dragons.

Liveliness

An active development and user community is vital to help you with using and hacking the library. Rucksack and CL-Prevalence don’t have that. CL-PEREC does.

Ease of use

Wow, CL-PEREC is the best negative example here. To try it, install about a dozen (no, that’s not hyperbole) packages from Darcs repositories. Then figure out from some test case output how it works. Yuk. Elephant doesn’t work out of the box either, but you only have to adapt a simple configuration file to your environment and comes with a good manual.

But ease of use is actually one of the big reasons for choosing object persistency instead of the all-popular SQL. I can just put away my objects instead of defining views, classes, relations in some DSL I don’t really wish to know details about.

Conclusion

Not mentioned here is PLOB!, which seems to me to be an unmaintained project using concepts that Elephant embodies in a clean way and expands upon.

The combination of the above factors was what drove me to use Elephant. Rucksack also seems to be sensible stuff, but it’s not mature yet (in terms of community, documentation, stability and features).

Naturally, there are some deficiencies in Elephant, but they don’t hit where it hurts. The code is a bit crufty in some spots, for example, it uses feature macros to work with the MOP instead of using a library like Closer to MOP (though that’s work in progress). It also doesn’t support advanced (semantical) schema evolution (but no other does), though it copes with slot addition and removal. When you change your storage semantices, you can convert the data manually by mapping over it.
The manual and Trac page of Elephant also state that there’s no query language, but in my experience Elephant already offers enough querying features for more purposes.

Problems being solved by none of the libraries, apart from semantical schema evolution, are function, closure and continuation serialization. But since this is Lisp you can for simple purposes just store forms and COMPILE them as needed, and there’s more than one silver lining on the horizon, see Paul’s Common Cold and David’s SB-HEAPDUMP (both SBCL-specific right now).

What are your experiences with object persistency and data storage in Lisp? I’m curious.

Comments(8)

For simplicity, it’s good to have only one store in Elephant.
Often however this doesn’t scale well with reality :)

For example, in the browser game I am developing, we have separate “worlds” with separate sets of players.
The items are to be administrated centrally, though, so the natural solution is having one shared store for them.

The easiest way to do this is having two store controllers. An alternative would be rolling your own solution using separate indexed B trees (I went down that road first), but handling things at the store controller level lets us use all the existing indexed class infrastructure.

Now there’s good and bad news: the good news is that it’s possible to do this. The bad news is that the interface is a bit clumsy. Most functions do not accept a store controller argument but refer directly to the global *store-controller*. Yuk.

So what we need to do is find the set of functions that need to refer to the alternative store controller and let them have a nice interface. For the item example (consider all Elephant symbols to be imported, I just added the package prefix to the functions for clarity):

(defun get-items-by-type (type)
  (let ((*store-controller* *item-sc*))
    (elephant:get-instances-by-class type)))

Try exceptionally hard to draw clear lines between the different store controllers, or you’ll be in imperative hell before you can say “Practical Extraction and Report Language”. If you’re nesting those functions, use dynamic variables (also known as “globals done right”):

(defun first-thing ()
  (declare (special *store-controller*)) ; refer to the dynamic binding
  (get-instances-by-value 'item 'price 5))
 
(defun do-complex-things ()
  (let ((*store-controller* *item-sc*))
    (declare (special *store-controller*)) ; provide a dynamic binding
    (first-thing)))

EDIT: As instructive as this is, Mikael Jansson has pointed out that we don’t need the whole DECLARE hanky-panky since DEFVAR/DEFPARAMETER automatically establish variables as dynamic. Seems to be another case of the “do what I mean” support in Common Lisp.

Another thing to take care of is opening and closing alternative stores.

Self-explanatory, but not that easy to find out if you’re just starting out with Elephant:

;;; init
(defvar *item-sc* nil) ; newbies note that DEFVAR only assigns to non-existing variables
(when (null *item-sc*)
  (setf *item-sc* (elephant:open-store *item-store*)) ; *item-store* defined elsewhere
 
;;; shutdown
(elephant:close-store *item-sc*)

If you’re interested in more Elephant posts, drop me a short comment.

Comments(7)

More than one week has passed since I started to post classic chiptunes. Time for another one, a real classic by jester/sanity (Volker Tripp), called Stardust Memories.

About 15 years have passed since its release at the World of Commodore compo.

See the linked first post if you want to know how to play this.

Comments(1)

Every serious programming team, even if it consists only of you, should use a distributed revision control system.
Over the last few years, I have tried the most popular. Here’s a quick comparison of them:

• Git: Jeez, everyone seems to like this one. While I might agree that it’s powerful stuff, I do not wish to spend weeks figuring out the innards of my RCS. I got things like work to do, you know.
• SVK: SVK is a hack to make SVN distributed. ‘Nuff said.
• Monotone: Easier than Git, and does the job. But still too much boilerplate.
• Bazaar: Does the job, but is a little conservative in its approach.
• Darcs: Best candidate for me, were it only more performant. Importing the Yahoo! User Interface library takes ages and load of memory. No user-defined hooks and nested repositories either. Sorry. But still great for small projects.
• Mercurial (hg): Easy to get started with. Nested repositories. Fine-grained control over user-based hooks.
  Extensions for everything (e.g. cherry picking, selective recording).

Darcs is my personal favorite, but it’s not possible to manage bigger projects with it, and you can hardly automate deployment or testing without hooks. Plus, certain virtualization software has problems with GHC’s memory allocation.

So I’m going to stay with Mercurial. See also this comparison between Darcs and Mercurial.

Comments(11)

Graphics and sound often are a major problem for free software games. But with the advancement of tools like Gimp or Inkscape this will definitely improve.

Add to it a bunch of good tutorials and the future looks bright: graphics artist Nicu has written an excellent one for isometric structures in Inkscape.

Comments(0)