Collecting ASDF system dependencies

Here’s a snippet that will recursively collect all dependencies of one or more ASDF systems:

(asdf:oos 'asdf:load-op 'asdf)
(asdf:oos 'asdf:load-op 'metatilities)
 
(defun direct-dependencies (component)
  (cdadr (asdf:component-depends-on 'asdf:load-op
                                    (asdf:find-component nil component))))
 
(defun normalize-system-id (id)
  (intern (symbol-name id) "KEYWORD"))
 
(defun %effective-dependencies (components)
  "Helper function."
  (when components
    (remove-duplicates
      (append components
              (%effective-dependencies
                (mapcar #'normalize-system-id
                        (remove-if #'null
                                   (metatilities:flatten
                                     (mapcar #'direct-dependencies
                                             components))))))
      :test #'eq)))
 
(defun effective-dependencies (components)
  "Find all dependencies needed for the list of COMPONENTS
(which may be an atom, too)."
  (let ((components (metatilities:ensure-list components)))
    (set-difference (%effective-dependencies components) components
                    :key #'normalize-system-id)))
 
;; usage
(effective-dependencies :weblocks)
 
(:LW-COMPAT :CLOSER-MOP :MOPTILITIES :METATILITIES-BASE :TRIVIAL-GRAY-STREAMS
 :ASDF-SYSTEM-CONNECTIONS :METABANG-DYNAMIC-CLASSES :FLEXI-STREAMS :CFFI
 :SB-GROVEL :CL-CONTAINERS :METABANG-BIND :CHUNGA :CL-BASE64 :CL-FAD :CL-PPCRE
 :CL+SSL :MD5 :RFC2388 :SB-BSD-SOCKETS :SB-POSIX :URL-REWRITE :PARENSCRIPT
 :FARE-UTILS :METATILITIES :HUNCHENTOOT :CL-WHO :CL-JSON :PURI :FARE-MATCHER
 :CL-CONT :ITERATE)

Lightning link checker

Wow!

urlcheck is lightweight, concurrent url checker. It uses minimal resources, and is multicore-capable.

In contrast to popular link checkers, it uses few resources, will readily take advantage of multiple cores, and is easily killable.

Works great and is fast as lightning. Plus, it’s written in Haskell.

Arch Linux users get it from AUR, all others from their distribution or directly from source.

Smart dates in CL

Sometimes it’s convenient to present dates in a way that depends on their offset from the current time.

For example, in different resolutions: 23 seconds ago, one minute ago, two days ago.

Another example, making use of human naming conventions: yesterday, Monday (implicitly assuming the closest Monday before the current date).

In Common Lisp, without further babbling:

(load "time.lisp")
;; http://cybertiggyr.com/gene/pdl/time.lisp
;; you could also use, for example, CL-L10N.
 
(defmacro base-bind (unit-var amount (&rest var-and-radix) &body code)
  "Thanks to Alan Crowe for this wonderful macro."
  (if (endp var-and-radix)
    `(let ((,unit-var ,amount)) ,@code)
    (let ((transfer (gensym)))
      `(multiple-value-bind (,transfer ,unit-var)
         (floor ,amount ,(cadar var-and-radix))
         (base-bind ,(caar var-and-radix) ,transfer ,(cdr var-and-radix)
                    ,@code)))))
 
(defun smart-date (then)
  (let ((now (get-universal-time)))
    (base-bind now-sec now ((now-min 60) (now-hour 60) (now-day 24))
      (base-bind then-sec then ((then-min 60) (then-hour 60) (then-day 24))
        (base-bind diff-sec (- now then) ((diff-min 60) (diff-hour 60) (diff-day 24))
          (cond
            ;; add more stuff here (e.g. negative offsets) and modify to suit your needs
            ((> diff-day 6) (CYBERTIGGYR-TIME:format-time nil CYBERTIGGYR-TIME:*FORMAT-TIME-FULL* then))
            ((> diff-day 1) (CYBERTIGGYR-TIME:format-time nil "%A" then))
            ((= diff-day 1) "Yesterday")
            ((> diff-hour 0) (format nil "~Dh~Dm ago" diff-hour diff-min))
            ((> diff-min 0) (format nil "~Dm~Ds ago" diff-min diff-sec))
            (t (format nil "~D seconds ago" diff-sec))))))))
 
; demonstration/test
(loop for offset in (list 36 90 120 130 3599 3600 3601 86400 86500 173000 14290010)
      do (format t "~D: ~A~%" offset (smart-date (- (get-universal-time) offset))))
 
; output:
36: 36 seconds ago
90: 1m30s ago
120: 2m0s ago
130: 2m10s ago
3599: 59m59s ago
3600: 1h0m ago
3601: 1h0m ago
86400: Yesterday
86500: Yesterday
173000: Tuesday
14290010: Sunday, 2008 January 27, 03:06 +1

Again, I’d like to see solutions from other languages.

Batch-renaming files

I needed to number all image files in the current directory:

# current:
16090-04.png       PySolFC_1.png
h3teampysol_20080303103813.jpg  pysol.gif
pysol_420_2.gif 175928_large.jpeg  Pysol.jpg
linux-game-pysol03.png          pysol460_big2.jpg
 
# goal:
pysolfc1.png  pysolfc2.jpeg  pysolfc3.gif
pysolfc4.jpg  pysolfc5.png  pysolfc6.jpg
pysolfc7.png  pysolfc8.gif  pysolfc9.jpg

Surprisingly easy in Common Lisp:

(loop for file in (directory "*")
     for i from 1
     do (rename-file file (format nil "pysolfc~D" i)))

In plain sh programming, I would have had to extract directory component, basename and extension, modify the basename without the extension and put them all together again. And keep track of the index manually, too.

I suppose some shell guru might come up with a neat (and probably quite unreadable) solution, but I’d rather stick to Common Lisp.

I’m curious about other solutions, esp. in Shell (ZSH allowed!), Python, Ruby and less-known languages.

Porting Perl’s qq to Common Lisp

Perl has the useful qq operator which lets you specify an arbitrary delimiter for the string following it:

# here with exclamation mark:
qq!we "often" use "quotes" "here".!

In Common Lisp, this would be useful as well, especially in docstrings and when generating foreign language code (think JavaScript without Parenscript, for example).

Let over Lambda shows us the useful sharp-doublequote reader macro that lets #" and "# act as delimiters.

This already helps a lot and looks very good, but sometimes you have a lot of double quotes and the terminating combination "# inside one string.

Take a look at this piece of JQuery code in Lisp (CL-WHO html generation):

(:a :onclick (format nil "$(\"#content\").load(\"~A.clhp\"); return true;" id))

Sharp-doublequote won’t work here because of "#. We can’t use single quotes here either because Hunchentoot will delimited the onclick part with them. We could probably add a space between " and #, but it would be a kludge and might not work in other cases anyway.

The bottom line is that using a fixed character or character combination won’t work for all cases (except when the delimiter is really long like MIME boundary strings, but this is obviously impractical).

So letting the user choose the delimiter on a case-by-case basis is a smart decision (as long as it is not overused and clutters the code with all sorts of delimiters).

The following code provides this functionality:

(defun |#q-reader| (stream sub-char numarg)
  (declare (ignore sub-char numarg))
  (let ((terminator (read-char stream)))
    (loop for ch = (read-char stream)
          until (eql ch terminator)
          collect ch into chars
          finally (return (coerce chars 'string)))))
 
(set-dispatch-macro-character
    #\# #\q #'|#q-reader|)

Quick test:

% clisp -repl qq.lisp
[1]> "foo"
"foo"
[2]> #q|foo|
"foo"
[3]> #q|foo bar baz|
"foo bar baz"
[4]> #q!foo bar baz!
"foo bar baz"
[5]> #q!foo bar "baz!
"foo bar \"baz"
[6]> #q!Hello! world! ; oops
"Hello"
[7]>
*** - SYSTEM::READ-EVAL-PRINT: variable WORLD! has no value

ZSH tip: handling URLs with url-quote-magic

The awesome Z Shell lets you forget about the quoting headeaches some URLs would cause. Its url-quote-magic line editing plugin automatically quotes metacharacters like question marks, quotes and ampersands while you type or paste them.

You just have to enable it in your .zshrc (after installing ZSH, of course):

autoload -U url-quote-magic
zle -N self-insert url-quote-magic

errormarker.vim: a Vim plugin for C/C++ programming

ErrorMarker.vim will gather all errors reported by the GCC C and C++ compilers after a :make command.

Trimming silence at the beginning and end of an audio file

Here’s a simple shell script that uses the amazing Ecasound to remove any silence from the beginning and end of an audio file:

#!/bin/sh
#
# from http://osdir.com/ml/audio.ecasound.general/2005-08/msg00002.html
#
# description: removes silence from the beginning and the end
#              of a file
# version: 20050807-1
# usage: ecatrimsilence.sh <inputfile>
 
tmp=ecatrimsilence-tmp.wav
 
if test -e "$tmp" ; then
  echo "error: temp file $tmp exists, unable to continue..."
  exit 1
fi
 
if test ! -e "$1" ; then
  echo "error: input file $1 does not exist, unable to continue..."
  exit 2
fi
 
format=`ecalength -sf "$1"`
 
echo "Trimming file ${1}."
echo "Removing silence at the end..."
ecasound -q -f:${format} -i reverse,"${1}" -o "${tmp}" -ge:1,0,0 -b:256
rm -f "${1}"
echo "Removing silence at the beginning..."
ecasound -q -f:${format} -i reverse,"${tmp}" -o "${1}" -ge:1,0,0 -b:256
rm -f "${tmp}"
echo "Done."
</inputfile>

I know some GUI programs that offer this functionality (Audacity, for example), but no other command-line tool.

Do you?

Shadowing a CL function definition

Shadowing functions is useful for example when testing.
Suppose we want to build a tiny test suite around the following function:

 
(defparameter *appointment* ...)
 
(defun overdue-p ()
  (>= (get-universal-time) *appointment*))

Testing OVERDUE-P obviously requires us to test two branches: one where the appointment is overdue and one where it is not.

Let’s say that you can’t change *APPOINTMENT* in your testing context for whatever reasons (the reason here being that this example is ultra-contrived for simplicity).
Assuming a fixed value for *APPOINTMENT* we need to change the return value of CL:GET-UNIVERSAL-TIME. This can be achieved by shadowing this function for the duration of our test.

Unfortunately, shadowing a function in Common Lisp isn’t obvious.

You can’t use FLET or LABELS because they have lexical scope.
You can’t use DEFUN either because it affects the global function namespace and doesn’t let you save or restore the old definition.

The only way I know of is using the function (SETF FDEFINITION):

(let ((orig (fdefinition 'get-universal-time)))
  (setf (fdefinition 'get-universal-time) (lambda () *my-testing-time*))
  (prog1
    (overdue-p) ; you'd run some test checks against the result here
    (setf (fdefinition 'bar) orig)))

Wrapping this in a macro is left as an exercise to the reader.

This doesn’t work for special operators, and neither for FLET or LABELS. See the CLHS entry for accessor FDEFINITION.