32.2. External Modules

32.2. External Modules
Prev	Chapter 32. Platform Specific Extensions	Next

Platform Dependent: UNIX, Win32 platforms only.

List of Examples

32.1. Create a module set with GNU libc bindings

Modules on Win32

Everything described in the section will work verbatim on Win32 when using Cygwin or MinGW, except for one thing - you will need to replace the run extension in lisp.run with the Win32 executable extension exe.

For historical reasons, all examples appear to assume UNIX and use the run file type (“extension”) for the CLISP runtime. This does not mean that they will not work on Win32.

32.2.1. Overview

External modules are a mechanism to add extensions (written in C, for example) to CLISP. Extending CLISP using an external module requires creating a module set and adding it to an existing linking set using clisp-link to prodice a new linking set which contains the extension.

Module Set

A module is a piece of code (C or Lisp) which defines extra (non-core) Lisp objects, symbols and functions. Together with link.sh, which describes how to add the module to an existing CLISP, it comprises a module set.

More formally, module set is a directory containing:

link.sh: some /bin/sh commands, which prepare the directory for linking, and set some environment variables, see the section called “Module set variables”
all other files that define the module functionality: needed by link.sh

In link.sh the module set directory is referred to as $modulename/.

A module name must consist of the characters A-Z, a-z, _, 0-9.

The module name “clisp” is reserved.

Linking Set

A linking set is a collection of files (runtime, memory image &c) which allows performing two major tasks:

Running CLISP: to run a CLISP contained in some linking set directory, call
```
$ directory/lisp.run -M directory/lispinit.mem
```
or
```
$ clisp -K directory
```
(recommended, since it also passes -B to the runtime).
Adding a module set to create a new linking set which will contain the module functionality.

The CLISP build directory contains three linking sets in directories boot, base, and full, and a CLISP installation normally contains two linking sets: base, and full

More formally, a linking set is a directory containing at least these files:

lisp.run

the executable runtime

lispinit.mem

the memory image

modules.h

the list of modules contained in this linking set

modules.o

the compiled list of modules contained in this linking set

makevars

some /bin/sh commands, setting the variables

`CC`	the C compiler
`CPPFLAGS`	flags for the C compiler, when preprocessing or compiling
`CFLAGS`	flags for the C compiler, when compiling or linking
`CLFLAGS`	flags for the C compiler, when linking
`LIBS`	libraries to use when linking (either present in the linking set directory, or system-wide)
`X_LIBS`	additional X Window System libraries to use
`RANLIB`	the ranlib command
`FILES`	the list of files needed when linking

all the FILES

listed in makevars

Manipulating module sets and linking sets

Use clisp-link to create and install module sets and add them to linking sets.

Module set variables

The following variables should be defined in link.sh.

NEW_FILES: the space-separated list of object files that belong to the module set and will belong to every new linking set linked with this module set.
NEW_LIBS: the space-separated list of object files, libraries or C compiler switches that need to be passed to the C compiler when linking the lisp.run belonging to a new linking set.
NEW_MODULES: the space-separated list of the module names belonging to the module set. Normally, every #P".c" file in the module set defines a module of its own. The module name is usually derived from the file name.
TO_LOAD: the space-separated list of Lisp files to load before building the lispinit.mem belonging to a new linking set.
TO_PRELOAD (optional): the space-separated list of Lisp files to load into an intermediate lispinit.mem file, before building the lispinit.mem belonging to a new linking set. This variable is usually used to create (or unlock) the Lisp PACKAGEs which must be present when the new #P".c" files are initialized. E.g., the FFI:DEF-CALL-IN functions must reside in already defined packages; see Example 32.7, “Calling Lisp from C”. You can find a live example in modules/syscalls/preload.lisp and modules/syscalls/link.sh.in.
Warning
If you are unlocking a package, you must also DELETE it from CUSTOM:*SYSTEM-PACKAGE-LIST* (see Section 31.2, “Saving an Image”) here and re-add it to CUSTOM:*SYSTEM-PACKAGE-LIST* in one of the TO_LOAD files. See, e.g., modules/i18n/preload.lisp and modules/i18n/link.sh.in.

32.2.2. Module initialization

Each module has two initialization functions:

void module__name__init_function_1 (struct module_t* module)

called only once when CLISP discovers while loading a memory image that there is a module present in the executable (lisp.run) which was not present at the time the image was saved. It can be used to create Lisp objects, e.g. functions or keywords, and is indeed used for that purpose by modprep.

You do not have to define this function yourself; modprep and “FFI” will do that for you.

If you use “FFI”, (FFI:C-LINES :init-once ...) will add code to this function.

Warning

The PACKAGEs must already exist and be unlocked, cf. TO_PRELOAD.

Warning

If you are using modprep and defining your own “init-once” function, it must call the module__name__init_function_1__modprep function!

void module__name__init_function_2 (struct module_t* module)

called every time CLISP starts. It can be used to bind names to foreign addresses, since the address will be different in each invocation of CLISP, and is indeed used for that purpose by “FFI” (e.g., by FFI:DEF-CALL-OUT). It can also be used to set parameters of the libraries to which the module interfaces, e.g., the pcre module sets pcre_malloc and pcre_free.

You do not have to define this function yourself; modprep and “FFI” will do that for you.

If you use “FFI”, (FFI:C-LINES :init-always ...) will add code to this function.

name is the module name.

32.2.3. Module finalization

Each module has a finalization function

void module__name__fini_function (struct module_t* module)

called before exiting CLISP.

You do not have to define this function yourself; modprep and “FFI” will do that for you.

If you use “FFI”, (FFI:C-LINES :fini ...) will add code to this function.

name is the module name.

32.2.4. Function `EXT:MODULE-INFO`

Function (EXT:MODULE-INFO &OPTIONAL name verbose) allows one to inquire about what modules are available in the currently running image. When called without arguments, it returns the list of module names, starting with “clisp”. When name is supplied and names a module, 3 values are returned - name, subr-count, object-count. When verbose is non-NIL, the full list of module lisp function names written in C (Subrs) and the full list of internal lisp objects available in C code are additionally returned for the total of 5 values.

When name is :FFI, returns the list of shared libraries opened using :LIBRARY. When verbose is non-NIL, return the association list of DLL names and all foreign objects associated with it.

32.2.5. Dynamic module loading

Platform Dependent: Only in CLISP built without configure flag `--without-dynamic-modules`.

Note

Dynamic loading does not work on all operating systems (dlopen or equivalent is required).

Note

You will probably never need to call the function SYS::DYNLOAD-MODULES explicitly (this is why it is in “SYSTEM” and not in “EXT”). You should install your module with

$ clisp-link install name

and load it with (REQUIRE name).

Function (SYS::DYNLOAD-MODULES filename ({name}+)) loads a shared object file or library containing a number of named external CLISP modules.

Note

This facility cannot be used to access arbitrary shared libraries. To do that, use the :LIBRARY argument to FFI:DEF-CALL-OUT and FFI:DEF-C-VAR instead.

External modules for CLISP are shared objects (dynamic libraries) that contain the module__name__subr_tab variable, among others. This serves to register external functions which operate on Lisp-level structures with CLISP.

To use dlopen with modules, you should add -fPIC to the module's compilation options. Something like

$ cc -shared -o name.so name.o

may be needed to produce the shared object file.

32.2.6. Example

List of Examples

32.1. Create a module set with GNU libc bindings

Example 32.1. Create a module set with GNU libc bindings

To link in the “FFI” bindings for the GNU/Linux operating system, the following steps are needed. (Step 1 and step 2 need not be executed in this order.)

Create a new module set

$ clisp-link create linux /pathname/bindings/linux.c

Modify the newly created linux/link.sh
1. add -lm to the libraries
  replace
```
NEW_LIBS="$file_list"
```
  with
```
NEW_LIBS="$file_list -lm"
```
2. load linux.fas before saving the memory image
  replace
```
TO_LOAD=''
```
  with
```
TO_LOAD='/pathname/bindings/linux.fas'
```

Compile linux.lisp, creating linux.c

$ clisp -c /pathname/bindings/linux.lisp

Create a new linking set
```
$ clisp-link add base base+linux linux
```

Run and try it

$ base+linux/lisp.run -M base+linux/lispinit.mem -x '(linux:stat "/tmp")'

32.2.7. Module tools

32.2.7.1. Modprep
32.2.7.2. clisp.h
32.2.7.3. Exporting

There are some tools to facilitate easy module writing.

32.2.7.1. Modprep

If your module is written in C, you can pre-process your sources with modprep in the CLISP distribution and define lisp functions with the DEFUN macro:

DEFUN(MY-PACKAGE:MY-FUNCTION-NAME, arg1 arg2 &KEY FOO BAR) {
  if (!boundp(STACK_0)) STACK_0 = fixnum(0); /* BAR */
  if (!boundp(STACK_1)) STACK_1 = fixnum(1); /* FOO */
  pushSTACK(`MY-PACKAGE::SOME-SYMBOL`); /* create a symbol in the package */
  pushSTACK(`#(:THIS :IS :A :VECTOR)`); /* some vector, created once */
  pushSTACK(``MY-PACKAGE::MY-FUNCTION-NAME``); /* double `` means FUNCTION */
  VALUES1(listof(7)); /* cons up a new list and clean up the STACK */
}

Then (MY-PACKAGE:MY-FUNCTION-NAME 'A 12 :FOO T) will return (A 12 T 0 MY-PACKAGE::SOME-SYMBOL #(:THIS :IS :A :VECTOR) #<ADD-ON-SYSTEM-FUNCTION MY-PACKAGE:MY-FUNCTION-NAME>) (assuming you EXPORTed MY-FUNCTION-NAME from “MY-PACKAGE”).

Note that the arguments are passed on the STACK (last argument being the top) which has to be cleaned up before exit.

Another useful macros are:

DEFVAR: create a GC-visible private object
DEFFLAGSET: define a C function which will remove several flag arguments from the STACK and return the combined flag value
DEFCHECKER: define a map from cpp constants to lisp symbols and functions that map between them, checking that the argument is appropriate

See modules/syscalls/calls.c and other included modules for more examples and file modprep for full documentation.

Warning

If you manipulate Lisp objects, you need to watch out for GC-safety.

32.2.7.2. clisp.h

If your module is written in C, you will probably want to #include "clisp.h" to access CLISP objects. You will certainly need to read "clisp.h" and some code in included modules, but here are some important hints that you will need to keep in mind:

Lisp objects have type object.
Variables of this type are invalidated by lisp memory allocation (allocate_*() functions) - but not C allocations (malloc et al) - and must be saved on the STACK using cpp macros pushSTACK(), popSTACK() and skipSTACK().
Access object slots using the appropriate TheFoo() macro, e.g., TheCons(my_cons)->Car, but first check the type with consp().
Arguments are passed on the STACK, as illustrated in the above example.
Wrap your system calls in begin_system_call()/end_system_call() pairs. These macros, defined in "clisp.h", save and restore registers used by CLISP which could be clobbered by a system call.
If the system call could block (e.g., read) you need to use begin_blocking_system_call() and end_blocking_system_call() instead. This will allow other threads to run while yours is inside the system call. This means that garbage-collection could happen while you are inside this system call and, thus, that all objects of type object are invalidated by the call. See also Section 35.5, “The burden of garbage-collection upon the rest of CLISP” and Section 35.8, “Garbage Collection and Multithreading”.

32.2.7.3. Exporting

If your module uses “FFI” to interface to a C library, you might want to make your module package :CASE-SENSITIVE and use exporting.lisp in the CLISP distribution to make “FFI” forms and DEFUN, DEFMACRO at al export the symbols they define. See modules/netica/, modules/matlab/ and modules/bindings/ for examples.

32.2.8. Trade-offs: “FFI” vs. C modules

When deciding how to write a module: whether to use “FFI” or to stick with C and modprep, one has to take into account several issues:

Speed: C wins

“FFI” has a noticeable overhead: compare RAWSOCK:HTONS (defined in modules/rawsock/rawsock.c) with

(FFI:DEF-CALL-OUT htons (:NAME "htons") (:LIBRARY :default)
  (:ARGUMENTS (s ffi:short)) (:RETURN-TYPE ffi:short) (:LANGUAGE :stdc))

and observe that RAWSOCK:HTONS is almost 3 times as fast (this really does compare the “FFI” overhead to the normal lisp function call because htons is computationally trivial). This difference will matter only if you call a simple function very many times, in which case it would make sense to put the loop itself into C.

Portability: C wins

“FFI” is not as widely ported as CLISP, so it is possible that you will face a platform where CLISP runs but “FFI” is not present.
It is much easier to handle portability in C: observe the alternative implementations of htonl et al in modules/rawsock/rawsock.c.
Certain C structures have different layout on different platforms, and functions may take 64-bit arguments on some platforms and 32-bit arguments on others; so the “FFI” code has to track those differences, while C will mostly take care of these things for you.

Code size: “FFI” wins

You need to type much fewer characters with “FFI”, and, if you use the :LIBRARY argument to FFI:DEF-CALL-OUT and FFI:DEF-C-VAR, you do not need to leave your CLISP session to try out your code. This is a huge advantage for rapid prototyping.

UI: C wins

To produce a nice lispy UI (using &OPTIONAL and &KEYword arguments etc), you will need to write wrappers to your FFI:FOREIGN-FUNCTIONs, while in C you can do that directly. The same goes for “polymorphism”: accepting different argument types (like, e.g., POSIX:RESOLVE-HOST-IPADDR does) would require a lisp wrapper for FFI:FOREIGN-FUNCTIONs.

Learning curve: unclear

If you are comfortable with C, you might find the CLISP C module facilities (e.g., modprep) very easy to use.

CLISP “FFI”, on the other hand, is quite high-level, so, if you are more comfortable with high-level languages, you might find it easier to write “FFI” forms than C code.

Safety: unclear

One can get a segfault either way: if your FFI:DEF-CALL-OUT form does not describe the C function's expectations with respect to the arguments and return values (including ALLOCATION), you will probably learn that the hard way. If the module is written in C, all the opportunities to shoot oneself in the foot (and other body parts) are wide open (although well known to most C users). However, with C, one has to watch for GC-safety too.

System requirements: “FFI” wins

Some Linux distributions offer separate library (containing shared libraries only) and development (containing static libraries and C headers) packages.

When developing using “FFI” with :LIBRARY (and avoiding FFI:DEF-C-CONST), the development package is not needed.
When developing with modprep, the development package is necessary.
When distributing module (built with either “FFI” or modprep), only the library package is needed.

Note

The granularity of the choice is per function: the same module can use both modprep and “FFI”.

Note

It is not a good idea to have both foo.lisp and foo.c files in a module, because if you ever add an “FFI” form to the former, COMPILE-FILE will overwrite the latter.

32.2.9. Modules included in the source distribution

32.2.9.1. Base Modules
32.2.9.2. Database, Directory et al
32.2.9.3. Mathematics, Data Mining et al
32.2.9.4. Matching, File Processing et al
32.2.9.5. Communication, Networking
32.2.9.6. Graphics
32.2.9.7. Bindings
32.2.9.8. Toys and Games
32.2.9.9. Miscellaneous

A few modules come with the source distribution of CLISP (but are not necessarily built in a particular binary distribution).

To use modules, read unix/INSTALL and build CLISP in a directory build-dir with, e.g.,

$ ./configure --with-module=pcre --with-module=clx/new-clx --cbc build-dir

then run it with

$ ./build-dir/clisp -K full

This will create a base linking set with modules i18n, regexp and syscalls (and maybe readline); and a full linking set with modules clx/new-clx and pcre in addition to the 3 (or 4) base modules.

Here we list the included modules by their general theme. See Chapter 33, Extensions Implemented as Modules for individual module documentation.

32.2.9.1. Base Modules

The default build process includes the following modules in both base and full linking sets:

i18n: Internationalization of User Programs.
regexp: The POSIX Regular Expressions matching, compiling, executing.
syscalls: Use some system calls in a platform-independent way.
readline (only when both GNU readline and “FFI” are available): Some advanced readline and history features are exported using this module.

The composition of the full linking set depends on the platform and on the vendor preferences.

32.2.9.2. Database, Directory et al

gdbm: Interface to GNU DataBase Manager by Masayuki Onjo.
berkeley-db: Berkeley DB interface.
dirkey: Directory Access (LDAP, Win32 registry etc).
postgresql: Access PostgreSQL from CLISP.
oracle: Access Oracle RDBMS from CLISP; by John Hinsdale.

32.2.9.3. Mathematics, Data Mining et al

libsvm: Build Support Vector Machine models using LibSVM inside CLISP.
pari: Interface to the computer algebra system PARI.
matlab: Do matrix computations via MATLAB.
netica: Work with Bayesian belief networks and influence diagrams using Netica C API.

32.2.9.4. Matching, File Processing et al

pcre: The Perl Compatible Regular Expressions matching, compiling, executing.
zlib: Compress VECTORs using ZLIB.

32.2.9.5. Communication, Networking

rawsock: Raw socket access.
dbus: Interface to D-Bus.
fastcgi: Access FastCGI from CLISP; by John Hinsdale.

32.2.9.6. Graphics

CLX

Call Xlib functions from CLISP. Two implementations are supplied:

clx/mit-clx, from MIT https://www.x.org/archive/unsupported/lib/CLX/

the standard implementation

clx/new-clx, by Gilbert Baumann

faster, with additional features, but not quite complete yet. Please try it first and use clx/mit-clx only if clx/new-clx does not work for you. clx/new-clx comes with several demos, please try them using

$ clisp -K full -i modules/clx/new-clx/demos/clx-demos.lisp -x '(clx-demos:run-all-demos)'

and follow the intructions.

This functionality is documented in the manual https://common-lisp.net/project/cmucl/doc/clx/, also available in the CLISP source distribution as modules/clx/clx-manual.tar.gz.

gtk2

Use GTK+ and Glade to create GUI by James Bailey.

32.2.9.7. Bindings

Call the operating system functions from CLISP. The following platforms are supported:

bindings/glibc: Linux/GNU libc
bindings/win32: Win32

32.2.9.8. Toys and Games

queens: Compute the number of solutions to the n-queens problem on a n×n chessboard (a toy example for the users to explore the CLISP module system).
modules/clx/new-clx/demos/sokoban.lisp: a demo which comes with clx/new-clx.

32.2.9.9. Miscellaneous

asdf: A system definition facility.

Prev	Up	Next
32.1. Random Screen Access	Home	32.3. The Foreign Function Call Facility