33.7. Berkeley DB access

33.7.1. Berkeley-DB Objects
33.7.2. Closing handles
33.7.3. Database Environment
33.7.4. Environment Configuration
33.7.5. Database Operations
33.7.6. Database Configuration
33.7.7. Database Cursor Operations
33.7.8. Lock Subsystem
33.7.9. Log Subsystem
33.7.9.1. Log Cursor Operations
33.7.9.2. Log Sequence Numbers
33.7.10. Memory Pool Subsystem
33.7.11. Replication
33.7.12. Sequences
33.7.13. Transaction Subsystem

This interface to Berkeley DB from Sleepycat Software/Oracle exports most functions in the official C API. Supported versions:

4.2
4.3
4.4
4.5
4.6
4.7
4.8
5.3

When this module is present, *FEATURES* contains the symbol :BERKELEY-DB.

See modules/berkeley-db/test.tst for sample usage.

Note

The symbol names do evolve with new Berkeley DB releases. E.g., version 4.3 introduced DB_DSYNC_LOG which was renamed to DB_LOG_DSYNC in version 4.7. The corresponding Lisp symbol was renamed from :DSYNC-LOG to :LOG-DSYNC when support for version 4.7 was added. This means that we always use Lisp names which correspond to the C names in the latest Berkeley DB release, even though we keep suport for older releases. This is convenient because the online documentation to which this manual refers documents the latest Berkeley DB release.

33.7.1. Berkeley-DB Objects

Thie module exports the following opaque STRUCTURE-OBJECT types:

BDB:DBE
environment handle
BDB:DB
database handle
BDB:DBC
cursor handle
BDB:TXN
transaction handle
BDB:LOGC
log cursor handle
BDB:MPOOLFILE
memory pool file handle
BDB:DBLOCK
lock handle

They contain the internal handle (a FFI:FOREIGN-POINTER), the LIST of parents, and the LIST of dependents.

33.7.2. Closing handles

CLOSE will close (or commit, in the case of a transaction, or put, in the case of a lock) the Berkeley-DB handle objects. The garbage-collector will also call CLOSE. Closing an object will CLOSE all its dependents and remove the object itself from the dependents lists of its parents (but see BDB:LOCK-CLOSE).

33.7.3. Database Environment

(BDB:DB-VERSION &OPTIONAL subsystems-too)

Return version information as multiple values:

  1. descriptive STRING (from db_version)
  2. major version number (FIXNUM)
  3. minor version number (FIXNUM)
  4. patch number (FIXNUM)

When the optional argument is non-NIL, returns the association list of the subsystem versions as the 5th value.

(BDB:DBE-CREATE &KEY PASSWORD ENCRYPT)
Create an environment handle (db_env_create), possibly using encryption with password (DB_ENV->set_encrypt).
(BDB:DBE-CLOSE dbe)
Close an environment (DB_ENV->close). You can also call CLOSE.
(BDB:DBE-MESSAGES dbe)
Return the verbose messages accumulated so far (requires Berkeley-DB 4.3 or better).
(BDB:DBREMOVE dbe file database &KEY TRANSACTION AUTO-COMMIT)
Remove a database (DB_ENV->dbremove).
(BDB:DBREMOVE dbe file database newname &KEY TRANSACTION AUTO-COMMIT)
Rename a database (DB_ENV->dbrename).
(BDB:DBE-OPEN dbe &KEY FLAGS HOME JOIN INIT-CDB INIT-LOCK INIT-LOG INIT-MPOOL INIT-TXN RECOVER RECOVER-FATAL USE-ENVIRON USE-ENVIRON-ROOT CREATE LOCKDOWN PRIVATE SYSTEM-MEM THREAD MODE)
Open an environment (DB_ENV->open). :FLAGS may be the value of a previous call to (BDB:DBE-GET-OPTIONS dbe :OPEN).
(BDB:DBE-REMOVE dbe &KEY HOME FORCE USE-ENVIRON USE-ENVIRON-ROOT)
Destroy an environment (DB_ENV->remove).
(BDB:WITH-DBE (var &KEY create options) &BODY body)
Create an environment, execute body, close it. create is a list of options to be passed to BDB:DBE-CREATE, options is a list of options to be passed to BDB:DBE-SET-OPTIONS.

33.7.4. Environment Configuration

(BDB:DBE-SET-OPTIONS dbe &KEY MSGFILE ERRFILE ERRPFX PASSWORD ENCRYPT LOCK-TIMEOUT TXN-TIMEOUT SHM-KEY TAS-SPINS TX-TIMESTAMP TX-MAX DATA-DIR TMP-DIR INTERMEDIATE-DIR-MODE LG-BSIZE LG-DIR LG-MAX LG-REGIONMAX NCACHE CACHESIZE CACHE LK-CONFLICTS LK-DETECT LK-MAX-LOCKERS LK-MAX-LOCKS LK-MAX-OBJECTS LOG-DIRECT LOG-DSYNC LOG-AUTO-REMOVE LOG-IN-MEMORY LOG-ZERO AUTO-COMMIT CDB-ALLDB DIRECT-DB NOLOCKING NOMMAP NOPANIC OVERWRITE PANIC-ENVIRONMENT REGION-INIT TXN-NOSYNC TXN-WRITE-NOSYNC YIELDCPU VERB-CHKPOINT VERB-DEADLOCK VERB-RECOVERY VERB-REPLICATION VERB-WAITSFOR VERBOSE)

Set some environment options using

DB_ENV->set_flagsDB_ENV->set_shm_keyDB_ENV->set_lg_regionmax
DB_ENV->log_set_config (v4.7+)DB_ENV->set_timeoutDB_ENV->set_lk_conflicts
DB_ENV->set_verboseDB_ENV->set_encryptDB_ENV->set_lk_detect
DB_ENV->set_intermediate_dir_mode (v4.7+)DB_ENV->set_errfileDB_ENV->set_lk_max_lockers
DB_ENV->set_tmp_dirDB_ENV->set_msgfileDB_ENV->set_lk_max_locks
DB_ENV->set_data_dirDB_ENV->set_errpfxDB_ENV->set_lk_max_objects
DB_ENV->set_tx_maxDB_ENV->set_lg_bsizeDB_ENV->set_cachesize
DB_ENV->set_tx_timestampDB_ENV->set_lg_dir 
DB_ENV->set_tas_spinsDB_ENV->set_lg_max 
(BDB:DBE-GET-OPTIONS dbe &OPTIONAL what)

Retrieve some environment options.

Values of what

missing
NIL
all options as a LIST
:TX-TIMESTAMP
Recover to the time specified by timestamp rather than to the most current possible date (DB_ENV->get_tx_timestamp)
:TX-MAX
the number of active transactions (DB_ENV->set_tx_max)
:DATA-DIR
list of data directories (DB_ENV->get_data_dir)
:TMP-DIR
temporary directory (DB_ENV->get_tmp_dir). May be NIL.
:INTERMEDIATE-DIR-MODE
mode for creating intermediate directories during recovery (v4.7+) (DB_ENV->get_intermediate_dir_mode).
:VERBOSE
the LIST of verbosity settings (DB_ENV->get_verbose).
:AUTO-COMMIT
:CDB-ALLDB
:DIRECT-DB
:LOG-DIRECT
:LOG-DSYNC
:LOG-AUTO-REMOVE
:LOG-IN-MEMORY
:LOG-ZERO
:NOLOCKING
:NOMMAP
:NOPANIC
:OVERWRITE
:PANIC-ENVIRONMENT
:REGION-INIT
:TXN-NOSYNC
:TXN-WRITE-NOSYNC
:YIELDCPU
:VERB-CHKPOINT
:VERB-DEADLOCK
:VERB-RECOVERY
:VERB-REPLICATION
:VERB-WAITSFOR
a BOOLEAN indicator of whether this option is set or not (DB_ENV->get_verbose, DB_ENV->get_flags, and DB_ENV->log_get_config).
:LG-BSIZE
log buffer size (DB_ENV->get_lg_bsize).
:LG-DIR
logging directory (DB_ENV->get_lg_dir).
:LG-MAX
log file size (DB_ENV->get_lg_max).
:LG-REGIONMAX
logging region size (DB_ENV->get_lg_regionmax).
:NCACHE
:CACHESIZE
:CACHE
cache parameters (DB_ENV->get_cachesize).
:LK-CONFLICTS
lock conflicts matrix (DB_ENV->get_lk_conflicts).
:LK-DETECT
automatic deadlock detection (DB_ENV->get_lk_detect).
:LK-MAX-LOCKERS
maximum number of lockers (DB_ENV->get_lk_max_lockers).
:LK-MAX-LOCKS
maximum number of locks (DB_ENV->get_lk_max_locks).
:LK-MAX-OBJECTS
maximum number of lock objects (DB_ENV->get_lk_max_objects).
:TAS-SPINS
the number of test-and-set spins (DB_ENV->get_tas_spins).
:SHM-KEY
base segment ID for shared memory regions (DB_ENV->get_shm_key).
:LOCK-TIMEOUT
:TXN-TIMEOUT
timeout values for locks or transactions in the database environment (DB_ENV->get_timeout).
:ENCRYPT
encryption flags (DB_ENV->get_encrypt_flags).
:ERRFILE
file descriptor or NIL (DB_ENV->get_errfile).
:MSGFILE
file descriptor or NIL (DB_ENV->get_msgfile).
:ERRPFX
STRING or NIL (DB_ENV->get_errpfx).
:DB-GID-SIZE
the LENGTH of the globally unique (VECTOR (UNSIGNED-BYTE 8)) which must be passed to DB_TXN->prepare.
:HOME
the home directory when open (DB_ENV->get_home).
:OPEN
the LIST of flags passed to BDB:DBE-OPEN (DB_ENV->get_open_flags).
:CACHE
database cache information (DB_ENV->get_cachesize).

33.7.5. Database Operations

(BDB:DB-CREATE dbe)
Create a database handle (db_create).
(BDB:DB-CLOSE db &KEY NOSYNC)
Close a database (DB->close). You can also call CLOSE.
(BDB:DB-DEL dbe key &KEY TRANSACTION AUTO-COMMIT)
Delete items from a database (DB->del).
(BDB:DB-FD db)
Return the file descriptor for the database (DB->fd).
(BDB:DB-GET db key &KEY ACTION AUTO-COMMIT DEGREE-2 DIRTY-READ MULTIPLE RMW TRANSACTION (ERROR T))

Get items from a database (DB->get). If :ERROR is NIL and the record is not found, no ERROR is SIGNALed, instead :NOTFOUND is returned. :ACTION should be one of

:CONSUME:GET-BOTH
:CONSUME-WAIT:SET-RECNO

(BDB:DB-PUT db key val &KEY AUTO-COMMIT ACTION TRANSACTION)

Store items into a database (DB->put). :ACTION should be one of

:APPEND:NODUPDATA:NOOVERWRITE

(BDB:DB-STAT db &KEY FAST-STAT TRANSACTION)
Return database statistics (DB->get_byteswapped, DB->get_type, DB->stat).
(BDB:DB-OPEN db file &KEY DATABASE TYPE MODE FLAGS CREATE DIRTY-READ EXCL NOMMAP RDONLY THREAD TRUNCATE AUTO-COMMIT TRANSACTION)

Open a database (DB->open). :TYPE should be one of

:BTREE:RECNO
:HASH:UNKNOWN (default)
:QUEUE 

:FLAGS may be the value of a previous call to (BDB:DB-GET-OPTIONS db :OPEN)

(BDB:DB-SYNC db)
Flush a database to stable storage (DB->sync).
(BDB:DB-TRUNCATE db &KEY TRANSACTION AUTO-COMMIT)
Empty a database (DB->truncate).
(BDB:DB-UPGRADE db file &KEY DUPSORT)
Upgrade a database (DB->upgrade).
(BDB:DB-RENAME db file database newname)
Rename a database (DB->rename).
(BDB:DB-REMOVE db file database)
Remove a database (DB->remove).
(BDB:DB-JOIN db cursor-sequence &KEY JOIN-NOSORT)
Create a specialized join cursor for use in performing equality or natural joins on secondary indices (DB->join).
(BDB:DB-KEY-RANGE db key &KEY TRANSACTION)
return an estimate of the proportion of keys that are less than, equal to, and greater than the specified key (DB->key_range). The underlying database must be of type Btree.
(BDB:DB-VERIFY db file &KEY DATABASE SALVAGE AGGRESSIVE PRINTABLE NOORDERCHK)
Verify/salvage a database (DB->verify). :SALVAGE, if supplied, should be the output file name. :DATABASE, if supplied, will force DB_ORDERCHKONLY.
(BDB:WITH-DB (var dbe file &KEY create options open) &BODY body)
Open the database, execute body, close it. create is a list of options to be passed to BDB:DB-CREATE, options is a list of options to be passed to BDB:DB-SET-OPTIONS, open is a list of options to be passed to BDB:DB-OPEN.
(BDB:DB-COMPACT db &KEY TRANSACTION START STOP FREE FILL TIMEOUT PAGES TYPE)
Call DB->compact. :FREE is either :LIST-ONLY or :SPACE. :FILL, :TIMEOUT, :PAGES are passed in DB_COMPACT. :TYPE is how the end is returned. The second return value is a BDB:DB-COMPACT.

33.7.6. Database Configuration

(BDB:DB-SET-OPTIONS db &KEY ERRFILE MSGFILE ERRPFX PASSWORD ENCRYPTION NCACHE CACHESIZE CACHE LORDER PAGESIZE BT-MINKEY H-FFACTOR H-NELEM Q-EXTENTSIZE RE-DELIM RE-LEN RE-PAD RE-SOURCE CHKSUM ENCRYPT TXN-NOT-DURABLE DUP DUPSORT INORDER RECNUM REVSPLITOFF RENUMBER SNAPSHOT)

Set some database options using

DB_ENV->set_errfileDB->set_pagesizeDB->set_re_len
DB_ENV->set_msgfileDB->set_bt_minkeyDB->set_re_pad
DB_ENV->set_errpfxDB->set_h_ffactorDB->set_re_source
DB->set_encryptDB->set_h_nelemDB->set_flags
DB->set_cachesizeDB->set_q_extentsize 
DB->set_lorderDB->set_re_delim 
(BDB:DB-GET-OPTIONS db &OPTIONAL what)

Retrieve some database options.

Values of what

missing
NIL
all options as a LIST
:FLAGS
all flags (DB_ENV->get_flags).
:CHKSUM
:ENCRYPT
:TXN-NOT-DURABLE
:DUP
:DUPSORT
:INORDER
:RECNUM
:REVSPLITOFF
:RENUMBER
:SNAPSHOT
a BOOLEAN indicator of whether this option is set or not (DB_ENV->get_verbose and DB_ENV->get_flags).
:CACHE
database cache information (DB->get_cachesize or DB_ENV->get_cachesize if the database was created within an environment).
:ENCRYPTION
encryption flags (DB_ENV->get_encrypt_flags).
:ERRFILE
file descriptor or NIL (DB_ENV->get_errfile).
:MSGFILE
file descriptor or NIL (DB_ENV->get_msgfile).
:ERRPFX
STRING or NIL (DB_ENV->get_errpfx).
:PAGESIZE
database page size (DB->get_pagesize).
:BT-MINKEY
the minimum number of key/data pairs intended to be stored on any single :BTREE leaf page underlying source file (DB->get_bt_minkey).
:H-FFACTOR
the desired density within the :HASH table (DB->get_h_ffactor).
:H-NELEM
an estimate of the final size of the :HASH table (DB->get_h_nelem).
:Q-EXTENTSIZE
the size of the extents used to hold pages in a :QUEUE database (DB->get_q_extentsize).
:RE-DELIM
the record delimiter for :RECNO databases (DB->get_re_delim).
:RE-LEN
database record length (DB->get_re_len).
:RE-PAD
database record pad byte (DB->get_re_pad).
:RE-SOURCE
the underlying source file for :RECNO databases (DB->get_re_source).
:LORDER
database byte order (DB->get_lorder).
:DBNAME
the file name and database name (DB->get_dbname)
:TRANSACTIONAL
the indicator whether the database is transactional (DB->get_transactional).
:OPEN
the flags passed to BDB:DB-OPEN (DB->get_open_flags).

Warning

Once you call a method for one type of access method, the handle can only be used for that type. The methods DB->get_re_delim and DB->get_re_source are for a :RECNO database so you cannot call them (by passing :RE-DELIM or :RE-SOURCE to this function) and then use the database handle to open a database of different type (e.g., :QUEUE).

33.7.7. Database Cursor Operations

(BDB:MAKE-DBC db &KEY DEGREE-2 DIRTY-READ WRITECURSOR TRANSACTION)
Create a cursor handle (DB->cursor).
(BDB:DBC-CLOSE cursor)
Close the cursor handle (DBCursor->close). You can also call CLOSE.
(BDB:DBC-COUNT cursor)
Return count of duplicates (DBCursor->count).
(BDB:DBC-DEL cursor &KEY CONSUME)
Delete by cursor (DBCursor->del).
(BDB:DBC-DUP cursor &KEY POSITION)
Duplicate a cursor (DBCursor->dup).
(BDB:DBC-GET cursor key data action &KEY DEGREE-2 DIRTY-READ MULTIPLE (ERROR T))

Retrieve by cursor (DBCursor->get). If :ERROR is NIL and the record is not found, no ERROR is SIGNALed, :NOTFOUND or :KEYEMPTY is returned instead, as appropriate. action should be one of

:CURRENT:GET-RECNO:NEXT-DUP:SET
:FIRST:JOIN-ITEM:NEXT-NODUP:SET-RANGE
:GET-BOTH:LAST:PREV:SET-RECNO
:GET-BOTH-RANGE:NEXT:PREV-NODUP 

(BDB:DBC-PUT cursor key data flag)
Store by cursor (DBCursor->put).
(BDB:WITH-DBC (var &REST options) &BODY body))
Open a cursor, execute body, close it. options are passed to BDB:MAKE-DBC.

33.7.8. Lock Subsystem

(BDB:LOCK-DETECT dbe action)
Perform deadlock detection (DB_ENV->lock_detect).
(BDB:LOCK-ID dbe)
Acquire a locker ID (DB_ENV->lock_id).
(BDB:LOCK-ID-FREE dbe id)
Release a locker ID (DB_ENV->lock_id_free). All associated locks should be released first.
(BDB:LOCK-GET dbe object locker mode &KEY NOWAIT)
Acquire a lock (DB_ENV->lock_get). The BDB:DBLOCK object returned by this function will not be released when the environment is closed. This permits long-lived locks.
(BDB:LOCK-PUT dbe lock)
Release a lock (DB_ENV->lock_put).
(BDB:LOCK-CLOSE lock)

Release a lock (DB_ENV->lock_put) using the environment with which it has been acquired. This is used to EXT:FINALIZE BDB:DBLOCK objects.

Warning

If that environment has already been closed, you are in a big trouble (segfault), so you better release your locks or do not drop them.

(BDB:LOCK-STAT dbe &KEY STAT-CLEAR)
Return lock subsystem statistics (DB_ENV->lock_stat).

33.7.9. Log Subsystem

(BDB:LOG-ARCHIVE dbe &KEY ARCH-ABS ARCH-DATA ARCH-LOG ARCH-REMOVE)
Return a list of log or database filenames (DB_ENV->log_archive).
(BDB:LOG-FILE dbe lsn)
Return the name of the file containing the record named by lsn (DB_ENV->log_file).
(BDB:LOG-FLUSH dbe lsn)
Flush log records to disk (DB_ENV->log_flush).
(BDB:LOG-PUT dbe data &KEY :FLUSH)
Write a log record (DB_ENV->log_put).
(BDB:LOG-STAT dbe &KEY STAT-CLEAR)
Logging subsystem statistics (DB_ENV->log_stat).

33.7.9.1. Log Cursor Operations

(BDB:LOG-CURSOR dbe)
Create a log cursor handle (DB_ENV->log_cursor).
(BDB:LOGC-CLOSE logc)
Close a log cursor handle (DB_LOGC->close).
(BDB:LOGC-GET logc action &KEY TYPE ERROR)

Retrieve a log record (DB_LOGC->get). If :ERROR is NIL and the record is not found, no ERROR is SIGNALed, :NOTFOUND is returned instead.

Valid actions

:CURRENT
:FIRST
:LAST
:NEXT
:PREV
Retrieve the appropriate record.
DB:LSN
Retrieve the specified record, as with DB_SET.

Returns two values: the datum of type specified by the :TYPE argument and the DB:LSN value of the record retrieved (when action is a DB:LSN, it is returned unchanged).

33.7.9.2. Log Sequence Numbers

Use EQUALP to check similarity of BDB:LSN objects.

(BDB:LOG-COMPARE lsn1 lsn2)
Compare two Log Sequence Numbers (log_compare).

33.7.10. Memory Pool Subsystem

not implemented yet, patches are welcome

33.7.11. Replication

not implemented yet, patches are welcome

33.7.12. Sequences

not implemented yet, patches are welcome

33.7.13. Transaction Subsystem

(BDB:TXN-BEGIN dbe &KEY DEGREE-2 PARENT DIRTY-READ NOSYNC NOWAIT SYNC)
Begin a transaction (DB_ENV->txn_begin).
(BDB:TXN-ABORT txn)
Abort a transaction (DB_TXN->abort).
(BDB:TXN-COMMIT txn &KEY NOSYNC SYNC)
Commit a transaction (DB_TXN->commit).
(BDB:TXN-DISCARD txn)
Discard a transaction (DB_TXN->discard).
(BDB:TXN-ID txn)
Return the transaction's ID (DB_TXN->id).
(BDB:TXN-CHECKPOINT dbe &KEY KBYTE MIN FORCE)
Checkpoint the transaction subsystem (DB_ENV->txn_checkpoint).
(BDB:TXN-PREPARE txn id)
Initiate the beginning of a two-phase commit (DB_TXN->prepare).
(BDB:TXN-RECOVER dbe &KEY FIRST NEXT)
Return a list of prepared but not yet resolved transactions (DB_ENV->txn_recover).
(BDB:TXN-SET-TIMEOUT txn timeout which)
Set timeout values for locks or transactions for the specified transaction (DB_TXN->set_timeout).
(BDB:TXN-STAT dbe &KEY STAT-CLEAR)
Transaction subsystem statistics (DB_ENV->txn_stat).

These notes document CLISP version 2.49.93+Last modified: 2018-02-19