⚠️ Alpha Software - Work in ProgressThis project is in early development and rapidly evolving. Expect breaking changes, rough edges, and incomplete documentation.
Help Wanted! If you find this useful, please consider contributing:
- Report bugs and issues you encounter
- Suggest improvements or new features
- Submit pull requests for fixes or enhancements
- Share your configuration patterns and workflows
- Help improve documentation and examples
Your feedback and contributions will help make this tool better for the entire Clojure community!
xitdb-clj is a embedded database for efficiently storing and retrieving immutable, persistent data structures.
The library provides atom-like semantics for working with the database from Clojure.
It is a Clojure interface for xitdb-java, itself a port of xitdb, written in Zig.
- Embeddable, tiny library.
- Supports writing to a file as well as purely in-memory use.
- Each transaction (done via
swap!) efficiently creates a new "copy" of the database, and past copies can still be read from. - Reading/Writing to the database is efficient, only the necessary nodes are read or written.
- Thread safe. Multiple readers, one writer.
- Append-only. The data you are writing is invisible to any reader until the very last step, when the top-level history header is updated.
- All heavy lifting done by the bare-to-the-jvm java library.
- Database files can be used from other languages, via xitdb Java library or the xitdb Zig library
Add the dependency to your project, start a REPL.
For the programmer, a xitdb database is like a Clojure atom.
reset! or swap! to reset or update, deref or @ to read.
(require '[xitdb.db :as xdb])
(def db (xdb/xit-db "my-app.db"))
;; Use it like an atom
(reset! db {:users {"alice" {:name "Alice" :age 30}
"bob" {:name "Bob" :age 25}}})
;; Read the entire database
(xdb/materialize @db)
;; => {:users {"alice" {:name "Alice", :age 30}, "bob" {:name "Bob", :age 25}}}
(get-in @db [:users "alice" :age])
;; => 30
(swap! db assoc-in [:users "alice" :age] 31)
(get-in @db [:users "alice" :age])
;; => 31One important distinction from the Clojure atom is that inside a transaction (eg. a swap!),
'change' operations on the received db argument are mutating the underlying data structure.
(with-db [db (xdb/xit-db :memory)]
(reset! db {})
(swap! db (fn [db]
(let [db1 (assoc db :foo :bar)]
(println "db1:" db1)
(println "db:" db)))))prints
db1: {:foo :bar}
db: {:foo :bar}
As you can see, (assoc db :foo :bar) changed the value of db, in contrast
to how it works with a Clojure persistent map. This is because, inside swap!,
db is referencing a WriteCursor, which writes the value to the underlying
ArrayList or HashMap objects inside xit-db-java.
The value will actually be commited to the database when the swap! function returns.
Reading from the database returns wrappers around cursors in the database file:
(type @db) ;; => xitdb.hash_map.XITDBHashMapThe returned value is a XITDBHashMap which is a wrapper around the xitdb-java's ReadHashMap,
which itself has a cursor to the tree node in the database file.
These wrappers implement the protocols for Clojure collections - vectors, lists, maps and sets,
so they behave exactly like the Clojure native data structures.
Any read operation on these types is going to return new XITDB types:
(type (get-in @db [:users "alice"])) ;; => xitdb.hash_map.XITDBHashMapSo it will not read the entire nested structure into memory, but return a 'cursor' type, which you can operate upon using Clojure functions.
Use materialize to convert a nested XITDB data structure to a native Clojure data structure:
(materialize (get-in @db [:users "alice"])) ;; => {:name "Alice" :age 31}Use filter, group-by, reduce, etc.
If you want a query engine, datascript works out of the box, you can store the datoms as a vector in the db.
Here's a taste of how your queries could look like:
(defn titles-of-songs-for-artist
[db artist]
(->> (get-in db [:songs-indices :artist artist])
(map #(get-in db [:songs % :title]))))
(defn what-is-the-most-viewed-song? [db tag]
(let [views (->> (get-in db [:songs-indices :tag tag])
(map (:songs db))
(map (juxt :id :views))
(sort-by #(parse-long (second %))))]
(get-in db [:songs (first (last views))])))
Since the database is immutable, all previous values are accessing by reading
from the respective history index.
The root data structure of a xitdb database is a ArrayList, called 'history'.
Each transaction adds a new entry into this array, which points to the latest value
of the database (usually a map).
It is also possible to create a transaction which returns the previous and current
values of the database, by setting the *return-history?* binding to true.
;; Work with history tracking
(binding [xdb/*return-history?* true]
(let [[history-index old-value new-value] (swap! db assoc :new-key "value")]
(println "old value:" old-value)
(println "new value:" new-value)))xitdb-clj builds on xitdb-java which implements:
- Hash Array Mapped Trie (HAMT) - For efficient map and set operations
- RRB Trees - For vector operations with good concatenation performance
- Structural Sharing - Minimizes memory usage across versions
- Copy-on-Write - Ensures immutability while maintaining performance
The Clojure wrapper adds:
- Idiomatic Clojure interfaces (
IAtom,IDeref) - Automatic type conversion between Clojure and Java types
- Thread-local read connections for scalability
- Integration with Clojure's sequence abstractions
- Maps - Hash maps with efficient key-value access
- Vectors - Array lists with indexed access
- Sets - Hash sets with unique element storage
- Lists - Linked lists and RRB tree-based linked array lists
- Primitives - Numbers, strings, keywords, booleans, dates.
- Read Operations: O(log16 n) for maps and vectors due to trie structure
- Write Operations: O(log16 n) with structural sharing for efficiency
- Memory Usage: Minimal overhead with automatic deduplication of identical subtrees
- Concurrency: Thread-safe with optimized read-write locks
MIT
