2.x Early Entry · XTDB

Issues you may strive proper now:

Be happy to combine and match the SQL and Datalog snippets – they’re interoperable!

In case you have any points with these particular examples, please tell us: hello@xtdb.com

Set up

By way of Docker or as a JVM library

;; deps.edn
{:mvn/repos {"ossrh-snapshots" {:url "https://s01.oss.sonatype.org/content material/repositories/snapshots"}}
 :deps {org.clojure/clojure {:mvn/model "1.11.1"}
        com.xtdb.labs/xtdb-api {:mvn/model "2.0.0-SNAPSHOT"}
        com.xtdb.labs/xtdb-core {:mvn/model "2.0.0-SNAPSHOT"}}
 ;; wanted on JDK16+
 :aliases {:xtdb {:jvm-opts ["--add-opens=java.base/java.nio=ALL-UNNAMED"
                             "-Dio.netty.tryReflectionSetAccessible=true"]}}}

;; Upon getting a REPL (began with clj -A:xtdb in case you’re on JDK 16+), you may create an in-memory XTDB node with:
(require '[xtdb.node :as xt.node]
         '[xtdb.datalog :as xt])

(def my-node (xt.node/start-node {:xtdb/server {:port 3001}
                                  :xtdb/pgwire {:port 5432}}))

;; Verify this API name returns efficiently
(xt/standing my-node)

# Use this one-line shell command to start out a Clojure REPL + XTDB server:
clojure -Sdeps '{:mvn/repos {"ossrh-snapshots" {:url "https://s01.oss.sonatype.org/content material/repositories/snapshots"}}
                 :deps {org.clojure/clojure {:mvn/model "1.11.1"}
                        com.bhauman/rebel-readline {:mvn/model "0.1.4"}
                        com.xtdb.labs/xtdb-api {:mvn/model "2.0.0-SNAPSHOT"}
                        com.xtdb.labs/xtdb-core {:mvn/model "2.0.0-SNAPSHOT"}
                        com.xtdb.labs/xtdb-http-server {:mvn/model "2.0.0-SNAPSHOT"}
                        com.xtdb.labs/xtdb-pgwire-server {:mvn/model "2.0.0-SNAPSHOT"}}
                 ;; wanted on JDK16+
                 :aliases {:xtdb {:jvm-opts ["--add-opens=java.base/java.nio=ALL-UNNAMED"
                             "-Dio.netty.tryReflectionSetAccessible=true"]}}}' 
        -M -e '(do (require (quote [xtdb.node :as xt.node])
                            (quote [xtdb.datalog :as xt])
                            (quote [rebel-readline.main]))
                   (def my-node (xt.node/start-node {:xtdb/server {:port 3001}
                                  :xtdb/pgwire {:port 5432}}))
                   (rebel-readline.principal/-main))' -r
        

# Boot the two.x Early Entry Docker picture in seconds.
docker pull ghcr.io/xtdb/xtdb-ea 
docker run -p 5432:5432 -p 3001:3000 ghcr.io/xtdb/xtdb-ea

Join

By way of both (1) HTTP Clojure shopper driver, (2) experimental SQL-only pgwire-server compatibility, or (3) use the embedded node began by way of your Clojure REPL (i.e. skip this step)

;; deps.edn
{:mvn/repos {"ossrh-snapshots" {:url "https://s01.oss.sonatype.org/content material/repositories/snapshots"}}
 :deps {org.clojure/clojure {:mvn/model "1.11.1"}
        com.xtdb.labs/xtdb-api {:mvn/model "2.0.0-SNAPSHOT"}
        com.xtdb.labs/xtdb-http-client-clj {:mvn/model "2.0.0-SNAPSHOT"}}}

;; Upon getting a repl began ...
(require '[xtdb.client :as xt.client]
         '[xtdb.datalog :as xt])

(def my-node (xt.shopper/start-client "http://localhost:3001"))

(xt/standing my-node)

# Use this one-line shell command to start out a Clojure REPL
# It connects to your XTDB container robotically:
clojure -Sdeps '{:mvn/repos {"ossrh-snapshots" {:url "https://s01.oss.sonatype.org/content material/repositories/snapshots"}}
                 :deps {org.clojure/clojure {:mvn/model "1.11.1"}
                        com.bhauman/rebel-readline {:mvn/model "0.1.4"}
                        com.xtdb.labs/xtdb-api {:mvn/model "2.0.0-SNAPSHOT"}
                        com.xtdb.labs/xtdb-http-client-clj {:mvn/model "2.0.0-SNAPSHOT"}}}' 
        -M -e '(do (require (quote [xtdb.client :as xt.client])
                            (quote [xtdb.datalog :as xt])
                            (quote [rebel-readline.main]))
                   (def my-node (xt.shopper/start-client "http://localhost:3001"))
                   (rebel-readline.principal/-main))' -r

# Use any Postgres-compatible shopper.
psql -h localhost

Prompt Schemaless Writes

XTDB doesn’t require any pre-defined schema or tables. Each insert, replace, and delete is immutable.

(xt/submit-tx my-node [[:put :posts {:xt/id 1234
                                     :user-id 5678
                                     :text "hello world!"}]])

(xt/q my-node '{:discover [text]
                :the place [($ :posts [text])]})

INSERT INTO posts (xt$id, user_id, textual content)
  VALUES (1234, 5678, 'Hey World!');

SELECT posts.textual content FROM posts;

First-Class Nested Knowledge

Relational querying for messy information.

;; XTDB has all the time dealt with ~arbitrary information nicely - simply smash your maps in!
(xt/submit-tx my-node [[:put :people {:xt/id 5678
                                      :name "Sarah"
                                      :friends [{:user "Dan"}
                                                {:user "Kath"}]}]])

;; Datalog now helps first-class nested lookups too:
(xt/q my-node '{:discover [friend]
                :the place [($ :people [friends])
                        [(nth friends 1) friend]]})

-- Nested information is pure!
-- XTDB SQL makes working with Arrays and Objects straightforward,
-- these will not be opaque or restricted JSONB varieties,
-- as a substitute they're seamlessly built-in with SQL:
INSERT INTO individuals (xt$id, title, associates)
  VALUES (5678, 'Sarah',
          [{'user': 'Dan'},
           {'user': 'Kath'}]);

SELECT individuals.associates[2] FROM individuals;

Native Bitemporality

No extra sustaining your personal timestamp columns for soft-delete, versioning or audit necessities.

;; Datalog makes use of as-of-now defaults
;; To view the complete historical past of a document:
(xt/q my-node '{:discover [person
                       valid-from valid-to
                       system-from system-to]
                :the place [($ :people [{:xt/id 5678
                                     :xt/* person
                                     :xt/valid-from valid-from
                                     :xt/valid-to valid-to
                                     :xt/system-from system-from
                                     :xt/system-to system-to}]
                                   {:for-valid-time :all-time
                                    :for-system-time :all-time})]})

-- XTDB SQL defaults to the SQL:2011 normal
-- utilizing 'SET VALID_TIME_DEFAULTS TO ISO_STANDARD;'
-- which means all of time seen by default
-- but it surely additionally gives different defaults for a extra intuitive expertise:
SET VALID_TIME_DEFAULTS TO AS_OF_NOW;

SELECT *
FROM individuals
         FOR ALL SYSTEM_TIME
         FOR ALL VALID_TIME
         AS x (xt$id,
               title,
               associates,
               xt$valid_from,
               xt$valid_to,
               xt$system_from,
               xt$system_to);

Cross-Time Queries

Time in XTDB is elective – however common. Superior purposes may even question time itself.

(xt/submit-tx my-node [[:put :posts
                             {:xt/id 9012 ;; original post
                              :user-id 5678
                              :text "Happy 2019!"}
                              {:for-valid-time
                               [:in #inst "2019-01-01" nil]}]
                       [:put :posts
                             {:xt/id 3456 ;; later post
                              :user-id 5678
                              :text "(New) Happy 2024!"}
                              {:for-valid-time
                               [:in #inst "2024-01-01" nil]}]
                       [:put :posts
                             {:xt/id 9012 ;; original post again
                              :user-id 5678
                              :text "Edited-2019-post: Happy 2025!"}
                              {:for-valid-time
                               [:in #inst "2025-01-01" nil]}]])

(xt/q my-node '{:discover [text]
                :the place [($ :posts [text]
                                  {:for-valid-time
                                   [:at #inst "2025-01-02"]})]})

(xt/q my-node '{:discover [text]
                :the place [($ :posts [text])]})

;; what different (variations of) posts had been legitimate throughout the identical time vary because the 2019 submit?
(xt/q my-node '{:discover [text]
                :the place [($ :posts [{:xt/id 3456
                                    :xt/valid-time vt}]
                                  {:for-valid-time :all-time})
                        ($ :posts [{:xt/id other-id
                                    :xt/valid-time other-vt} text]
                                   {:for-valid-time :all-time})
                        [(<> other-id 3456)]
                        [(overlaps? vt other-vt)]]})

INSERT INTO posts (xt$id, user_id, textual content, xt$valid_from)
  VALUES (9012, 5678, 'Completely satisfied 2019!', DATE '2019-01-01'),
         (3456, 5678, '(New) Completely satisfied 2024!', DATE '2024-01-01'),
         (9012, 5678, 'Edited-2019-post: Completely satisfied 2025!', DATE '2025-01-01');

SELECT posts.textual content FROM posts
  FOR VALID_TIME AS OF DATE '2025-01-02';

SELECT posts.textual content FROM posts;

SELECT P2.textual content
FROM posts FOR ALL VALID_TIME AS P1,
     posts FOR ALL VALID_TIME AS P2
WHERE P1.xt$id = 3456
  AND P2.xt$id <> 3456
  AND P1.VALID_TIME OVERLAPS P2.VALID_TIME;

Native Apache Arrow

XTDB embraces Apache Arrow each internally and externally.

SELECT more_posts.textual content
  FROM ARROW_TABLE('https://xtdb.com/more_posts.arrow')
  AS more_posts;

;; Coming quickly!

See Also

Intel places 1nm course of (10A) on the roadmap for 2027 — additionally plans for totally AI-automated factories with ‘Cobots’

Full Erasure

To adjust to privateness legal guidelines, erasure is important when dealing with immutable information.

;; we're going to rename :evict to :erase within the close to future
(xt/submit-tx my-node [[:evict :people 5678]])

ERASE FROM individuals WHERE individuals.xt$id=5678;

What subsequent?

Why not Learn Datalog Today! Or generate your personal playground starter challenge utilizing our Clojure templates!

New for 1.x customers

The two.x design expands on the important qualities of 1.x and offers important new capabilities:

• “Cross time” queries, e.g. temporal joins and temporal vary scans – the place 1.x is optimized just for point-in-time queries
• Large and inexpensive scale by way of a separated storage and compute structure, utilizing object storage for dealing with giant chunks of Apache Arrow information on demand
• Intensive relational algebra optimizations (e.g. subquery decorrelation)
• First-class SQL expertise alongside Datalog with a wholly new SQL parser and planner (not utilizing Apache Calcite)
• Arrange data into tables for efficiency (question planning, partitioning and many others.), introspection, and seamless SQL interoperability
• Datalog enriched with SQL-derived performance, with a SQL-based core library, “Bag” / “multi-set” semantics, and Three-Valued Logic (3VL) – supersedes 1.x Clojure classpath interop

The roadmap in the direction of Basic Availability

This roadmap is an expression of intent and is topic to alter.

• Further storage backends: a full suite of cloud storage adapters optimized for varied Kafka-like and S3-like managed companies
• Monitoring: database well being, OpenTelemetry
• Deployment: zero-configuration Kubernetes setup for managed platforms (i.e. EKS/AKS/GKS)
• Tiered caching: block administration, clustering
• Content material indexing: mild & adaptive indexes
• Benchmarking: fine-tuning towards AuctionMark, TPC-H and past
• Hardening: transaction processing, execution engine compilation, complicated Arrow utilization
• Be part of planning: further statistics (e.g. HyperLogLog, histograms)
• Language drivers: e.g. Java & Kotlin
• SQL drivers: advanced pgwire-server assist (at present all information is returned as JSON), dynamic Flight SQL
• API Completeness – for instance:
  • INFORMATION_SCHEMA: customers ought to have the opportunity introspect all information and elements of the database as built-in digital tables by the first question APIs. e.g. an preliminary model of the `xt$txs` desk is already carried out (Change Knowledge Feed performance to observe)
  • Person-definable transaction metadata: improved provenance, “Occasion sourcing with out the trouble”
  • Database VIEWs: gradual schema, writeable views, nested views, materialized views
  • Batch writes: concurrent & atomic bulk loading of knowledge
  • Pull-like API: nested information retrieval that’s extra ergonomic than scalar subqueries
  • Ongoing SQL enhancements: e.g. implement further ISO specification features and operators