What is a query to list all partitions of a Datomic database?
This should return
[[:db.part/db] [:db.part/tx] [:db.part/user] .... ]
where .... is all of the user defined partitions.
You should be able to get a list of all partitions in the database by searching for all entities associated with the :db.part/db entity via the :db.install/partition attribute:
(ns myns
(:require [datomic.api :as d]))
(defn get-partitions [db]
(d/q '[:find ?ident :where [:db.part/db :db.install/partition ?p]
[?p :db/ident ?ident]]
db))
Note
The current version of Datomic (build 0.8.3524) has a shortcoming such that :db.part/tx and :db.part/user (two of the three built-in partitions) are treated specially and aren't actually associated with :db.part/db via :db.install/partition, so the result of the above query function won't include the two.
This problem is going to be addressed in one of the future builds of Datomic. In the meantime, you should take care of including :db.part/tx and :db.part/user in the result set yourself.
1st method - using query
=> (q '[:find ?i :where
[:db.part/db :db.install/partition ?p] [?p :db/ident ?i]]
(db conn))
2nd method - from db object
(filter #(instance? datomic.db.Partition %) (:elements (db conn)))
The second method returns sequence of datomic.db.Partition objects which may be useful if we want to get additional info about the partition.
Both methods have known bug/inconsistency: they don't return :db.part/tx and :db.part/user built-in partitions.
Related
In my schema I have the attribute :base/type that is supposed to exist for every entity created. To check that this is indeed true, I'm trying to find entities where it is missing:
[:find [?entities ...]
:in $ :where
[(missing? $ ?entities :base/type)]]
Unfortunately this gives me back:
Execution error (Exceptions$IllegalArgumentExceptionInfo) at datomic.error/arg (error.clj:57).
:db.error/insufficient-binding [?entities] not bound in expression clause: [(missing? $ ?entities :base/type)]
How should this query be constructed?
This is because your query is too general. You need at least one positive clause in where statement if you use query API. You can access raw index to get the result, though. If you have enough RAM, you can:
(def snapshot (d/db connection)) ; your db snapshot
(def all-datoms (d/datoms snapshot :eavt))
(def base-type-id (:db/id (d/entity snapshot :base/type))
(def entities (group-by #(.e %) all-datoms))
(def entities-without-base-type (map
(comp #(into {} %) (partial d/entity snapshot) first)
(filter (fn [[k l]]
(empty? (filter #(= base-type-id (.a %))
l)))
entities)))
(def only-relevant-entities (filter #(not (or (:db/ident %) (:db/txInstant %))) entities-without-base-type))
only-relevant-entities
The last filter is to get rid of attribute definitions and transactions (they are stored as datoms in the db as well!).
If you have too many entities you can chunk datoms using the async atoms API.
Using ::singer/songs as an example attribute, this is how to do the query:
[:find [?entities ...]
:in $ :where
[?entities ::singer/songs ?s]
[(missing? $ ?entities :base/type)]]
Unfortunately (in my answer) many such queries would be required until the whole database has been covered. So another query with ::song/composers, etc...
I want to represent following SQL query in datomic
SELECT A.a C.c
FROM A, B, C
WHERE A.id = B.id and B.index = C.index
What willl be datomic query representation for this?
Also, will the same datomic query work if WHERE conditions are reversed i.e. we have "B.index = C.index and A.id = B.id"?
Here is a literal translation:
(d/q '[:find ?a-val ?c-val
:where
[?A :a.id ?id]
[?B :b.id ?id]
[?B :b.index ?index]
[?C :c.index ?index]
[?A :a ?a-val]
[?C :c ?c-val]]
db)
I've written that so you should be able to see how to translate from the question to the answer.
Hopefully it's clear that using ?index in two places that is doing the same job as = in the SQL where clause.
The clauses can be reordered anyhow and remain logically equivalent. The difference will be in performance: put the most narrowing/specific clauses first for better performance.
Some caveats:
Whether you want to have distinct attributes for the index and id concepts (for example a.index and b.index or only index) is down to your requirements. Datomic has good docs and tutorials.
There are many differences between SQL and Datalog, for example no tables in Datalog (meaning variable names ?A,?B,?C are arbitrary), but again this is all well documented.
I have a datomic database that I have updated with multiple schemas. I am looking to get the entire schema back as a map seperating each schema. This is the query I have currently:
(d/q '[:find ?id
:where [:db.part/db :db.install/attribute ?p]
[?p :db/ident ?id]] db)
You want to query to find everything with :db.install/attribute, you can see an example in this gist as well as the datomic java examples repo on github.
I've reproduced the example from the gist here:
(require '[datomic.api :as d] 'clojure.pprint)
;; You can run this with bin/run in the $DATOMIC_DIR
(println "Printing database schema...")
(def conn (d/connect (first *command-line-args*))) ;; call with db-uri as arg
; Find and pretty-print each attribute in schema
(let [db (d/db conn)]
(clojure.pprint/pprint
(map #(->> % first (d/entity db) d/touch)
(d/q '[:find ?v
:where [_ :db.install/attribute ?v]]
db))))
Suppose I have one million article entities in my backend with an inst attribute called date, or one million player entities with an int attribute called points. What's a good way to select the 10 latest articles or top-scoring players?
Do I need to fetch the whole millions to the peer and then sort and drop from them?
Until getting hold of the reverse index becomes a Datomic feature, you could manually define one.
e.g. for a :db.type/instant, create an additional attribute of type :db.type/long which you would fill with
(- (Long/MAX_VALUE) (.getTime date))
and the latest 10 articles could be fetched with
(take 10 (d/index-range db reverse-attr nil nil))
Yes, you would need to fetch all the data, since there's no index that would help you out here.
I would have created my own "index" and normalized this data. You can have a separate set of N entities where you keep as many as you'd like. You could start with 10, or consider storing 100 to trade some (possibly negligible) speed for more flexibility. This index can be stored on a separate "singleton" entity that you add as part of your schema.
;; The attribute that stores the index
{:db/id #db/id[:db.part/db]
:db/ident :indexed-articles
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/many
:db.install/_attribute :db.part/db}
;; The named index entity.
{:db/id #db/id[:db.part/db]
:db/ident :articles-index}
You can have a database function that does this. Every time you insert a new entity that you want to "index", call this function.
[[:db/add tempid :article/title "Foo]
[:db/add tempid :article/date ....]
[:index-article tempid 10]]
The implementation of index-article could look like this:
{:db/id #db/id[:db.part/user]
:db/ident :index-article
:db/fn #db/fn {:lang "clojure"
:params [db article-id idx-size]
:code (concat
(map
(fn [article]
[:db/retract
(d/entid db :articles-index)
:indexed-articles
(:db/id article)])
(->> (datomic.api/entity db :articles-index)
(sort-by (fn [] ... implement me ... ))
(drop (dec idx-size))))
[[:db/add (d/entid db :articles-index) :indexed-articles article-id]])}}
Disclaimer: I haven't actually tested this function, so it probably contains errors :) The general idea is that we remove any "overflow" entities from the set, and add the new one. When idx-size is 10, we want to ensure that only 9 items are in the set, and we add our new item to it.
Now you have an entity you can lookup from index, :articles-index, and the 10 most recent articles can be looked up from the index (all refs are indexed), without causing a full database read.
;; "indexed" set of articles.
(d/entity db :articles-index)
I've been looking into this and think I have a slightly more elegant answer.
Declare your attribute as indexed with :db/index true
{:db/id #db/id[:db.part/db -1]
:db/ident :ocelot/number
:db/valueType :db.type/long
:db/cardinality :db.cardinality/one
:db/doc "An ocelot number"
:db/index true
:db.install/_attribute :db.part/db}
This ensures that the attribute is included in the AVET index.
Then the following gives you access to the "top ten", albeit using the low-level datoms call.
(take-last 10 (d/datoms (db conn) :avet :ocelot/number))
Obviously if you need to do any further filtering ("who are the top ten scorers in this club ?") then this approach won't work, but at that point you have a much smaller amount of data in your hand and shouldn't need to worry about the indexing.
I did look extensively at the aggregation functions available from Datalog and am having trouble getting my head around them - and am uncertain that e.g. max would use this index rather than a full scan of the data. Similarly the (index-range ...) function almost certainly does use this index but requires you to know the start and/or end values.
I'm trying to do an 'outer join' in Datomic via the REST API. From https://github.com/Datomic/day-of-datomic/blob/master/tutorial/social_news.clj I have taken the final example:
(defn maybe
"Returns the set of attr for e, or nil if e does not possess
any values for attr."
[db e attr]
(seq (map :a (d/datoms db :eavt e attr))))
;; find all users
(q '[:find ?e ?upvote
:where
[?e :user/email]
[(user/maybe $ ?e :user/upVotes) ?upvote]]
(db conn))
I inserted the maybe function into my database, and it can be queried thus:
[:find ?n ?v :in $ :where [?e ?a ?v] [?a :db/ident :db/fn] [?e :db/ident ?n]]
returns
:maybe #db/fn{:code "(seq (map :a (d/datoms db :eavt e attr)))", :params [db e attr], :requires [], :imports [], :lang :clojure}
However, I am unable to work out how to call the function in a query. I have an :data/user attribute on some transactions, which I want to get the value for where it exists. Here's the query I'm trying to run; I would like :maybe to be the database function defined above.
[:find ?attr ?v ?when ?who :where
[17592186045423 ?a ?v ?tx true]
[?a :db/ident ?attr]
[(:maybe $ ?tx :data/user) ?who]
[?tx :db/txInstant ?when]]
I'm pretty sure I'm missing something pretty obvious, but I've been stuck on this for a day now. Thanks for any help!
You need to use d/invoke. So your example would look like this:
[:find ?attr ?v ?when ?who :where
[17592186045423 ?a ?v ?tx true]
[?a :db/ident ?attr]
[(d/invoke $ :maybe ?tx :data/user) ?who]
[?tx :db/txInstant ?when]]
According to the Query Doc of Datomic, you can use any pure function in queries. You don't have to install them first. Functions you have to install are transaction functions.
Query functions don't need to be installed because they run in your application like all other functions. Datomic has nothing like a database server that executes queries. Queries are always executed in your application by the Peer Library.
The only type of functions you need to install are transaction functions because they run inside the Transactor. The Transactor is a single, special process that handles all the writes in Datomic.