In the official docs for Datomic (http://docs.datomic.com/schema.html) under the heading 'Creating new partitions' it says that a new partition (communities) can be created like this:
{:db/id #db/id[:db.part/db]
:db/ident :communities}
Here the ':communities' is not written as 'db.part/communities'
I can not install a new partition this way. For me it has to be with the leading 'db.part/'. Is the documentation wrong, or am I not seeing the bigger picture?
If you read further in the documentation, you'll see that you're missing another datom required for that transaction (labeled with "Here is the complete transaction..."). That datom is (with user assigned tempid as -1 optional):
[:db/add :db.part/db :db.install/partition #db/id[:db.part/db -1]]
Anything transacted with a tempid that resolves to the system partition (:db.part/db) must also include a datom marking the installation, as with :db.install/partition and :db.install/attribute (the reverse ref version for attribute included in the map is more common).
Transacting the full example from the docs works fine:
(def tx [{:db/id #db/id[:db.part/db -1]
:db/ident :communities}
[:db/add :db.part/db :db.install/partition #db/id[:db.part/db -1]]])
#(d/transact conn tx)
;; returns successful tx map
Related
Datomic newbie here. Playing with various valueTypes and can't get tuple data type to work.
Here's how defined the schema:
(d/transact conn {
:tx-data [{
:db/ident :df/Errors
:db/valueType :db.type/tuple
:db/tupleType :db.type/string
:db/cardinality :db.cardinality/many}]})
This worked. However, I can't figure out how to enter sample data. I tried
(d/transact conn {:tx-data [{
:df/Errors ["Error-code" "sample error message"]}]})
But it gives an error:
Invalid tuple value
As per docs, the tuple value is a vector with 2 to 8 elements. So, not sure what I'm doing wrong. Please help.
Since the cardinality is many, the sample data code should look like this:
(d/transact conn {:tx-data [{:df/Errors [["Error-code" "sample error message"]]}]})
To elaborate, most relational databases have the idea of a database constraint. Here is the Postgres documentation on constraints. What tools does Datomic offer to constrain data or maintain some set of invariants on the data stored?
EDIT 2019-06-28: Since 0.9.5927 (Datomic On-Prem) / 480-8770 (Datomic Cloud), Datomic supports finer write-time validation via Attribute Predicates, Entity Specs and Entity Predicates. This makes most of the initial answer invalid or irrelevant.
In particular, observe that Entity Predicates accept a database value as a parameter, so they can actually enforce invariants that span several Entities.
By default, Datomic enforces only a very limited set of constraints regarding what data may be written, including mostly:
uniqueness constraints: see Identity and Uniqueness
type constraints, e.g you may not write a number to an attribute that is :db.type/string
entity resolution: operations like [:db/add [:my.app/id "fjdsklfjsl"] :my.app/content 42] will fail if the [:my.app/id "fjdsklfjsl"] lookup-ref does not resolve to an existing entity
conflicts, e.g Datomic won't let you :db/add 2 different values for the same entity-attribute pair if the attribute cardinality is one.
(I may be forgetting some, if so please comment.)
In particular at the time of writing, there is no built-in way to add custom validation or 'foreign-key' constraint to a given attribute.
However, combining Transaction Functions and speculative writes (a.k.a db.with()) gives you a powerful way of enforcing arbitrary invariants. For instance, you can wrap a transaction in a transaction function that applies the transaction speculatively using db.with(), then searches the speculative result for invariant violations, throwing an Exception if it finds some. You can even make this transaction function very generic by implementing the 'search invariant violations' part in Datalog.
Here's an example of what the API may look like:
[:myapp.fns/checking-invariants
;; a description of the invariant
{:query
[:find ?message ?user-id
:in $db-before $db-after ?tx-data ?tempids ?user-id
:where
[$db-before ?user :myapp.user/id ?user-id]
[$db-before ?user :myapp.user/email ?email-before]
[$db-after ?user :myapp.user/email ?email-after]
[(not= ?email-before ?email-after)]
[(ground "A user may not change her email") ?message]]
:inputs ["user-id-12342141"]}
;; the wrapped transaction
[[:db/add 125315815291 :myapp.user/email "hello.world#yopmail.com"]
[:db/add 125315815291 :myapp.user/name "Foo Bar"]]]
Here's an (untested) implementation of :myapp.fns/checking-invariants:
{:db/ident :myapp.fns/checking-invariants,
:db/fn #db/fn{:lang :clojure,
:imports [],
:requires [[datomic.api :as d]],
:params [db invariant-q tx],
:code
(let [{:keys [query inputs]} invariants-q
{:keys [db-before db-after tx-data tempids]}
(d/with db tx)]
(when-some [violations (apply d/q query
db-before db-after tx-data tempids
inputs)]
(throw (ex-info
"Transaction would violate invariants."
{:tx tx
:violations violations
:t (d/basis-t db-before)})))
tx)}}
Limitations:
you can only protect externally: the client has to opt in to using this invariant-checking transaction function.
be careful about performance - abusing this approach may put too much load on the Transactor. In cases where it is safe to do so, you may prefer to perform validation on the Peer using db.invoke()
make sure your transaction is deterministic, as it will be run twice (more precisely, make sure that whether your transaction violates the invariant is deterministic)
One way is to use transaction functions that modify data and do constraint validation during modification: http://docs.datomic.com/database-functions.html#uses-for-transaction-functions
Suppose I have one million article entities in my backend with an inst attribute called date, or one million player entities with an int attribute called points. What's a good way to select the 10 latest articles or top-scoring players?
Do I need to fetch the whole millions to the peer and then sort and drop from them?
Until getting hold of the reverse index becomes a Datomic feature, you could manually define one.
e.g. for a :db.type/instant, create an additional attribute of type :db.type/long which you would fill with
(- (Long/MAX_VALUE) (.getTime date))
and the latest 10 articles could be fetched with
(take 10 (d/index-range db reverse-attr nil nil))
Yes, you would need to fetch all the data, since there's no index that would help you out here.
I would have created my own "index" and normalized this data. You can have a separate set of N entities where you keep as many as you'd like. You could start with 10, or consider storing 100 to trade some (possibly negligible) speed for more flexibility. This index can be stored on a separate "singleton" entity that you add as part of your schema.
;; The attribute that stores the index
{:db/id #db/id[:db.part/db]
:db/ident :indexed-articles
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/many
:db.install/_attribute :db.part/db}
;; The named index entity.
{:db/id #db/id[:db.part/db]
:db/ident :articles-index}
You can have a database function that does this. Every time you insert a new entity that you want to "index", call this function.
[[:db/add tempid :article/title "Foo]
[:db/add tempid :article/date ....]
[:index-article tempid 10]]
The implementation of index-article could look like this:
{:db/id #db/id[:db.part/user]
:db/ident :index-article
:db/fn #db/fn {:lang "clojure"
:params [db article-id idx-size]
:code (concat
(map
(fn [article]
[:db/retract
(d/entid db :articles-index)
:indexed-articles
(:db/id article)])
(->> (datomic.api/entity db :articles-index)
(sort-by (fn [] ... implement me ... ))
(drop (dec idx-size))))
[[:db/add (d/entid db :articles-index) :indexed-articles article-id]])}}
Disclaimer: I haven't actually tested this function, so it probably contains errors :) The general idea is that we remove any "overflow" entities from the set, and add the new one. When idx-size is 10, we want to ensure that only 9 items are in the set, and we add our new item to it.
Now you have an entity you can lookup from index, :articles-index, and the 10 most recent articles can be looked up from the index (all refs are indexed), without causing a full database read.
;; "indexed" set of articles.
(d/entity db :articles-index)
I've been looking into this and think I have a slightly more elegant answer.
Declare your attribute as indexed with :db/index true
{:db/id #db/id[:db.part/db -1]
:db/ident :ocelot/number
:db/valueType :db.type/long
:db/cardinality :db.cardinality/one
:db/doc "An ocelot number"
:db/index true
:db.install/_attribute :db.part/db}
This ensures that the attribute is included in the AVET index.
Then the following gives you access to the "top ten", albeit using the low-level datoms call.
(take-last 10 (d/datoms (db conn) :avet :ocelot/number))
Obviously if you need to do any further filtering ("who are the top ten scorers in this club ?") then this approach won't work, but at that point you have a much smaller amount of data in your hand and shouldn't need to worry about the indexing.
I did look extensively at the aggregation functions available from Datalog and am having trouble getting my head around them - and am uncertain that e.g. max would use this index rather than a full scan of the data. Similarly the (index-range ...) function almost certainly does use this index but requires you to know the start and/or end values.
Using Clojure's Datomic APi, I have an architecture where I'd like to pass in a transact function, to be executed. However, trying to call the passed in transact function doesn't work. The repl recognizes it as a Symbol. And it evaluates, but no data is committed, and no future is returned, meaning, there's no returned transaction ID.
However, directly calling (datomic.api/transact conn [adatom]), works as expected. How can I make the abouve scenario work?
(defn some-fn[conn mapped-fn adatom]
(datomic.api/transact conn [adatom]) ;; works
#_(mapped-fn conn [adatom]) ;; does not work - evaluates, but no data committed, no future returned, meaning, no returned transaction ID
)
Thanks
It's not very clear what you are trying to do. For example:
why not call d/transact inside some-fn?
is mapped-fn a function that will be applied to many facts (and in this case it is d/transact)?
what is an overall intent?
To just blindly follow your example, it does work and returns a "future":
user=> (use '[datomic.api :only (db) :as d])
nil
user=> (d/create-database "datomic:mem://unsure-db")
false
user=> (def c (d/connect "datomic:mem://unsure-db"))
#'user/c
user=> (defn f [conn mapped-fn fact] (mapped-fn conn fact))
#'user/f
user=> (f c d/transact [])
#<promise$settable_future$reify__4526#2da07336: {:db-before datomic.db.Db#8835fddc, :db-after datomic.db.Db#6e2a2e78, :tx-data [#Datum{:e 13194139534313 :a 50 :v #inst "2013-09-03T15:23:34.977-00:00" :tx 13194139534313 :added true}], :tempids {}}>
Make sure you have a valid connection (e.g. you are connected [to the right database]), and the database is there.
What is a query to list all partitions of a Datomic database?
This should return
[[:db.part/db] [:db.part/tx] [:db.part/user] .... ]
where .... is all of the user defined partitions.
You should be able to get a list of all partitions in the database by searching for all entities associated with the :db.part/db entity via the :db.install/partition attribute:
(ns myns
(:require [datomic.api :as d]))
(defn get-partitions [db]
(d/q '[:find ?ident :where [:db.part/db :db.install/partition ?p]
[?p :db/ident ?ident]]
db))
Note
The current version of Datomic (build 0.8.3524) has a shortcoming such that :db.part/tx and :db.part/user (two of the three built-in partitions) are treated specially and aren't actually associated with :db.part/db via :db.install/partition, so the result of the above query function won't include the two.
This problem is going to be addressed in one of the future builds of Datomic. In the meantime, you should take care of including :db.part/tx and :db.part/user in the result set yourself.
1st method - using query
=> (q '[:find ?i :where
[:db.part/db :db.install/partition ?p] [?p :db/ident ?i]]
(db conn))
2nd method - from db object
(filter #(instance? datomic.db.Partition %) (:elements (db conn)))
The second method returns sequence of datomic.db.Partition objects which may be useful if we want to get additional info about the partition.
Both methods have known bug/inconsistency: they don't return :db.part/tx and :db.part/user built-in partitions.