Updating value with cardinality many - datomic

I have a schema like this:
[{:db/id #db/id[:db.part/db]
:db/ident :person/name
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "A person's name"
:db.install/_attribute :db.part/db}
{:db/id #db/id[:db.part/db]
:db/ident :person/roles
:db/valueType :db.type/keyword
:db/cardinality :db.cardinality/many
:db/doc "A person's role"
:db.install/_attribute :db.part/db}]
And a code like this:
;; insert new person
(def new-id (-> (d/transact conn [{:db/id (d/tempid :db.part/user)
:person/name "foo"
:person/roles #{:admin}}])
(:tempids)
(vals)
(first)))
(defn get-roles
[db eid]
(d/q '[:find ?roles .
:in $ ?eid
:where [?eid :user/roles ?roles]]))
(get-roles (d/db conn) new-id) ;; => [:admin]
;; update a person
(d/transact conn [{:db/id new-id
:person/roles #{:client}}])
(get-roles (d/db conn) new-id) ;; => [:admin :client]
It seems the default behaviour on it is, it will just assoc the new value.
How can I get this result, after doing the updating transaction:
(get-roles (d/db conn) new-id) ;; => [:client]

if what you want is to "reset" the list of roles to a new value (an 'absolute' operation 'in contrast to the 'relative' operations of just adding or removing roles), you'll have to use a transaction function to perform a diff and retract the values that need be.
Here's a basic generic implementation:
{:db/id (d/tempid :db.part/user),
:db/ident :my.fns/reset-to-many,
:db/fn
(d/function
{:lang :clojure,
:requires '[[datomic.api :as d]],
:params '[db e attr new-vals],
:code
'(let [ent (or (d/entity db e)
(throw (ex-info "Entity not found"
{:e e :t (d/basis-t db)})))
entid (:db/id ent)
old-vals (get ent attr)]
(into
[{:db/id (:db/id ent)
;; adding the new values
attr new-vals}]
;; retracting the old values
(comp
(remove (set new-vals))
(map (fn [v]
[:db/retract entid attr v])))
old-vals)
)})}
;; Usage
(d/transact conn [[:my.fns/reset-to-many new-id :person/roles #{:client}]])

Here is a solution once suggested from Robert Stuttaford
(defn many-delta-tx
"Produces the transaction necessary to have
`new` be the value for `entity-id` at `attr`
`new` is expected to be a set."
[db entity-id attr new]
(let [current (into #{} (map :v)
(d/datoms db :eavt
entity-id
(d/entid db attr)))]
(concat (for [id (clojure.set/difference new current)]
[:db/add entity-id attr id])
(for [id (clojure.set/difference current new)]
[:db/retract entity-id attr id]))))
For ease of testing, I would like to slightly modify part of the example of original question.
Change of the schema. I add db/unique
[{:db/id #db/id[:db.part/db]
:db/ident :person/name
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/unique :db.unique/identity
:db/doc "A person's name"
:db.install/_attribute :db.part/db}
{:db/id #db/id[:db.part/db]
:db/ident :person/roles
:db/valueType :db.type/keyword
:db/cardinality :db.cardinality/many
:db/doc "A person's role"
:db.install/_attribute :db.part/db}]
get-roles
(defn get-roles
[db eid]
(d/q '[:find [?roles ...]
:in $ ?eid
:where [?eid :person/roles ?roles]] db eid))
My testing example
;; (many-delta-tx (d/db conn) [:person/name "foo"] :person/roles #{:sales :client})
;; => ([:db/add [:person/name "foo"] :person/roles :sales] [:db/retract [:person/name "foo"] :person/roles :admin]))

Related

datomic / datalevin performance and db size - is this schema wrong?

I am trying to enter what's essentially tabular data in a datalevin dB. I'm wondering whether I'm doing something deeply wrong, maybe with the schema? The database grows extremely fast on disk (100x bigger than the equivalent CSV) and query time is also quite slow, and increasing with the size of the dB.
Each transaction below ("fake-portfolio") is around 900k datoms, it takes several seconds to write, and after about 10 transactions on my machine it just crashes.
The typical query I'd like to do is something akin to
(d/q '[:find ?date ?portfolio (sum ?nav)
:in $
:keys date portfolio nav
:with ?position/bond
:where
[?e :position/portfolio ?portfolio]
[?e :position/date ?date]
[?e :position/bond-id-ref ?bond]
[?e :position/weight ?nav]
]
(d/db test-connection)
(def test-schema {
:position/bond-id-ref {:db/valueType :db.type/long :db/cardinality :db.cardinality/one} ;will link to bond above
:position/date {:db/valueType :db.type/long :db/cardinality :db.cardinality/one}
:position/portfolio {:db/valueType :db.type/string :db/cardinality :db.cardinality/one}
:position/weight {:db/valueType :db.type/double :db/cardinality :db.cardinality/one}
:position/bm-weight {:db/valueType :db.type/double :db/cardinality :db.cardinality/one}
:position/weight-delta {:db/valueType :db.type/double :db/cardinality :db.cardinality/one}
:position/contrib-yield {:db/valueType :db.type/double :db/cardinality :db.cardinality/one}
:position/contrib-zspread {:db/valueType :db.type/double :db/cardinality :db.cardinality/one}
:position/contrib-gspread {:db/valueType :db.type/double :db/cardinality :db.cardinality/one}
:position/contrib-mdur {:db/valueType :db.type/double :db/cardinality :db.cardinality/one}
:position/mdur-delta {:db/valueType :db.type/double :db/cardinality :db.cardinality/one}
:position/bm-contrib-eir-duration {:db/valueType :db.type/double :db/cardinality :db.cardinality/one}
:position/bm-contrib-yield {:db/valueType :db.type/double :db/cardinality :db.cardinality/one}
:position/bm-contrib-govt-spread {:db/valueType :db.type/double :db/cardinality :db.cardinality/one}
:position/contrib-beta-1y-daily {:db/valueType :db.type/double :db/cardinality :db.cardinality/one}
:position/duration-times-spread-weight {:db/valueType :db.type/double :db/cardinality :db.cardinality/one}
:position/quant-value-4d {:db/valueType :db.type/double :db/cardinality :db.cardinality/one}
:position/quant-value-2d {:db/valueType :db.type/double :db/cardinality :db.cardinality/one}})
(def test-database-path "/tmp/datalevin/testpositions")
(defn fake-portfolio [d]
(into []
(for [p (range 0 25)
b (range 0 2000)]
{
:position/bond-id-ref b
:position/date d
:position/portfolio (str "portfolio-" p)
:position/weight 0.5
:position/bm-weight 0.2
:position/weight-delta 0.3
:position/contrib-yield 0.8
:position/contrib-zspread 3.4
:position/contrib-gspread 6.7
:position/contrib-mdur 8.756
:position/mdur-delta 45.6568
:position/bm-contrib-eir-duration 0.3
:position/bm-contrib-yield 0.0
:position/bm-contrib-govt-spread 78.5
:position/contrib-beta-1y-daily 45.5
:position/duration-times-spread-weight 0.3
:position/quant-value-4d 0.2
:position/quant-value-2d 0.7}
)
)
)
(def test-connection (d/get-conn test-database-path test-schema))
(defn fill-test-db [n]
(doseq [i (range n)]
(d/transact! test-connection (fake-portfolio i))))
Thank you,

Pull expression in query with limit/default?

How can I use limit/default in at pull expression in a query? Given a cardinality-many attribute, how can I control how many of its values is returned (default is max 1000 values!).
(Found it hard to figure out the correct syntax from the documentation/examples)
Limit (for cardinality-many attributes)
Return max 2 values of cardinality-many attribute :ns/ints:
[:find (pull ?a [(limit :ns/ints 2)])
:where [?a :ns/str ?b]]
Return all values of cardinality-many attribute :ns/ints:
[:find (pull ?a [(limit :ns/ints nil)])
:where [?a :ns/str ?b]]
Default
Return default value 2000 if attribute :ns/ints has no value:
[:find (pull ?a [(default :ns/ints 2000)])
:where [?a :ns/str ?b]]
Return default values 2000 and 2001 if cardinality-many attribute :ns/ints has no values:
[:find (pull ?a [(default :ns/ints [2000 2001])])
:where [?a :ns/str ?b]]

Clojure nested doseq loop

I'm new to Clojure and I have a question regarding nested doseq loops.
I would like to iterate through a sequence and get a subsequence, and then get some keys to apply a function over all the sequence elements.
The given sequence has an structure more or less like this, but with hundreds of books, shelves and many libraries:
([:state/libraries {6 #:library {:name "MUNICIPAL LIBRARY OF X" :id 6
:shelves {3 #:shelf {:name "GREEN SHELF" :id 3 :books
{45 #:book {:id 45 :name "NECRONOMICON" :pages {...},
{89 #:book {:id 89 :name "HOLY BIBLE" :pages {...}}}}}}}}])
Here is my code:
(defn my-function [] (let [conn (d/connect (-> my-system :config :datomic-uri))]
(doseq [library-seq (read-string (slurp "given-sequence.edn"))]
(doseq [shelves-seq (val library-seq)]
(library/create-shelf conn {:id (:shelf/id (val shelves-seq))
:name (:shelf/name (val shelves-seq))})
(doseq [books-seq (:shelf/books (val shelves-seq))]
(library/create-book conn (:shelf/id (val shelves-seq)) {:id (:book/id (val books-seq))
:name (:book/name (val books-seq))})
)))))
The thing is that I want to get rid of that nested doseq mess but I don't know what would be the best approach, since in each iteration keys change. Using recur? reduce? Maybe I am thinking about this completely the wrong way?
Like Carcigenicate says in the comments, presuming that the library/... functions are only side effecting, you can just write this in a single doseq.
(defn my-function []
(let [conn (d/connect (-> my-system :config :datomic-uri))]
(doseq [library-seq (read-string (slurp "given-sequence.edn"))
shelves-seq (val library-seq)
:let [_ (library/create-shelf conn
{:id (:shelf/id (val shelves-seq))
:name (:shelf/name (val shelves-seq))})]
books-seq (:shelf/books (val shelves-seq))]
(library/create-book conn
(:shelf/id (val shelves-seq))
{:id (:book/id (val books-seq))
:name (:book/name (val books-seq))}))))
I would separate "connecting to the db" from "slurping a file" from "writing to the db" though. Together with some destructuring I'd end up with something more like:
(defn write-to-the-db [conn given-sequence]
(doseq [library-seq given-sequence
shelves-seq (val library-seq)
:let [{shelf-id :shelf/id,
shelf-name :shelf/name
books :shelf/books} (val shelves-seq)
_ (library/create-shelf conn {:id shelf-id, :name shelf-name})]
{book-id :book/id, book-name :book/name} books]
(library/create-book conn shelf-id {:id book-id, :name book-name})))

using core.async / ajax data in om component

I am currently experimenting with om and try to load external data for displaying it in a component.
My detail component:
(defn detail-component [app owner opts]
(reify
om/IInitState
(init-state [_]
(om/transact! app [:data] (fn [] "Test")))
om/IWillMount
(will-mount [_]
(go (let [foo (<! (fetch-something 1))]
(om/update! app #(assoc % :data foo))
)))
om/IRender
(render [_]
(dom/div nil
(dom/h1 nil "Foo")
(om/build another-component app)
)
))
)
(fetch-something is retrieving data from an API).
(defn another-component [{:keys [data]}]
(om/component
(.log js/console data)
(dom/h2 nil "Another component goes here")
(dom/h2 nil (data :description))
)
)
So to summarize, detail-component fetches data before mounting, attaches it to app and builds another-component. another-component then takes the description out of that data and displays it.
However when executing, I am getting Uncaught TypeError: Cannot read property 'call' of null at the point where I am trying to access the description. This indicates to me that at the point when another-component is getting built, the data is not there yet and it fails.
How can I tell om to build the app when data is available? Or do I have to build in some nil? checks?
Working example using state:
project.clj
(defproject asajax "0.0.1-SNAPSHOT"
:description "FIXME: write description"
:url "http://example.com/FIXME"
:license {:name "Eclipse Public License - v 1.0"
:url "http://www.eclipse.org/legal/epl-v10.html"
:distribution :repo}
:min-lein-version "2.3.4"
:source-paths ["src/clj" "src/cljs"]
:dependencies [[org.clojure/clojure "1.6.0"]
[org.clojure/clojurescript "0.0-2371"]
[org.clojure/core.async "0.1.267.0-0d7780-alpha"]
[om "0.7.3"]
[com.facebook/react "0.11.2"]]
:plugins [[lein-cljsbuild "1.0.4-SNAPSHOT"]]
:hooks [leiningen.cljsbuild]
:cljsbuild
{:builds {:asajax
{:source-paths ["src/cljs"]
:compiler
{:output-to "dev-resources/public/js/asajax.js"
:optimizations :whitespace
:pretty-print true}}}})
core.cljs
(ns asajax.core
(:require [om.core :as om :include-macros true]
[om.dom :as dom :include-macros true]
[cljs.core.async :as async])
(:require-macros [cljs.core.async.macros :refer (go)]))
(enable-console-print!)
(def app-state (atom {}))
(defn another-component [{:keys [data]}]
(reify
om/IRenderState
(render-state [_ state]
(dom/div nil
;; (.log js/console data)
(dom/h2 nil "Another component goes here")
(dom/h2 nil (:description state))))))
(defn fetch-something [x]
(let [c (async/chan)]
(go
;; pretend a blocking call
;; wait for 2 sec
(<! (async/timeout 2000))
(>! c {:data "Roast peach & Parma ham salad"
:description "This is a lovely light starter with fantastic sweet, salty and creamy flavours"}))
c))
(defn detail-component [app owner opts]
(reify
om/IInitState
(init-state [_]
{:data "Test"})
om/IWillMount
(will-mount [_]
(go (let [foo (<! (fetch-something 1))]
;; (prn "GOT" foo)
(om/set-state! owner foo))))
om/IRenderState
(render-state [_ state]
(dom/div nil
(dom/h1 nil (:data state))
(om/build another-component app
{:state (om/get-state owner)})))))
(om/root
detail-component
app-state
{:target (. js/document (getElementById "app"))})
UPDATE2
Here is one using sablono and not using set-state!
Add to project.clj
[sablono "0.2.22"]
Full core.cljs
(ns asajax.core
(:require [om.core :as om :include-macros true]
[om.dom :as dom :include-macros true]
[cljs.core.async :as async]
[sablono.core :as html :refer-macros [html]])
(:require-macros [cljs.core.async.macros :refer (go)]))
(enable-console-print!)
(def app-state (atom {:data "Initial"
:description "Loading..."}))
(defn another-component [{:keys [description]}]
(om/component
(html
[:.description
[:h2 "Another component"]
[:h2 description]])))
(defn fetch-something [x]
(let [c (async/chan)]
(go
;; pretend a blocking call
;; wait for 2 sec
(<! (async/timeout 2000))
(>! c {:data "Roast peach & Parma ham salad"
:description "This is a lovely light starter with fantastic sweet, salty and creamy flavours"}))
c))
(defn detail-component [app owner opts]
(reify
om/IWillMount
(will-mount [_]
(go (let [foo (<! (fetch-something 1))]
;; (prn "GOT" foo)
(om/update! app foo))))
om/IRender
(render [_]
(html [:div
[:h1 (:data app)]
(om/build another-component app)
]))))
(om/root
detail-component
app-state
{:target (. js/document (getElementById "app"))})

Clojure how to insert a blob in database?

How to insert a blob in database using the clojure.contrib.sql?
I've tried the following reading from a file but I'm getting this exception:
SQLException:
Message: Invalid column type
SQLState: 99999
Error Code: 17004
java.lang.Exception: transaction rolled back: Invalid column type (repl-1:125)
(clojure.contrib.sql/with-connection
db
(clojure.contrib.sql/transaction
(clojure.contrib.sql/insert-values :test_blob [:blob_id :a_blob] [3 (FileInputStream. "c:/somefile.xls")]) ))
Thanks.
I was able to solve this by converting the FileInputStream into a ByteArray.
(clojure.contrib.sql/with-connection
db
(clojure.contrib.sql/transaction
(clojure.contrib.sql/insert-values :test_blob [:blob_id :a_blob] [3 (to-byte-array(FileInputStream. "c:/somefile.xls"))]) ))
In theory, you can use any of the clojure.contrib.sql/insert-* methods to insert a blob, passing the blob as either a byte array, java.sql.Blob or a java.io.InputStream object. In practice, it is driver-dependent.
For many JDBC implementations, all of the above work as expected, but if you're using sqlitejdbc 0.5.6 from Clojars, you'll find your blob coerced to a string via toString(). All the clojure.contrib.sql/insert-* commands are issued via clojure.contrib.sql/do-prepared, which calls setObject() on a java.sql.PreparedStatement. The sqlitejdbc implementation does not handle setObject() for any of the blob data types, but defaults to coercing them to a string. Here's a work-around that enables you to store blobs in SQLite:
(use '[clojure.contrib.io :only (input-stream to-byte-array)])
(require '[clojure.contrib.sql :as sql])
(defn my-do-prepared
"Executes an (optionally parameterized) SQL prepared statement on the
open database connection. Each param-group is a seq of values for all of
the parameters. This is a modified version of clojure.contrib.sql/do-prepared
with special handling of byte arrays."
[sql & param-groups]
(with-open [stmt (.prepareStatement (sql/connection) sql)]
(doseq [param-group param-groups]
(doseq [[index value] (map vector (iterate inc 1) param-group)]
(if (= (class value) (class (to-byte-array "")))
(.setBytes stmt index value)
(.setObject stmt index value)))
(.addBatch stmt))
(sql/transaction
(seq (.executeBatch stmt)))))
(defn my-load-blob [filename]
(let [blob (to-byte-array (input-stream filename))]
(sql/with-connection db
(my-do-prepared "insert into mytable (blob_column) values (?)" [blob]))))
A more recent reply to this thread with the code to read the data as well :
(ns com.my-items.polypheme.db.demo
(:require
[clojure.java.io :as io]
[clojure.java.jdbc :as sql]))
(def db {:dbtype "postgresql"
:dbname "my_db_name"
:host "my_server"
:user "user",
:password "user"})
(defn file->bytes [file]
(with-open [xin (io/input-stream file)
xout (java.io.ByteArrayOutputStream.)]
(io/copy xin xout)
(.toByteArray xout)))
(defn insert-image [db-config file]
(let [bytes (file->bytes file)]
(sql/with-db-transaction [conn db-config]
(sql/insert! conn :image {:name (.getName file) :data bytes}))))
(insert-image db (io/file "resources" "my_nice_picture.JPG"))
;;read data
(defn read-image [db-config id]
(-> (sql/get-by-id db-config :image id)
:data
(#(new java.io.ByteArrayInputStream %))))
I believe it's just the same way you'd insert any other value: use one of insert-records, insert-rows or insert-values. E.g.:
(insert-values :mytable [:id :blobcolumn] [42 blob])
More examples: http://github.com/richhickey/clojure-contrib/blob/master/src/test/clojure/clojure/contrib/test_sql.clj

Resources