New to NoSQL and a little confused with creating collections - database
Im a student just starting out on NoSQL and its just not clicking with me. im a little confused on a few points.
Any help would be greatly appreciated
1.Can documents belong to multiple collections?
2.Have I the correct syntax here for creating the Collection?
The pic is the collection er and a is just a snippet of the full er.
db.Animal.insert ( {
“animal_ID” : “XXXXXXX “,
“common_name” : “Red Squirrel”,
“IUCN” : “Least Concern (declining)”,
“photo” : “qs451xkx6qf4j”,
“extinct” : {
“when” : “null “,
“reason” : “null”
},
“invasive” : {
“threat_level” : “null”,
“threat” : “null”,
“how_to_help” : “null”
},
“native” : {
“endangerment” : “population declining“,
“how_to_help” : “providing a little extra food, planting some red squirrel-friendly shrubs and reporting any red or grey squirrel activity”
},
“Fact_sheet” : “{
“fact_id” : “ “,
“animal_id” : “ XXXXXXX “,
“order” : “ Rodentia “,
“family” : “Sciuridae “ ,
“species” : “Sciurus vulgaris “ ,
“size” : “body length 19 to 23 cm, tail length 15 to 20 cm “ ,
“weight” : “250 to 340 g “ ,
“lifespan” : “3 years , 7 to 10 in captivity “ ,
“extra” : “In Norse mythology, Ratatoskr is a red squirrel who runs up and down with messages in the world tree, Yggdrasil, and spreads gossip “ ,
“habitat” : { [
“name” : “woodland “,
“description” : “a low-density forest forming open habitats with plenty of sunlight and limited shade “
]
});
Can documents belong to multiple collections?
In MongoDB, no. In other databases, I don't know.
2.Have I the correct syntax here for creating the Collection?
To create a collection you would use https://docs.mongodb.com/manual/reference/method/db.createCollection/. This call also permits you to pass various collection options.
You are inserting a document. In MongoDB when a document is inserted, if the destination collection doesn't exist, it is created automatically by the server.
Related
ElasticSearch order based on type of hit
I started using ElasticSearch in my ReactJS project but I'm having some troubles with it. When I search, I'd like to have my results ordered based on this table Full-Hit Start-hit Sub-hit Fuzzy-hit category 1 3 5 10 trade name 2 4 6 11 official name 7 8 9 12 The definition (the way I see it, unless I'm wrong) are like this: Full-hit examples: Term "John" has a full-hit on "John doe" Term "John" doesn't have a full-hit on "JohnDoe" Start-hit examples: Term "John" has a start-hit on "Johndoe" Term "Doe" doesn't have a start-hit on "JohnDoe" sub-hit examples: Term "baker" has a sub-hit on "breadbakeries" Term "baker" doesn't have a sub-hit on "de backer" fuzzy-hit From my understanding fuzzy-hit is when the queried word has 1 mistake or 1 letter is missing examples: Term "bakker" has a fuzzy-hit on "baker" Term "bakker" doesn't have a fuzzy-hit on "bakers" I found out that we can boost fields like this fields = [ `category^3`, `name^2`, `official name^1`, ]; But that is not based on the full-, start-, sub-, or fuzzy-hit Is this doable in ReactJS with Elasticsearch?
I need to understand your problem. In a nutshell 1."If a full-hit is found in the category field, then we should boost it by 1". If a full-hit is found in the official_name field we should boost by 7.. and so on for all the 12 possibilities? If this is what you want, you are going to need 12 seperate queries, all covered under one giant bool -> should clause. I won't write out the query for you, but I will give you some pointers, on how to structure the 4 subtypes of the queries. Full Hit { "term" : {"field" : "category/tradE_name/official_name", "value" : "the_search_term"} } Start-hit { "match_phrase_prefix" : {"category/trade_name/official_name" : "the search term"} } Sub-hit { "regexp" : { "category/official/trade" : {"value" : "*term*"} } } Fuzzy { "fuzzy" : { "category/trade/official" : {"value" : "term"} } } You will need one giant bool { "query" : { "bool" : { "should" : [ // category field queries, 4 total clauses. { } // official field queries, 4 clauses, to each clauses assign the boost as per your table. that's it. ] } } } To each clause, assign a boost as per your table. That;s it. HTH.
Mongodb TTL Index not expiring documents from collection
I have TTL index in collection fct_in_ussd as following db.fct_in_ussd.createIndex( {"xdr_date":1}, { "background": true, "expireAfterSeconds": 259200} ) { "v" : 2, "key" : { "xdr_date" : 1 }, "name" : "xdr_date_1", "ns" : "appdb.fct_in_ussd", "background" : true, "expireAfterSeconds" : 259200 } with expiry of 3 days. Sample document in collection is as following { "_id" : ObjectId("5f4808c9b32ewa2f8escb16b"), "edr_seq_num" : "2043019_10405", "served_imsi" : "", "ussd_action_code" : "1", "event_start_time" : ISODate("2020-08-27T19:06:51Z"), "event_start_time_slot_key" : ISODate("2020-08-27T18:30:00Z"), "basic_service_key" : "TopSim", "rate_event_type" : "", "event_type_key" : "22", "event_dir_key" : "-99", "srv_type_key" : "2", "population_time" : ISODate("2020-08-27T19:26:00Z"), "xdr_date" : ISODate("2020-08-27T19:06:51Z"), "event_date" : "20200827" } Problem Statement :- Documents are not getting removed from collection. Collection still contains 15 days old documents. MongoDB server version: 4.2.3 Block compression strategy is zstd storage.wiredTiger.collectionConfig.blockCompressor: zstd Column xdr_date is also part of another compound index. Observations as on Sep 24 I have 5 collections with TTL index. It turns out that data is getting removed from one of the collection and rest of the collections remains unaffected. Daily insertion rate is ~500M records (including 5 collections). This observation left me confused. TTL expiration thread run on single. Is it too much data for TTL to expire ?
JSONata sort / order by of an array
I want to order an array. The JSONata expression below has an incoming array as follows. [{"id":"Air-1a", "Controller":"ESP62", "Cntr-TaskNo":10, "Cntr-GPIO":13, "name":"Air", "valueName":"Humidity", "Sensor":"DHT22", (and many other key pairs)}, {next object}, ...] I then transform the array with the following JSONata expression: payload.( { "Controller" : $.Controller, "Cntr-TaskNo": $.CntrDef.TaskNo, "Cntr-GPIO" : $.CntrDef.GPIO, "name" : $.name, "valueName" : $.valueName, "Sensor" : $.Sensor, "id" : $.id } ) But now I want to - in the same JSONata expression, sort on firstly the Controller, and then the GPIO. To tried with the Controller only first. I tried: payload.( { $sort("Controller",function($l, $r){$l.Controller > $r.Controller}) : $.Controller , "Cntr-TaskNo": $.CntrDef.TaskNo, "Cntr-GPIO" : $.CntrDef.GPIO, "name" : $.name, "valueName" : $.valueName, "Sensor" : $.Sensor, "id" : $.id } ) As well as trying to add the sort function at the end with the ~> chaining command. I also tried the order-by operator. Could anyone point me in the right direction? //---------- The new flow with the changed 'ESP62' to '-' that does not work: [{"id":"874b0c77.f87418","type":"inject","z":"6f27a311.d135bc","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":200,"y":180,"wires":[["8c196590.c20638"]]},{"id":"8c196590.c20638","type":"change","z":"6f27a311.d135bc","name":"Dataset","rules":[{"t":"set","p":"payload","pt":"msg","to":"[{\"id\":\"Air-1a\",\"Controller\":\"ESP62\",\"CntrTaskNo\":10,\"CntrGPIO\":13,\"name\":\"Air\",\"valueName\":\"Humidity\",\"Sensor\":\"DHT22\",\"aaa\":\"111\",\"bbb\":\"222\",\"ccc\":\"333\"},{\"id\":\"Air-2a\",\"Controller\":\"ESP72\",\"CntrTaskNo\":11,\"CntrGPIO\":14,\"name\":\"Air\",\"valueName\":\"Humidity\",\"Sensor\":\"DHT22\",\"aaa\":\"444\",\"bbb\":\"555\",\"ccc\":\"666\"},{\"id\":\"Air-1a\",\"Controller\":\"ESP62\",\"CntrTaskNo\":2,\"CntrGPIO\":9,\"name\":\"Air\",\"valueName\":\"Humidity\",\"Sensor\":\"DHT22\",\"aaa\":\"777\",\"bbb\":\"888\",\"ccc\":\"999\"},{\"id\":\"Air-1a\",\"Controller\":\"-\",\"CntrTaskNo\":10,\"CntrGPIO\":12,\"name\":\"Air\",\"valueName\":\"Humidity\",\"Sensor\":\"DHT22\",\"aaa\":\"777\",\"bbb\":\"888\",\"ccc\":\"999\"}]","tot":"json"}],"action":"","property":"","from":"","to":"","reg":false,"x":360,"y":180,"wires":[["13981162.14e28f"]]},{"id":"c8a256a5.a170c8","type":"debug","z":"6f27a311.d135bc","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":690,"y":180,"wires":[]},{"id":"13981162.14e28f","type":"change","z":"6f27a311.d135bc","name":"Jsonata $sort","rules":[{"t":"set","p":"payload","pt":"msg","to":"($sort(payload,function($l , $r){$l.Controller > $r.Controller}) ; \t$sort(payload,function($l , $r){$l.CntrGPIO > $r.CntrGPIO}))","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":520,"y":180,"wires":[["c8a256a5.a170c8"]]}]
I suggest first sorting the dataset and afterward transform the already sorted array of objects. The transformation is trivial and you want to know how to sort, so I show below one possible solution. It uses an expression with two concatenated $sort functions. Edited after a better understanding of the requirement. I tested successfully a Node-RED flow using this expression in a change node: ($a := $sort(payload,function($l , $r){$l.Controller > $r.Controller}) ; $sort($a,function($l , $r){(($l.Controller = $r.Controller) and ($l.CntrGPIO > $r.CntrGPIO))})) Flow (contain dataset set hardcoded): [{"id":"a7814b7e.3adeb8","type":"tab","label":"Flow 4","disabled":false,"info":""},{"id":"8bf10833.c71748","type":"inject","z":"a7814b7e.3adeb8","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":140,"y":140,"wires":[["9e365564.edca08"]]},{"id":"9e365564.edca08","type":"change","z":"a7814b7e.3adeb8","name":"Dataset","rules":[{"t":"set","p":"payload","pt":"msg","to":"[{\"id\":\"Air-1a\",\"Controller\":\"ESP62\",\"CntrTaskNo\":10,\"CntrGPIO\":13,\"name\":\"Air\",\"valueName\":\"Humidity\",\"Sensor\":\"DHT22\",\"aaa\":\"111\",\"bbb\":\"222\",\"ccc\":\"333\"},{\"id\":\"Air-2a\",\"Controller\":\"ESP72\",\"CntrTaskNo\":11,\"CntrGPIO\":14,\"name\":\"Air\",\"valueName\":\"Humidity\",\"Sensor\":\"DHT22\",\"aaa\":\"444\",\"bbb\":\"555\",\"ccc\":\"666\"},{\"id\":\"Air-1a\",\"Controller\":\"ESP62\",\"CntrTaskNo\":2,\"CntrGPIO\":9,\"name\":\"Air\",\"valueName\":\"Humidity\",\"Sensor\":\"DHT22\",\"aaa\":\"777\",\"bbb\":\"888\",\"ccc\":\"999\"},{\"id\":\"Air-1a\",\"Controller\":\"-\",\"CntrTaskNo\":10,\"CntrGPIO\":12,\"name\":\"Air\",\"valueName\":\"Humidity\",\"Sensor\":\"DHT22\",\"aaa\":\"777\",\"bbb\":\"888\",\"ccc\":\"999\"}]","tot":"json"}],"action":"","property":"","from":"","to":"","reg":false,"x":300,"y":140,"wires":[["762f6421.074fec"]]},{"id":"f827bddb.c9acd","type":"debug","z":"a7814b7e.3adeb8","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":630,"y":140,"wires":[]},{"id":"762f6421.074fec","type":"change","z":"a7814b7e.3adeb8","name":"Jsonata $sort","rules":[{"t":"set","p":"payload","pt":"msg","to":"($a := $sort(payload,function($l , $r){$l.Controller > $r.Controller}) ; $sort($a,function($l , $r){(($l.Controller = $r.Controller) and ($l.CntrGPIO > $r.CntrGPIO))}))","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":460,"y":140,"wires":[["f827bddb.c9acd"]]}] also tested in Jsonata exerciser: http://try.jsonata.org/S1IlT3y-E
You can sort the array using the following expression: payload^(Controller, CntrDef.GPIO) The order-by operator ^ will sort the array, first by increasing value of Controller, then by increasing value of CntrGPIO. You can then transform each object within that array payload^(Controller, CntrDef.GPIO).{ "Controller" : Controller, "Cntr-TaskNo": CntrDef.TaskNo, "Cntr-GPIO" : CntrDef.GPIO, "name" : name, "valueName" : valueName, "Sensor" : Sensor, "id" : id }
How to give a score if the element is present in the array in elastic search?
"functions": [{ "script_score": { "script": "(doc['content.acknowledgement'].value) ? ((doc['content.acknowledgementUsers'].values.contains(24)) ? 0 : 1 ) : 0" } } ], "score_mode": "sum" This is the syntax i have so far. I'm trying to give a score based on these conditions. If the users is in acknowledgementUsers give a score of 0 else give a score of 1. For some reason everytime the script is giving me score of 0. I'm using elastic search 5.2 Can some one please help me with this?
Viewing unique fields
Is there a juttle program I can run to view all unique fields within a given query? I'm trying comb through a list of events of the same type that have a ton of different fields. I know I could just use the #table sink and scroll right but, I'd like to view unique fields in a list if possible.
Hacky but works: events -from :5 minutes ago: -to :now: | head 1 | #logger -display.style 'pretty' You get: { "bytes" : 7745, "status" : "200", "user_agent" : "Mozilla/5.0 (iPhone; CPU iPhone OS 511 like Mac OS X) AppleWebKit/534.46 (KHTML like Gecko) Version/5.1 Mobile/9B206 Safari/7534.48.3", "version" : "1.1", "ctry" : "US", "ident" : "-", "message" : "194.97.17.121 - - [2014-02-25T09:00:00-08:00] \"GET /black\" 200 7745 \"http://google.co.in/\" \"Mozilla/5.0 (iPhone; CPU iPhone OS 511 like Mac OS X) AppleWebKit/534.46 (KHTML like Gecko) Version/5.1 Mobile/9B206 Safari/7534.48.3\"", "auth" : "-", "verb" : "GET", "url" : "/black", "source_host" : "www.jut.io", "referer" : "http://google.co.in/", "space" : "default", "type" : "event", "time" : "2014-12-11T23:46:21.905Z", "mock_type" : "demo", "event_type" : "web" }
You can use the split proc in combination with reduce by to get this list. emit -limit 1 |( put field1 = 1, field2 = 2; put field2 = 2, field3 = 3; )| split // break each point into one point for each field, assigning each field name into the point's name field | reduce by name // get unique list of name field values | sort name | #logger {"name":"field3"} {"name":"field2"} {"name":"field1"} ==============================================================