I am trying to figure out the best way to store trip itinerary data into DynamoDB. Just for your info, my code is written in Python3 and I am using Boto3 to interact with DynamoDB.
After researching on this resource - https://schema.org/Trip, this is what I think would be the data classes of the objects.
from marshmallow_dataclass import dataclass
from typing import List, Optional
#dataclass(frozen=True)
class Itinerary:
id: str
startTime: int
endTime: int
dayTripId: str
placeName: str
placeCategory: str
estimatedCost: float
#dataclass(frozen=True)
class DayTrip:
id: str
day: str
parentTripId: str
date: Optional[str]
itinerary: List[Itinerary]
#dataclass(frozen=True)
class UserTrip:
tripId: str
userId: str
tripName: str
subTrip: List[DayTrip]
Essentially, the structure is as follows:
A person can have many UserTrips
A UserTrip can consist of one day or multiple day of DayTrip, e.g. Day 1, Day 2, Day 3
A DayTrip can have one or multiple places to visit (Itinerary)
An Itinerary is the lowest level that describes the place to visit
It wouldn't be good to store the UserTrip as is, with nested JSON structure consisting of DayTrip, then Itinerary, right? It would mean that the subTrip attribute of a particular UserTrip will be a huge chuck of JSON. So I think everyone here would agree this is a no, no. Is that correct?
Another alternative that I could think of was to store only the id of each entity. What I mean by this is, for example, a UserTrip will have its subTrip attribute containing a list of the DayTrip id. This means there will be another table to store DayTrip items and we can connect it to the corresponding UserTrip via the parentTripId attribute. And so on for the list of Itinerary.
Using this approach, I will have 3 x tables as follows:
user-trip-table to store UserTrip where subTrip will contain the list of DayTrip.ids
user-day-trip-table to store DayTrip where itinerary will contain the list of Itinerary.ids. The parentTripId will enable the mapping back to the original UserTrip
user-itinerary-table to store Itinerary where it can be mapped back to the original DayTrip via dayTripId attribute.
I am not sure if this is a good practice as there will be a lot of lookups happening and asynchronous operations are not possible here. This is because, to fetch the Itinerary, I need to wait for the completion of GetItem operation to get UserTrip, then, I will have the ids of the DayTrip and then, I will do another GetItem to fetch the DayTrip, then, finally, another GetItem to fetch the Itinerary.
Could the community here suggest a better, simpler solution?
Thanks!
Regarding the data structure, I don't see an absolute need for DayTrip, as you can get all that data from Itinerary. So in UserTrip I would keep a list of Itineraries instead of a list of DayTrips.
It wouldn't be good to store the UserTrip as is, with nested JSON
structure consisting of DayTrip, then Itinerary, right? It would mean
that the subTrip attribute of a particular UserTrip will be a huge
chuck of JSON. So I think everyone here would agree this is a no, no.
Is that correct?
Actually this is recommended in NoSQL databases, to have all data denormalised/embedded in the object. You use more storage, but avoid joins/processing. But keep in mind DynamoDB's item size limitation (currently 400KB).
In general, in NoSQL, you need to create your schema based on the queries you will need. For example in your case, you want to fetch all Itineraries of a UserTrip. Simply add userTripId to the Itinerary table. Create a GSI on Itinerary
with userTripId as hash key so you can query it efficiently. This way you will get all itinerary objects of a user trip.
Related
I have a collection called products which contains products like so
{
"name" :
"price" :
"category" :
"quality" :
}
and I have a user object that contains an array of products like so
{
"products" : []
}
My question - what is the best way to go about storing "product objects" in the products array.
Do I just put the corresponding ObjectID's inside the "products" array?
Do I literally put each "dictionary" as an entry in the products array?
The first approach would actually create a pointer so it seems the most logical, however that means that after I fetch the products array I would have to fetch each objectId from the products array and parse that.
The second option means that I wouldn't have to parse the each ObjectID, but there wouldn't be any link between the products in the product array and the product in the products collection
Any other way to go about this?
Thanks
In MongoDB we avoid joins and we reduce the number of collections for perfomance
and simplicity reasons.Document model allows us to do it.
But there are other things to consider
Are those data accessed together? (if yes + to embed)
for example when we need the users, we need the products also?
How big will be the products array? (if not too big + to embed)
How often those products will be updated? (if reraly + to embed)
in case of embed this means that we will have to update in many locations
How much duplication we will have? (if small + to embed)
for example a product can be in 5000 customers? this means that if a product changes we need to do all those extra updates
Is update even needed? (if no + to embed/duplicate)
for example if a customer gets on its products array a product that its price changes, it might be correct to pay the price it had when he added it to his product array, so update with refence would cause wrong results
Do we need both collections? (if no + to embed)
for example if products are always accessed from the customers, maybe we can avoid having a second collection
Data staleness is acceptable? (if its ok + to embed)
for example if product price changes and and the user has in its product array
a product with the old price is it ok?
(this might last some milliseconds but still can be problem)
Is data staleness problems, fixable? (if its ok + to embed)
for example lets say that the price changed and person go to pay,
and we are so unlucky that thise milliseconds/seconds caused the problem,
if it can detected at pay time(like find by productID and price),
then its fixable
if data staleness is not acceptable and not fixable then you can consider
transactions, but in this case i think reference is better
I guess there are more creteria to think about, but i think the above are
some of the basic things.
I'd like to know how I would deal with object states in a FireBase environment.
What do I mean by states? Well, let's say you have an app with which you organize order lists. Each list consists of a bunch of orders, so it can be considered a hierarchical data structure. Furthermore each list has a state which might be one of the following:
deferred
open
closed
sent
acknowledged
ware completely received
ware partially received
something else
On the visual (HTML) side the lists shall be distinguished by their state. Each state shall be presented to the client in its own, say, div-element, listing all the related orders beneath.
So the question is, how do I deal with this state in FireBase (or any other document based database)?
structure
Do I...
... (option 1) use a state-field for each orderlist and filter on the clientside by using if or something similar:
orderlist1.state = open
order1
order2
orderlist2.state = open
order1
orderlist3.state = closed
orderlist4.state = deferred
... (option 2) use the hierarchy of FireBase to classify the orderlists like so:
open
orderlist1
order1
order2
orderlist2
order1
closed
orderlist3
deferred
orderlist4
... (option 3) take a totally different approach?
So, what's the royal road here?
retrieval, processing & visual output of option 2
Since for option 1 the answer to this question is apparantly pretty straight forward (if state == ...) I continue with option 2: how do I retrieve the data in option 2? Do I use a Firebase-object for each state, like so:
var closedRef = new Firebase("https://xxx.firebaseio.com/closed");
var openRef = new Firebase("https://xxx.firebaseio.com/open");
var deferredRef = new Firebase("https://xxx.firebaseio.com/deferred");
var somethingRef = new Firebase("https://xxx.firebaseio.com/something");
Or what's considered the best approach to deal with that sort of data/structure?
There is no universal answer to this question. The "best approach" is going to depend on the particulars of your use case, which you haven't provided here. Specifically, how you will be reading and manipulating the data.
Data architecture in NoSQL is all about working hard on writes to make reads easy. It's all about how you plan to use the data. (It's also enough material for a chapter in a book.)
The advantage to "option 1" is that you can easily iterate all the entire list. Great if your list is measured in hundreds. This is a great approach if you want to fetch the list and manipulate it on the fly on the client side.
The advantage to "option 2" is that you can easily grab a subset of the list. Great if your list is measured in thousands and you will typically be fetching open issues only rather than closed ones. This is great for archiving/new/old lists like yours.
There are other options as well.
Sorted Data using Priorities
Perhaps the most universal approach is to use ordered data. This allows you to query a subset of your records using something like:
new Firebase(URL).startAt('open').endAt('open').limit(10);
This is sufficient in most cases where you have only one criteria, or when you can create a unique identifier from multiple criteria (e.g. 'open:marketing') without difficulty. Examples are scoreboards, state lists like yours, data ordered by timestamps.
Using an index
You can also create custom subsets of your data by creating an index of keys and using that to fetch the others.
This is most useful when there is no identifiable characteristic of your subsets. For example, if I pick them from a list and store my favorites.
I think my this plnkr can help you for this.
Here, click on edit/add and just check the country(order in your case) - State(state in your case) dependent dropdown may be the same as you want.just one single thing you may need to add is filter it.
They both are different tables in db.
You can also get it from git.
After working with neo4j and now coming to the point of considering to make my own entity manager (object manager) to work with the fetched data in the application, i wonder about neo4j's output format.
When i run a query it's always returned as tabular data. Why is this??
Sure tables keep a big place in data and processing, but it seems so strange that a graph database can only output in this format.
Now when i want to create an object graph in my application i would have to hydrate all the objects and this is not really good for performance and doesn't leverage true graph performace.
Consider MATCH (A)-->(B) RETURN A, B when there is one A and three B's, it would return:
A B
1 1
1 2
1 3
That's the same A passed down 3 times over the database connection, while i only need it once and i know this before the data is fetched.
Something like this seems great http://nigelsmall.com/geoff
a load2neo is nice, a load-from-neo would also be nice! either in the geoff format or any other formats out there https://gephi.org/users/supported-graph-formats/
Each language could then implement it's own functions to create the objects directly.
To clarify:
Relations between nodes are lost in tabular data
Redundant (non-optimal) format for graphs
Edges (relations) and vertices (nodes) are usually not in the same table. (makes queries more complex?)
Another consideration (which might deserve it's own post), what's a good way to model relations in an object graph? As objects? or as data/method inside the node objects?
#Kikohs
Q: What do you mean by "Each language could then implement it's own functions to create the objects directly."?
A: With an (partial) graph provided by the database (as result of a query) a language as PHP could provide a factory method (in C preferably) to construct the object graph (this is usually an expensive operation). But only if the object graph is well defined in a standard format (because this function should be simple and universal).
Q: Do you want to export the full graph or just the result of a query?
A: The result of a query. However a query like MATCH (n) OPTIONAL MATCH (n)-[r]-() RETURN n, r should return the full graph.
Q: you want to dump to the disk the subgraph created from the result of a query ?
A: No, existing interfaces like REST are prefered to get the query result.
Q: do you want to create the subgraph which comes from a query in memory and then request it in another language ?
A: no i want the result of the query in another format then tabular (examples mentioned)
Q: You make a query which only returns the name of a node, in this case, would you like to get the full node associated or just the name ? Same for the edges.
A: Nodes don't have names. They have properties, labels and relations. I would like enough information to retrieve A) The node ID, it's labels, it's properties and B) the relation to other nodes which are in the same result.
Note that the first part of the question is not a concrete "how-to" question, rather "why is this not possible?" (or if it is, i like to be proven wrong on this one). The second is a real "how-to" question, namely "how to model relations". The two questions have in common that they both try to find the answer to "how to get graph data efficiently in PHP."
#Michael Hunger
You have a point when you say that not all result data can be expressed as an object graph. It reasonable to say that an alternative output format to a table would only be complementary to the table format and not replacing it.
I understand from your answer that the natural (rawish) output format from the database is the result format with duplicates in it ("streams the data out as it comes"). I that case i understand that it's now left to an alternative program (of the dev stack) to do the mapping. So my conclusion on neo4j implementing something like this:
Pro's - not having to do this in every implementation language (of the application)
Con's - 1) no application specific mapping is possible, 2) no performance gain if implementation language is fast
"Even if you use geoff, graphml or the gephi format you have to keep all the data in memory to deduplicate the results."
I don't understand this point entirely, are you saying that these formats are no able to hold deduplicated results (in certain cases)?? So infact that there is no possible textual format with which a graph can be described without duplication??
"There is also the questions on what you want to include in your output?"
I was under the assumption that the cypher language was powerful enough to specify this in the query. And so the output format would have whatever the database can provide as result.
"You could just return the paths that you get, which are unique paths through the graph in themselves".
Useful suggestion, i'll play around with this idea :)
"The dump command of the neo4j-shell uses the approach of pulling the cypher results into an in-memory structure, enriching it".
Does the enriching process fetch additional data from the database or is the data already contained in the initial result?
There is more to it.
First of all as you said tabular results from queries are really commonplace and needed to integrate with other systems and databases.
Secondly oftentimes you don't actually return raw graph data from your queries, but aggregated, projected, sliced, extracted information out of your graph. So the relationships to the original graph data are already lost in most of the results of queries I see being used.
The only time that people need / use the raw graph data is when to export subgraph-data from the database as a query result.
The problem of doing that as a de-duplicated graph is that the db has to fetch all the result data data in memory first to deduplicate, extract the needed relationships etc.
Normally it just streams the data out as it comes and uses little memory with that.
Even if you use geoff, graphml or the gephi format you have to keep all the data in memory to deduplicate the results (which are returned as paths with potential duplicate nodes and relationships).
There is also the questions on what you want to include in your output? Just the nodes and rels returned? Or additionally all the other rels between the nodes that you return? Or all the rels of the returned nodes (but then you also have to include the end-nodes of those relationships).
You could just return the paths that you get, which are unique paths through the graph in themselves:
MATCH p = (n)-[r]-(m)
WHERE ...
RETURN p
Another way to address this problem in Neo4j is to use sensible aggregations.
E.g. what you can do is to use collect to aggregate data per node (i.e. kind of subgraphs)
MATCH (n)-[r]-(m)
WHERE ...
RETURN n, collect([r,type(r),m])
or use the new literal map syntax (Neo4j 2.0)
MATCH (n)-[r]-(m)
WHERE ...
RETURN {node: n, neighbours: collect({ rel: r, type: type(r), node: m})}
The dump command of the neo4j-shell uses the approach of pulling the cypher results into an in-memory structure, enriching it and then outputting it as cypher create statement(s).
A similar approach can be used for other output formats too if you need it. But so far there hasn't been the need.
If you really need this functionality it makes sense to write a server-extension that uses cypher for query specification, but doesn't allow return statements. Instead you would always use RETURN *, aggregate the data into an in-memory structure (SubGraph in the org.neo4j.cypher packages). And then render it as a suitable format (e.g. JSON or one of those listed above).
These could be a starting points for that:
https://github.com/jexp/cypher-rs
https://github.com/jexp/cypher_websocket_endpoint
https://github.com/neo4j-contrib/rabbithole/blob/master/src/main/java/org/neo4j/community/console/SubGraph.java#L123
There are also other efforts, like GraphJSON from GraphAlchemist: https://github.com/GraphAlchemist/GraphJSON
And the d3 json format is also pretty useful. We use it in the neo4j console (console.neo4j.org) to return the graph visualization data that is then consumed by d3 directly.
I've been working with neo4j for a while now and I can tell you that if you are concerned about memory and performances you should drop cypher at all, and use indexes and the other graph-traversal methods instead (e.g. retrieve all the relationships of a certain type from or to a start node, and then iterate over the found nodes).
As the documentation says, Cypher is not intended for in-app usage, but more as a administration tool. Furthermore, in production-scale environments, it is VERY easy to crash the server by running the wrong query.
In second place, there is no mention in the docs of an API method to retrieve the output as a graph-like structure. You will have to process the output of the query and build it.
That said, in the example you give you say that there is only one A and that you know it before the data is fetched, so you don't need to do:
MATCH (A)-->(B) RETURN A, B
but just
MATCH (A)-->(B) RETURN B
(you don't need to receive A three times because you already know these are the nodes connected with A)
or better (if you need info about the relationships) something like
MATCH (A)-[r]->(B) RETURN r
I have a mongo collection with documents that have a schema structured like the following:
{ _id : bla,
fname : foo,
lname : bar,
subdocs [ { subdocname : doc1
field1 : one
field2 : two
potentially_huge_array : [...]
}, ...
]
}
I'm using the ruby mongo driver that currently does not support elemMatch. I use an aggregation when extracting from subdocs via a project, unwind and match pipeline.
What I would now like to do is to page results from the potentially_huge_array array contained in the subdocument. I have not been able to figure out how to grab just a subset of the array without dragging the entire subdoc, huge array and all, out of the db into my app.
Is there some way to do this?
Would a different schema be a better way to handle this?
Depending on how huge is huge, you definitely don't want it embedded into another document.
The main reason is that unless you always want the array returned with the document, you probably don't want to store it as part of the document. How you can store it in another collection would depend on exactly how you want to access it.
Reviewing the types of queries you most often perform on your data will usually suggest the best schema - one that will allow you to be efficient about number of queries, the amount of data returned and ease of indexing the data.
If you field really huge and changes often, just placed it in separate collection.
I am trying a case where we changed a field name in our entity. we have something like this for example
class Person {
String name; //The original declaration was "String fullName"
}
According to objectify you have to use annonation #AutoLoad(""). This is ok and it works as Google Datastore doesn't delete the data Actually but it makes a new field so this annotation is like a mapping between the old and the new field. No problem when you are reading the whole table.
The problem arises when you apply a filter on your query (Suppose you made 5 objects with old name and 5 with new name). The result of your query depends on whether you used the old variable name or the new one (returns back only 5 but never the 10). It won't fetch both of them and map them. Any Suggestions for this problem? I hope i explained it in a clear way.
Thanks in advance
The simplest straight forward solution. fetch all data with the annonation "AutoLoad()". Then store them again. In this way they will be saved as the new field. The old one doesn't exist anymore or at least it doesn't contain any data anymore. It is like migrating the data from the old name to the new name. Anyone has better suggestions ?
If you've changed the name of your field, you need to load and re-put all your data (using the mapreduce API would be one option here). There's no magic way around this - the data you've stored exists with two different names on disk.
You can use #OldName
http://www.mail-archive.com/google-appengine-java#googlegroups.com/msg05586.html