I'm building a Threaded Messaging System that will be hosted on Google AppEngine
I've modeled it after the technique described by Brett Slatkin in Building Scalable, Complex Apps on App Engine
class Message(db.Model):
sender = db.StringProperty()
body = db.TextProperty()
class MessageIndex(db.Model):
receivers = db.StringListProperty()
The issue I'm having to determining the most efficient way to track the message state for a User.
For example is a message read, archived, deleted for a particular user.
Here are the solution I have come up with so far.
I'm using Datastore+'s StructuredProperty to add a state to the message MessageIndex
class Message(model.Model):
sender = model.StringProperty()
body = model.TextProperty()
class _DisplayState(model.Model):
user_key = model.KeyProperty()
state = model.IntegerProperty(default=0) # 0-unread, 1-read, 2-archived
class MessageIndex(model.Model):
receivers = model.StructuredProperty(_DisplayState, repeated=True)
This solution, while simple, negates the benefit of the MessageIndex. Additionally since the the MessageIndex is in the same entity group as the message datastore writes will be limited.
Full Source Code
What would be the most efficient way to accomplish this? Would adding an additional entity group be a better solution?
class MessageState(model.Model):
user_key = model.KeyProperty()
message_key = model.KeyPropery()
message_state = model.IntegerProperty(default=0) # 0-unread, 1-read, 2-archived
For the easiest querying - split your 'receivers' list into four different lists - 'unread', 'read', 'archived', 'deleted' and shuffle the receiver record between the lists as appropriate.
Related
My app links invoices, contracts and services with Many-to-One-Relationships:
class Invoice(models.Model):
contract = models.ForeignKey(Contract, on_delete=models.CASCADE)
class Contract(models.Model):
service = models.ForeignKey(Service, on_delete=models.CASCADE)
Whenever a new invoice is registered, it can be linked to a service and split/billed internally. Unfortunately, some contracts/invoices need to be linked to more than one service according to a fixed split (e.g. 30/70).
For this to work on the surface, I could to reverse the relationship between contracts and services –
class Service(models.Model):
contract = models.ForeignKey(Contract, on_delete=models.CASCADE)
– or change the ForeignKey field on the Contract class to a ManyToManyField.
But in both cases, I will not be able to get back from the invoice to the service easily anymore, as with the following statement:
invoices = Invoice.objects.filter(models.Q(contract__service__building=self.tenant.unit.building), models.Q(begin__lte=self.begin, end__gt=self.begin) | models.Q(begin__gt=self.begin, begin__lt=self.end))
Is it wise to insert an intermediate helper model (ContractService) with two ForeignKey fields to keep the current app logic and add the option to link a contract to more than one service?
Ok just to clarify one example:
Contract is "Cleaning of House"
Services are "Cleaning of first floor" and "Cleaning of second floor"
Invoices are "Invoice1", "Invoice2", ...
You want a relationship that "Invoice1" can be linked to "Cleaning of first floor" AND "Cleaning of second floor".
models.py
class Contract(models.Model):
"""Can hold multiple Services"""
pass
class Service(models.Model):
"""Is linked to one specific Contract"""
contract = models.ForeignKey(Contract, on_delete=models.CASCADE)
class Invoice(models.Model):
"""Can hold multiple Services and one Service can hold multiple Invoices"""
service = models.ManyToManyField(Service)
Now your question is:
"I can easily get the contract when I have the Service object, but how can I get the Service when I have the Contract object?
con = Contract.objects.all().first()
queryset = con.service_set.all() # gives you all related Services for that specific Contract
Read more about ManytoOne
And:
"How can I get the Invoice when I have the Service object?"
ser = Service.objects.all().first()
queryset = ser.invoice_set.all() # gives you all related Invoices for that specific Service
Read more about ManyToMany
Let me know how it goes
Thanks to #Tarquinius for your help – I found a solution based on his suggestion. The three models in question are connected as follows:
class Service(models.Model):
pass
class Contract(models.Model):
services = models.ManyToManyField(Service)
class Invoice(models.Model):
contract = models.ForeignKey(Contract, on_delete=models.CASCADE)
The query quoted above did not even need to be modified (apart from the different fieldname) to reflect the possibility of more than one service per contract, I just had to add the distinct() function:
invoices = Invoice.objects.filter(models.Q(contract__services__building=self.tenancy_agreement.unit.building), models.Q(begin__lte=self.begin, end__gt=self.begin) | models.Q(begin__gt=self.begin, begin__lt=self.end)).distinct()
In hindsight, this is quite obvious (and simple).
i am doing the finally degree work in Google App Engine, but i am having problems when i try this:
class Predicate(ndb.Model):
name = ndb.StringProperty()
parameters = ndb.JsonProperty()
class State(ndb.Model):
predicates = ndb.StructuredProperty(Predicate, repeated=True)
class Action(ndb.Model):
name = ndb.StringProperty()
parameters = ndb.StringProperty(repeated=True)
preconditions = ndb.StructuredProperty(Predicate, repeated=True)
predicatesToAdd = ndb.StructuredProperty(Predicate, repeated=True)
predicatesToDel = ndb.StructuredProperty(Predicate, repeated=True)
class Plan(ndb.Model):
plan = ndb.StructuredProperty(Predicate, repeated=True)
class Problem(ndb.Model):
initialState = ndb.StructuredProperty(Predicate)
goalState = ndb.StructuredProperty(Predicate)
actions = ndb.StructuredProperty(Action, repeated=True)
i get this error:
TypeError: This StructuredProperty cannot use repeated=True because its model class (Predicate) contains repeated properties (directly or indirectly).
StructuredProperty, if it contains repetitions, can not be replicated another StructuredProperty. But I need this structure models. How can i solve this?
And sorry for my bad english :(
I solved this problem using LocalStructuredProperty, but I think it will not work at all
The problem with your design is that ndb does not allow nested repeated properties. In other words, you cannot have a repeated structured property, which in turn has its own repeated property. If you remove the repeated=True from the parameters property, it will work.
You will need to re-think your design to work around this. One possible solution may be to use a JsonProperty for parameters, and store the list of strings as a JSON string. You won't be able to query them then of course, but it may work out depending on your requirements.
I need to store a lot of ModelA entities, and because app engine pricing is based on the number of entities written/read, I bundle together 100 entities and store them as one ModelB.
class ModelA(ndb.Expando):
a1 ... a20 = ndb.IntegerProperty()
class ModelB(ndb.Model):
data = ndb.StructuredProperty(ModelA, repeated=True)
I have only 80 such ModelB entities in my datastore, that should use around 1-2MB of memory, yet ModelB.query().fetch() takes 5 seconds. Is there any way to make this faster? Would using LocalStructuredProperty instead of StructuredProperty be better?
If you don't need to index the values (for queries), you might want to store them as an opaque object:
class ModelB(ndb.Model):
data = ndb.JsonProperty(indexed=False)
def add_data(self, model_a)
if not self.data:
self.data = []
self.data.append(model_a)
This will avoid the overhead of handling structured properties and validation.
I want to read related to model entity. What API I should use?
For example:
class DeleteMe(db.Model):
x = db.FloatProperty()
DeleteMe(key_name = '1').put()
How to read raw entity from datastore for key_name = '1'?
To get the corresponding model that you just put, use get_by_key_name. (https://developers.google.com/appengine/docs/python/datastore/modelclass#Model_get_by_key_name)
DeleteMe.get_by_key_name('1')
However, I noticed you're using the db package and not ndb. I would encourage you to use ndb as it has many optimizations and a more powerful API to the datastore.
https://developers.google.com/appengine/docs/python/ndb/
Corresponding code for NDB might look like:
from google.appengine.ext import ndb
class DeleteMe(ndb.Model):
x = ndb.FloatProperty()
DeleteMe(id='1').put()
DeleteMe.get_by_id('1')
I'm designing a model for a posting system where an entry contains an image with or without a comment. An user can reply to it as either a comment or as an image entry as well.
As there can be more properties for ImageEntry, I came up with this design with Polymodel. Not sure if this is the best way to do this. Storage-wise, is CommentEntry less than ImageEntry?
Any suggestions would be great.
class Entry(polymodel.PolyModel):
comment = db.TextProperty()
reply_to = db.SelfReferenceProperty() # reference to the entry
created_at = properties.DateTimeProperty(auto_now_add=True)
updated_at = properties.DateTimeProperty(auto_now=True)
class CommentEntry(Entry):
created_by = db.ReferenceProperty(User, collection_name='comment_entries')
class ImageEntry(Entry):
created_by = db.ReferenceProperty(User, collection_name='image_entries')
image_url = db.LinkProperty(indexed=False)
slug = db.StringProperty(indexed=False)
this model will work fine, and yes, a CommentEntry will be smaller than an ImageEntry from the same user if the ImageEntry has an image URL and/or slug.
however, i'd make this much simpler by putting
created_by, image_url, and slug into Entry and getting rid
of CommentEntry and ImageEntry altogether. since
the app engine datastore is schemaless,
and properties are optional by default,
you'll only pay the cost of the image_url and slug properties when you fill them
in for image entries.