Same code, in production don't work while in development yes.
models:
class Comput(ndb.Model):
#staticmethod
def membership(user):
q = ndb.gql("SELECT * FROM Members WHERE member = :1", user)
result = q.fetch()
return [m.comput.get() for m in result]
#return [m.comput.get() for m in q] #same issue (dev yes, prod no)
class Members(ndb.Model):
comput = ndb.KeyProperty(kind=Comput)
member = ndb.UserProperty()
handler:
comput_list = Comput.membership(users.get_current_user())
template:
{% for comput in comput_list %}
<tr onclick="location.href='/comput?id={$ comput.key.id() $}'">
Traceback (only in production):
UndefinedError: 'None' has no attribute 'key'
This appears even when comput_list have entities.
update: No autogenerate index for Members, I tried to add manually but nothing changes
Looks like one of your Members objects has None for its comput property. I would use the production data viewer (or add a log statements) to find which member it is.
Related
I am facing a very weird problem in one of my django projects. In my project I have a custom field class that handles foreign keys, one to one and many 2 many model fields. The class is some thing like the following.
from django import forms
class CustomRelatedField(forms.Field):
def __init__(self, model, limit=None, multiple=False, create_objects=True, *args, *kwargs):
self.model = model
self.limit = limit
self.multiple = multiple
self.create_objects = create_objects
super(CustomRelatedField, self).__init__(*args, **kwargs)
def clean(self, value):
""" Calls self.get_objects to get the actual model object instance(s)
from the given unicode value.
"""
# Do some value processing here
return self.get_objects(value)
def get_objects(self, values):
""" Returns the model object instances for the given unicode values.
"""
results = []
for value in values:
try:
obj = self.model.object.get_or_create(name=value)[0]
results.append(obj)
except Exception, err:
# Log the error here.
return results
def prepare_value(self, value):
""" Returns the value to be sent to the UI. The value
passed to this method is generally the object id or
a list of object id's (in case it is a many to many object).
So we need to make a database query to get the object and
then return the name attribute.
"""
if self.multiple:
result = [obj.name for obj in self.model.filter(pk__in=value)]
else:
result = self.model.object.get(pk=value)
return result
Recently while I was playing with the django-toolbar, I found out one of the pages that has a form with the above mentioned fields was ridiculously making multiple queries for the same objects again and again.
While debugging, I found out the prepare_value method was being called again and again. After some more debugging, I realized the culprit was the template. I have a generic template that I use for forms, It looks something like the following:
{% for field in form %}
{% if field.is_hidden %}
<!-- Do something here -->
{% endif %}
{% if field.field.required %}
<!-- Do something here -->
{% endif %}
<label>{{ field.label }}</label>
<div class="form-field">{{ field }}</div>
{% if field.field.widget.attrs.help_text %}
<!-- Do something here -->
{% elif field.errors %}
<!-- Do something here -->
{% endif %}
{% endfor %}
In the above code, each if statement calls the field class which calls the prepare_value method which then makes the database queries. Each of the following listed is making a database query, I am totally lost to why this is happening and have no clue about any solutions. Any help, suggestions would be really appreciated. Thanks.
field.is_hidden
field.field.required
field.label
field.label_tag
field
field.field.widget.attrs.help_text
field.errors
Also, why does this happen with my custom field class only, other fields (FKs, O2Os, M2M's) in the application and the application admin, just make one query, even though they are using a similar template.
Problem is with your prepare_value() method which does explicit queries. .get() does not get cached and always hits the db while iterating on .filter() queryset will evaluate that.
This might be causing you multiple queries.
This is not seen in default fields because they do not do any queries in prepare_value().
To resolve this, you can try to cache the value and result. If value hasn't changed, return cached result. Something like:
class CustomRelatedField(forms.Field):
def __init__(self, model, limit=None, multiple=False, create_objects=True, *args, *kwargs):
self.cached_result = None
self.cached_value = None
...
def prepare_value(self, value):
#check we have cached result
if value == self.cached_value:
return self.cached_result
if self.multiple:
result = [obj.name for obj in self.model.filter(pk__in=value)]
else:
result = self.model.object.get(pk=value)
#cache the result and value
self.cached_result = result
self.cached_value = value
return result
Not sure how good/bad this work around though!
Recently updated to Mongoid 3.1 from 3.0.3 and this resulted in some broken code and confusion on my side.
Say you have a pair of classes with a belongs_to/has_many relationship, like so:
class Band
include Mongoid::Document
field :name, type: String
has_many :members, :autosave => true
end
class Member
include Mongoid::Document
field :name, type: String
belongs_to :band
end
Saving all this to the database like so:
b = Band.new
b.name = "Sonny and Cher"
b.members << Member.new(name: "Sonny")
b.members << Member.new(name: "Cher")
b.save
I would in my API, be able to return a 'member' object like so:
m = Member.where(name: "Sonny").first
m.to_json
which yields the following, as expected:
{"_id":"<removed>","band_id":"5151d89f5dd99dd9ec000002","name":"Sonny"}
My client can request the full band object with a subsequent call if it wants to. However, in some cases I DO want to include the referenced item directly. With 3.0.3, I would just do the following:
m = Member.where(name: "Sonny").first
m[:band] = m.band
m.to_json
and this would add a new field with the full band information to it. With 3.1, however (it may have started in earlier versions, but I didn't test), I now get this:
{"_id":"<removed>","band_id":{"_id":"5151dc025dd99d579e000002","name":"Sonny and Cher"},"name":"Sonny"}
So, it looks like the band info has been eager-loaded into the field? Why is it stored under the key ':band_id' and not ':band'? I guess ':band' is protected, but I still don't think the data should be stored under the ':band_id' key. I suspect I am missing something here. Any ideas?
You can specify an :include option for to_json like so:
m.to_json(include: :band)
The JSON will then have a key band with the Band object converted to JSON and band_id will still be present.
django nonrel's documentation states: "you have to manually write code for merging the results of multiple queries (JOINs, select_related(), etc.)".
Can someone point me to any snippets that manually add the related data? #nickjohnson has an excellent post showing how to do this with the straight AppEngine models, but I'm using django-nonrel.
For my particular use I'm trying to get the UserProfiles with their related User models. This should be just two simple queries, then match the data.
However, using django-nonrel, a new query gets fired off for each result in the queryset. How can I get access to the related items in a 'select_related' sort of way?
I've tried this, but it doesn't seem to work as I'd expect. Looking at the rpc stats, it still seems to be firing a query for each item displayed.
all_profiles = UserProfile.objects.all()
user_pks = set()
for profile in all_profiles:
user_pks.add(profile.user_id) # a way to access the pk without triggering the query
users = User.objects.filter(pk__in=user_pks)
for profile in all_profiles:
profile.user = get_matching_model(profile.user_id, users)
def get_matching_model(key, queryset):
"""Generator expression to get the next match for a given key"""
try:
return (model for model in queryset if model.pk == key).next()
except StopIteration:
return None
UPDATE:
Ick... I figured out what my issue was.
I was trying to improve the efficiency of the changelist_view in the django admin. It seemed that the select_related logic above was still producing additional queries for each row in the results set when a foreign key was in my 'display_list'. However, I traced it down to something different. The above logic does not produce multiple queries (but if you more closely mimic Nick Johnson's way it will look a lot prettier).
The issue is that in django.contrib.admin.views.main on line 117 inside the ChangeList method there is the following code: result_list = self.query_set._clone(). So, even though I was properly overriding the queryset in the admin and selecting the related stuff, this method was triggering a clone of the queryset which does NOT keep the attributes on the model that I had added for my 'select related', resulting in an even more inefficient page load than when I started.
Not sure what to do about it yet, but the code that selects related stuff is just fine.
I don't like answering my own question, but the answer might help others.
Here is my solution that will get related items on a queryset based entirely on Nick Johnson's solution linked above.
from collections import defaultdict
def get_with_related(queryset, *attrs):
"""
Adds related attributes to a queryset in a more efficient way
than simply triggering the new query on access at runtime.
attrs must be valid either foreign keys or one to one fields on the queryset model
"""
# Makes a list of the entity and related attribute to grab for all possibilities
fields = [(model, attr) for model in queryset for attr in attrs]
# we'll need to make one query for each related attribute because
# I don't know how to get everything at once. So, we make a list
# of the attribute to fetch and pks to fetch.
ref_keys = defaultdict(list)
for model, attr in fields:
ref_keys[attr].append(get_value_for_datastore(model, attr))
# now make the actual queries for each attribute and store the results
# in a dict of {pk: model} for easy matching later
ref_models = {}
for attr, pk_vals in ref_keys.items():
related_queryset = queryset.model._meta.get_field(attr).rel.to.objects.filter(pk__in=set(pk_vals))
ref_models[attr] = dict((x.pk, x) for x in related_queryset)
# Finally put related items on their models
for model, attr in fields:
setattr(model, attr, ref_models[attr].get(get_value_for_datastore(model, attr)))
return queryset
def get_value_for_datastore(model, attr):
"""
Django's foreign key fields all have attributes 'field_id' where
you can access the pk of the related field without grabbing the
actual value.
"""
return getattr(model, attr + '_id')
To be able to modify the queryset on the admin to make use of the select related we have to jump through a couple hoops. Here is what I've done. The only thing changed on the 'get_results' method of the 'AppEngineRelatedChangeList' is that I removed the self.query_set._clone() and just used self.query_set instead.
class UserProfileAdmin(admin.ModelAdmin):
list_display = ('username', 'user', 'paid')
select_related_fields = ['user']
def get_changelist(self, request, **kwargs):
return AppEngineRelatedChangeList
class AppEngineRelatedChangeList(ChangeList):
def get_query_set(self):
qs = super(AppEngineRelatedChangeList, self).get_query_set()
related_fields = getattr(self.model_admin, 'select_related_fields', [])
return get_with_related(qs, *related_fields)
def get_results(self, request):
paginator = self.model_admin.get_paginator(request, self.query_set, self.list_per_page)
# Get the number of objects, with admin filters applied.
result_count = paginator.count
# Get the total number of objects, with no admin filters applied.
# Perform a slight optimization: Check to see whether any filters were
# given. If not, use paginator.hits to calculate the number of objects,
# because we've already done paginator.hits and the value is cached.
if not self.query_set.query.where:
full_result_count = result_count
else:
full_result_count = self.root_query_set.count()
can_show_all = result_count self.list_per_page
# Get the list of objects to display on this page.
if (self.show_all and can_show_all) or not multi_page:
result_list = self.query_set
else:
try:
result_list = paginator.page(self.page_num+1).object_list
except InvalidPage:
raise IncorrectLookupParameters
self.result_count = result_count
self.full_result_count = full_result_count
self.result_list = result_list
self.can_show_all = can_show_all
self.multi_page = multi_page
self.paginator = paginator
i'm building a django app from an existing database. for better or worse, i have a couple of views that i would like to base my models off. they look something like this:
class Device(Model):
id = models.IntegerField( primary_key=True, db_column='node_id' )
name = models.CharField(max_length=127, db_column='node' )
class Meta:
db_table = 'node' # db view
managed = False
class Entity(Model):
id = models.IntegerField( primary_key=True, db_column='_id' )
device = models.ForeignKey(Device, db_column='node_id' )
class Meta:
db_table = 'entity' # db view
managed = Fase
so things work okay. however, when i try to use a template that utilises the ForeignKey it is very slow:
{% for e in entities %}
{{ e.device.name }}
{% endfor %}
looking at the logs, it appears to be repeating queries for each 'node_id', and ultimately timeout out.
(of course, if i do not include e.device.name is all quick)
is there a way i can optimise this?
to be fair, the 'entity' view already has the name of the device as another field (node), so i could use this instead but i would like the relation to exist.
have you try .select_related() in your view?
entities = Entity.objects.select_related('device').filter(...)
in the other hand, if it is and old database, and no autogenerated by django, probably node_id may not be an index in the database, this will slow down a lot any JOIN.
I just have a hunch about this. But if feels like I'm doing it the wrong way. What I want to do is to have a db.StringProperty() as a unique identifier. I have a simple db.Model, with property name and file. If I add another entry with the same "name" as one already in the db.Model I want to update this.
As of know I look it up with:
template = Templates.all().filter('name = ', name)
Check if it's one entry already:
if template.count() > 0:
Then add it or update it. But from what I've read .count() is every expensive in CPU usage.
Is there away to set the "name" property to be unique and the datastore will automatic update it or another better way to do this?
..fredrik
You can't make a property unique in the App Engine datastore. What you can do instead is to specify a key name for your model, which is guaranteed to be unique - see the docs for details.
I was having the same problem and came up with the following answer as the simplest one :
class Car(db.Model):
name = db.StringProperty(required=True)
def __init__(self,*args, **kwargs):
super(Car, self).__init__(*args, **kwargs)
loadingAnExistingCar = ("key" in kwargs.keys() or "key_name" in kwargs.keys())
if not loadingAnExistingCar:
self.__makeSureTheCarsNameIsUnique(kwargs['name'])
def __makeSureTheCarsNameIsUnique(self, name):
existingCarWithTheSameName = Car.GetByName(name)
if existingCarWithTheSameName:
raise UniqueConstraintValidationException("Car should be unique by name")
#staticmethod
def GetByName(name):
return Car.all().filter("name", name).get()
It's important to not that I first check if we are loading an existing entity first.
For the complete solution : http://nicholaslemay.blogspot.com/2010/07/app-engine-unique-constraint.html
You can just try to get your entity and edit it, and if not found create a new one:
template = Templates.gql('WHERE name = :1', name)
if template is None:
template = Templates()
# do your thing to set the entity's properties
template.put()
That way it will insert a new entry when it wasn't found, and if it was found it will update the existing entry with the changes you made (see documentation here).
An alternative solution is to create a model to store the unique values, and store it transationally using a combination of Model.property_name.value as key. Only if that value is created you save your actual model. This solution is described (with code) here:
http://squeeville.com/2009/01/30/add-a-unique-constraint-to-google-app-engine/
I agree with Nick. But, if you do ever want to check for model/entity existence based on a property, the get() method is handy:
template = Templates.all().filter('name = ', name).get()
if template is None:
# doesn't exist
else:
# exists
I wrote some code to do this. The idea for it is to be pretty easy to use. So you can do this:
if register_property_value('User', 'username', 'sexy_bbw_vixen'):
return 'Successfully registered sexy_bbw_vixen as your username!'
else:
return 'The username sexy_bbw_vixen is already in use.'
This is the code. There are a lot of comments, but its actually only a few lines:
# This entity type is a registry. It doesn't hold any data, but
# each entity is keyed to an Entity_type-Property_name-Property-value
# this allows for a transaction to 'register' a property value. It returns
# 'False' if the property value is already in use, and thus cannot be used
# again. Or 'True' if the property value was not in use and was successfully
# 'registered'
class M_Property_Value_Register(db.Expando):
pass
# This is the transaction. It returns 'False' if the value is already
# in use, or 'True' if the property value was successfully registered.
def _register_property_value_txn(in_key_name):
entity = M_Property_Value_Register.get_by_key_name(in_key_name)
if entity is not None:
return False
entity = M_Property_Value_Register(key_name=in_key_name)
entity.put()
return True
# This is the function that is called by your code, it constructs a key value
# from your Model-Property-Property-value trio and then runs a transaction
# that attempts to register the new property value. It returns 'True' if the
# value was successfully registered. Or 'False' if the value was already in use.
def register_property_value(model_name, property_name, property_value):
key_name = model_name + '_' + property_name + '_' + property_value
return db.run_in_transaction(_register_property_value_txn, key_name )