I am using SQLAlchemy and try to manage a model "Media" which has a many-to-one relationship with a "Booking". Is it safe to call scoped_session.delete() from within a before_commit event?
def before_commit(session):
r""" Invokes the ``before_commit`` method on all items in the session.
This allows the models to perform an update-action depending on their
new data. """
for item in session.deleted:
if hasattr(item, 'before_commit'):
item.before_commit(session, 'deleted')
for item in session.dirty:
if hasattr(item, 'before_commit'):
item.before_commit(session, 'dirty')
for item in session.new:
if hasattr(item, 'before_commit'):
item.before_commit(session, 'new')
event.listen(db.session.__class__, 'before_commit', before_commit)
class Booking(db.Model):
# ...
media = db.relationship(Media, backref='booking')
def before_commit(self, session, status):
r""" Validates the booking's data. If the booking is being deleted,
all its media will be deleted with it. """
if status == 'deleted':
# Delete all the media that is associated with this booking.
for media in self.media:
session.delete(media)
a mass delete() using either session.execute("delete...") or session.query(cls).delete() should be fine, it just emits that SQL on the current connection.
As far as session.delete(obj), it looks like before_commit() is invoked before the final flush(), so in that sense you can treat it like a before_flush() event. Try it out and you should see the DELETE being emitted, and if so then you're fine.
Related
I working on an application that has its own database and gets user information from another serivce (an LDAP is this case, through an API package).
Say I have a tables called Articles, with a column user_id. There is no Users table, instead a user or set of users is retrieved through the external API:
$user = LDAPConnector::getUser($user_id);
$users = LDAPConnector::getUsers([1, 2, 5, 6]);
Of course I want retrieving data from inside a controller to be as simple as possible, ideally still with something like:
$articles = $this->Articles->find()->contain('Users');
foreach ($articles as $article) {
echo $article->user->getFullname();
}
I'm not sure how to approach this.
Where should I place the code in the table object to allow integration with the external API?
And as a bonus question: How to minimise the number of LDAP queries when filling the Entities?
i.e. it seems to be a lot faster by first retrieving the relevant users with a single ->getUsers() and placing them later, even though iterating over the articles and using multiple ->getUser() might be simpler.
The most simple solution would be to use a result formatter to fetch and inject the external data.
The more sophisticated solution would a custom association, and a custom association loader, but given how database-centric associations are, you'd probably also have to come up with a table and possibly a query implementation that handles your LDAP datasource. While it would be rather simple to move this into a custom association, containing the association will look up a matching table, cause the schema to be inspected, etc.
So I'll stick with providing an example for the first option. A result formatter would be pretty simple, something like this:
$this->Articles
->find()
->formatResults(function (\Cake\Collection\CollectionInterface $results) {
$userIds = array_unique($results->extract('user_id')->toArray());
$users = LDAPConnector::getUsers($userIds);
$usersMap = collection($users)->indexBy('id')->toArray();
return $results
->map(function ($article) use ($usersMap) {
if (isset($usersMap[$article['user_id']])) {
$article['user'] = $usersMap[$article['user_id']];
}
return $article;
});
});
The example makes the assumption that the data returned from LDAPConnector::getUsers() is a collection of associative arrays, with an id key that matches the user id. You'd have to adapt this accordingly, depending on what exactly LDAPConnector::getUsers() returns.
That aside, the example should be rather self-explanatory, first obtain a unique list of users IDs found in the queried articles, obtain the LDAP users using those IDs, then inject the users into the articles.
If you wanted to have entities in your results, then create entities from the user data, for example like this:
$userData = $usersMap[$article['user_id']];
$article['user'] = new \App\Model\Entity\User($userData);
For better reusability, put the formatter in a custom finder. In your ArticlesTable class:
public function findWithUsers(\Cake\ORM\Query $query, array $options)
{
return $query->formatResults(/* ... */);
}
Then you can just do $this->Articles->find('withUsers'), just as simple as containing.
See also
Cookbook > Database Access & ORM > Query Builder > Adding Calculated Fields
Cookbook > Database Access & ORM > Retrieving Data & Results Sets > Custom Finder Methods
I have a many-to-many structure mapped to entity framework. This is a sample of what it looks like:
User UserTag Tag
------- -------- -------
IdUser(PK) IdUserTag(PK) IdTag(PK)
Name IdUser(FK) TagName
Desc IdTag(FK) Active
Now, I needed to exclude from any request of any method the viewing of Tags that were Active=false.
First, I tried doing it manually in every method, like:
public User GetById(int id)
{
var item = UserRepository.GetById(id); //This is just a repository that calls the EF context
//EF automatically maps it to the *UserTags* property
foreach(var tag in item.UserTags)
{
if(tag.Tag.Active == false)
item.UserTags.Remove(tag);
}
}
But it throws the following exception:
The relationship could not be changed because one or more of the foreign-key properties is non-nullable
So, I wanted to know if there's a way to conditionaly filter every request made to a specific table, whether it is select or a join request.
Try this in your GetById method:
var user.UserTags = dbContext.Entry(user)
.Collection(u => u.UserTags)
.Query()
.Where(ut => ut.Active == true)
.ToList();
The supplied code fails because it is attempting to remove items from the data entities not the list. If you want to pass the data entity around instead of the data model, you need to not use Remove. Something like the below (untested should work).
tags = item.UserTags.Where((ut) => ut.Active).ToList();
This line will get you a list of data entities that are active. However, you should really map all of this into a data model (see AutoMapper) and then you would not be removing items from the database.
My app has sales listing functionality that will allow the user to add 1 or more photos for the product that they want to sell.
I'm attempting to use the upload/filestore_image of ATK with a Join table to create the relationship - my models:
class Model_Listing extends Model_Table {
public $entity_code='listing';
function init(){
parent::init();
$this->addField('name');
$this->addField('body')->type('text');
$this->addField('status');
$this->addField('showStatus')->calculated(true);
}
function calculate_showStatus(){
return ($this->status == 1) ? "Sold" : "For Sale" ;
}
}
class Model_listingimages extends Model_Table {
public $entity_code='listing_images';
function init(){
parent::init();
$this->addField('listing_id')->refModel('Model_Listing');
$this->addField('filestore_image_id')->refModel('Model_Filestore_Image');
}
}
In my page manager class I have added the file upload to the crud:
class page_manager extends Page {
function init(){
parent::init();
$tabs=$this->add('Tabs');
$s = $tabs->addTab('Sales')->add('CRUD');
$s->setModel('Listing',array('name','body','status'),array('name','status'));
if ($s->form) {
$f = $s->form;
$f->addField('upload','Add Photos')->setModel('Filestore_Image');
$f->add('FileGrid')->setModel('Filestore_Image');
}
}
}
My questions:
I am getting a "Unable to include FileGrid.php" error - I want the user to be able to see the images that they have uploaded and hoped that this would be the best way to do so - by adding the file grid to bottom of the form. - EDIT - ignore this question, I created a FileGrid class based on the code in the example link below - that fixed the issue.
How do I make the association between the CRUD form so that a submit will save the uploaded files and create entries in the join table?
I have installed the latest release of ATK4, added the 4 filestore tables to the db and referenced the following page in the documentation http://codepad.agiletoolkit.org/image
TIA
PG
By creating model based on Filestore_File
You need to specify a proper model. By proper I mean:
It must be extending Model_Filestore_File
It must have MasterField set to link it with your entry
In this case, however you must know the referenced ID when the images are being uploaded, so it won't work if you upload image before creating record. Just to give you idea the code would look
$mymodel=$this->add('Model_listingimages');
$mymodel->setMasterField('listing_id',$listing_id);
$upload_field->setModel($mymodel);
$upload_field->allowMultiple();
This way all the images uploaded through the field will automatically be associated with your listing. You will need to inherit model from Model_Filestore_File. The Model_Filestore_Image is a really great example which you can use. You should add related entity (join) and define fields in that table.
There is other way too:
By doing some extra work in linking images
When form is submitted, you can retrieve list of file IDs by simply getting them.
$form->get('add_photos')
Inside form submission handler you can perform some manual insertion into listingimages.
$form->onSubmit(function($form) uses($listing_id){
$photos = explode(',',$form->get('add_photos'));
$m=$form->add('Model_listingimages');
foreach($photos as $photo_id){
$m->unloadDdata()->set('listing_id',$listing_id)
->set('filestore_image_id',$photo_id)->update();
}
}); // I'm not sure if this will be called by CRUD, which has
// it's own form submit handler, but give it a try.
You must be careful, through, if you use global model inside the upload field without restrictions, then user can access or delete images uploaded by other users. If you use file model with MVCGrid you should see what files they can theoretically get access to. That's normal and that's why I recommend using the first method described above.
NOTE: you should not use spaces in file name, 2nd argument to addField, it breaks javascript.
django nonrel's documentation states: "you have to manually write code for merging the results of multiple queries (JOINs, select_related(), etc.)".
Can someone point me to any snippets that manually add the related data? #nickjohnson has an excellent post showing how to do this with the straight AppEngine models, but I'm using django-nonrel.
For my particular use I'm trying to get the UserProfiles with their related User models. This should be just two simple queries, then match the data.
However, using django-nonrel, a new query gets fired off for each result in the queryset. How can I get access to the related items in a 'select_related' sort of way?
I've tried this, but it doesn't seem to work as I'd expect. Looking at the rpc stats, it still seems to be firing a query for each item displayed.
all_profiles = UserProfile.objects.all()
user_pks = set()
for profile in all_profiles:
user_pks.add(profile.user_id) # a way to access the pk without triggering the query
users = User.objects.filter(pk__in=user_pks)
for profile in all_profiles:
profile.user = get_matching_model(profile.user_id, users)
def get_matching_model(key, queryset):
"""Generator expression to get the next match for a given key"""
try:
return (model for model in queryset if model.pk == key).next()
except StopIteration:
return None
UPDATE:
Ick... I figured out what my issue was.
I was trying to improve the efficiency of the changelist_view in the django admin. It seemed that the select_related logic above was still producing additional queries for each row in the results set when a foreign key was in my 'display_list'. However, I traced it down to something different. The above logic does not produce multiple queries (but if you more closely mimic Nick Johnson's way it will look a lot prettier).
The issue is that in django.contrib.admin.views.main on line 117 inside the ChangeList method there is the following code: result_list = self.query_set._clone(). So, even though I was properly overriding the queryset in the admin and selecting the related stuff, this method was triggering a clone of the queryset which does NOT keep the attributes on the model that I had added for my 'select related', resulting in an even more inefficient page load than when I started.
Not sure what to do about it yet, but the code that selects related stuff is just fine.
I don't like answering my own question, but the answer might help others.
Here is my solution that will get related items on a queryset based entirely on Nick Johnson's solution linked above.
from collections import defaultdict
def get_with_related(queryset, *attrs):
"""
Adds related attributes to a queryset in a more efficient way
than simply triggering the new query on access at runtime.
attrs must be valid either foreign keys or one to one fields on the queryset model
"""
# Makes a list of the entity and related attribute to grab for all possibilities
fields = [(model, attr) for model in queryset for attr in attrs]
# we'll need to make one query for each related attribute because
# I don't know how to get everything at once. So, we make a list
# of the attribute to fetch and pks to fetch.
ref_keys = defaultdict(list)
for model, attr in fields:
ref_keys[attr].append(get_value_for_datastore(model, attr))
# now make the actual queries for each attribute and store the results
# in a dict of {pk: model} for easy matching later
ref_models = {}
for attr, pk_vals in ref_keys.items():
related_queryset = queryset.model._meta.get_field(attr).rel.to.objects.filter(pk__in=set(pk_vals))
ref_models[attr] = dict((x.pk, x) for x in related_queryset)
# Finally put related items on their models
for model, attr in fields:
setattr(model, attr, ref_models[attr].get(get_value_for_datastore(model, attr)))
return queryset
def get_value_for_datastore(model, attr):
"""
Django's foreign key fields all have attributes 'field_id' where
you can access the pk of the related field without grabbing the
actual value.
"""
return getattr(model, attr + '_id')
To be able to modify the queryset on the admin to make use of the select related we have to jump through a couple hoops. Here is what I've done. The only thing changed on the 'get_results' method of the 'AppEngineRelatedChangeList' is that I removed the self.query_set._clone() and just used self.query_set instead.
class UserProfileAdmin(admin.ModelAdmin):
list_display = ('username', 'user', 'paid')
select_related_fields = ['user']
def get_changelist(self, request, **kwargs):
return AppEngineRelatedChangeList
class AppEngineRelatedChangeList(ChangeList):
def get_query_set(self):
qs = super(AppEngineRelatedChangeList, self).get_query_set()
related_fields = getattr(self.model_admin, 'select_related_fields', [])
return get_with_related(qs, *related_fields)
def get_results(self, request):
paginator = self.model_admin.get_paginator(request, self.query_set, self.list_per_page)
# Get the number of objects, with admin filters applied.
result_count = paginator.count
# Get the total number of objects, with no admin filters applied.
# Perform a slight optimization: Check to see whether any filters were
# given. If not, use paginator.hits to calculate the number of objects,
# because we've already done paginator.hits and the value is cached.
if not self.query_set.query.where:
full_result_count = result_count
else:
full_result_count = self.root_query_set.count()
can_show_all = result_count self.list_per_page
# Get the list of objects to display on this page.
if (self.show_all and can_show_all) or not multi_page:
result_list = self.query_set
else:
try:
result_list = paginator.page(self.page_num+1).object_list
except InvalidPage:
raise IncorrectLookupParameters
self.result_count = result_count
self.full_result_count = full_result_count
self.result_list = result_list
self.can_show_all = can_show_all
self.multi_page = multi_page
self.paginator = paginator
I want to retrieve all the data from 3 tables
users , properties and users_properties.
So I decided I would use the manytomanyRowset. But to my surprise I get the data from the properties and users_properties table but no data from the users table. Why is that? I need some columns from the users table is there a way to tell the manytomanyrowset function that I need the data from the current table as well?
this is my function
public function fetchRegisteredProperties()
{
$userTable = $this->getTable();
require_once APPLICATION_PATH . '/models/DbTable/UsersPropertiesDB.php';
require_once APPLICATION_PATH . '/models/DbTable/PropertiesDB.php';
$propertiesRowset = $table->fetchAll();
$allProperties = array();
foreach ($propertiesRowset as $row) {
$propertiesRowset = $row->findManyToManyRowset(
'Model_DbTable_Properties','Model_DbTable_UsersProperties');
$allProperties = array_merge($tempArray,$propertiesRowset->toArray());
}
return $allProperties;
}
thanks in adavance
I designed and coded the table-relationships features in Zend Framework.
The answer to your question is no, the findManyToManyRowset() method only fetches rows from the related table, it does not merge them into the corresponding Row object. The reason is that a Row object in ZF can save() itself back to the database, and if you add fields it won't know what to do with them.
So you should implement a custom Row object to hold both user fields and the collection of user properties -- store the user properties as a Rowset object.
Then extend __get() and __set() so that it knows how to map fields into the correct array when you read or write object properties. That is, if one tries to read or write a field that isn't part of the user row, it falls back to the user properties Rowset.
Also extend save() to save not only the current row, but also call save() on the Rowset of user properties.