What options are there there for resolving nested resources in ngResource responses?
There have been some related questions about resolving endpoints for nested resource in ngResource, but this question is about when a REST response contains a second resource nested in the collection that is being queried, especially 1-to-1 mappings where you wouldn't have e.g. pets/<id>/owner as its own resource.
Say there are two resources, Pets and Owners:
GET /pets:
[{
name: 'spark',
type: 'dog',
owner: '/owners/3/' # alternatively just '3' or the full object.
}]
As a developer, I sometimes want to query the Owner resource as a whole, sometimes I want to query the Pet resource and then I automatically want to resolve the owner attribute into a resource instance.
This is my current solution:
.factory('Pet', function ($resource, Owner) {
var Pet = $resource('/pets/:id', {id: '#id'});
Pet.prototype.getOwner = function () {
return new Owner(this.owner); // or Owner.get({id: this.owner})
}
return Pet;
})
Problems here are many. There's integrity – for one. This implementation, I believe, allows for multiple instances of the same resource. Then there's practicality. You also have additional attributes to keep track of (owner and getOwner(), instead of just owner; possibly setOwner if you want to be able to save the model).
An alternative solution could be built on transformResponse, but it would feel like a hack to include that in every resource that has a nested mapping.
I believe this is the exact reason why Martin Gontovnikas created Restangular. He didn't like having to deal with nested $resources in the main angular framework. I think his Restangular solution would fit nicely into your needs. His code is on GitHub here and he's got a nice intro video on youtube here.
Check it out. I think you'll find it does exactly what you want it to do.
Update: I ended up working on this for a bit and have started a new angular module, available on GitHub. The answer below is about the Gist I wrote originally.
There doesn't seem to be anything around there like what I have been looking for. I have started an implementation of a solution that only supports get and getList (query) operations. The remaining methods should be trivial to add since I've pretty much kept with the layout of the ngResource module. The Gist for my implementation is below.
https://gist.github.com/lyschoening/7102262
Resources can be embedded in JSON either as full objects that simply get wrapped in the correct Resource model, or as URIs, which get resolved automatically. In addition to embedded resources, the module also supports typical nested resources, either as true parent-child collections (where the resource is only accessible after selecting the parent) or as cross-referenced collection.
Yard = Resource('/yard') # resource model
Yard.$nested('trees') # embedded item or list of items
Chair = Resource('/chair')
Yard.$nested('/chair') # sub-collection without its own model
# (for many-to-many)
Tree = Resource('/tree')
# child-collection with its own model
TreeHouse = Tree.$childResource('/treehouse')
yard = Yard.get(1)
# GET /yard/1
# {
# "uri": "/yard/1",
# "trees": [
# "/tree/15", -- reference, looked-up automatically with GET
# {"uri": "/tree/16", "name": "Apple tree"}
# -- full object, resolved to Tree instance
# ]
# }
# GET /tree/16
# {"uri": "/tree/15", "name": "Pine tree"}
yard.chair.getList()
# GET /yard/1/chair
# [{"uri": "/chair/1", ...}, ..]
# -- model inferred from URI
yard.trees[0].treehouse.getList()
# GET /tree/15/treehouse
# [{"uri": "/tree/15/treehouse/1", ...}, ..]
# -- automatically resolved to TreeHouse instance
Related
I working on an application that has its own database and gets user information from another serivce (an LDAP is this case, through an API package).
Say I have a tables called Articles, with a column user_id. There is no Users table, instead a user or set of users is retrieved through the external API:
$user = LDAPConnector::getUser($user_id);
$users = LDAPConnector::getUsers([1, 2, 5, 6]);
Of course I want retrieving data from inside a controller to be as simple as possible, ideally still with something like:
$articles = $this->Articles->find()->contain('Users');
foreach ($articles as $article) {
echo $article->user->getFullname();
}
I'm not sure how to approach this.
Where should I place the code in the table object to allow integration with the external API?
And as a bonus question: How to minimise the number of LDAP queries when filling the Entities?
i.e. it seems to be a lot faster by first retrieving the relevant users with a single ->getUsers() and placing them later, even though iterating over the articles and using multiple ->getUser() might be simpler.
The most simple solution would be to use a result formatter to fetch and inject the external data.
The more sophisticated solution would a custom association, and a custom association loader, but given how database-centric associations are, you'd probably also have to come up with a table and possibly a query implementation that handles your LDAP datasource. While it would be rather simple to move this into a custom association, containing the association will look up a matching table, cause the schema to be inspected, etc.
So I'll stick with providing an example for the first option. A result formatter would be pretty simple, something like this:
$this->Articles
->find()
->formatResults(function (\Cake\Collection\CollectionInterface $results) {
$userIds = array_unique($results->extract('user_id')->toArray());
$users = LDAPConnector::getUsers($userIds);
$usersMap = collection($users)->indexBy('id')->toArray();
return $results
->map(function ($article) use ($usersMap) {
if (isset($usersMap[$article['user_id']])) {
$article['user'] = $usersMap[$article['user_id']];
}
return $article;
});
});
The example makes the assumption that the data returned from LDAPConnector::getUsers() is a collection of associative arrays, with an id key that matches the user id. You'd have to adapt this accordingly, depending on what exactly LDAPConnector::getUsers() returns.
That aside, the example should be rather self-explanatory, first obtain a unique list of users IDs found in the queried articles, obtain the LDAP users using those IDs, then inject the users into the articles.
If you wanted to have entities in your results, then create entities from the user data, for example like this:
$userData = $usersMap[$article['user_id']];
$article['user'] = new \App\Model\Entity\User($userData);
For better reusability, put the formatter in a custom finder. In your ArticlesTable class:
public function findWithUsers(\Cake\ORM\Query $query, array $options)
{
return $query->formatResults(/* ... */);
}
Then you can just do $this->Articles->find('withUsers'), just as simple as containing.
See also
Cookbook > Database Access & ORM > Query Builder > Adding Calculated Fields
Cookbook > Database Access & ORM > Retrieving Data & Results Sets > Custom Finder Methods
I have a case that I am curious about for a long, long time. Say I have 3 models:
class Product(models.Model):
manufacturer = models.CharField(max_length=100)
description = models.TextField(blank=True)
class PurchasedProduct(models.Model):
product = models.ForeignKey(Product)
purchase = models.ForeignKey('Purchase')
quantity = models.PositiveIntegerField()
class Purchase(models.Model):
customer = models.ForeignKey('customers.Customer')
products = models.ManyToManyField(Product, through=PurchasedProduct)
comment = models.CharField(max_length=200)
I have an API and client application written in some JavaScript framework. So now I need to communicate between them! I am not sure how should I handle this situation in DRF, naturally I would expect to get something like this when accessing /purchase/1/
{
"id": 1,
"customer": 1,
"comment": "Foobar",
"products": [
{
"id": 1,
"product": {
"id": 1,
....
},
....
},
....
]
}
So I created proper serializer specifying that products field should use PurchasedProductSerializer which in turn uses nested ProductSerializer. It is fine cause I get all necessary info to, say, display what specific products where purchased and in what quantity during shopping using appropriate components in say React.
The problem for me is however when I need to POST new PurchasedProduct. I would expect the most convenient form to be:
{
"quantity": 10,
"purchase": 1,
"product": 1
}
As it carries all necessary info and has the smallest footprint. However I can't be accomplished using PurchasedProductSerializer as it requires product to be object instead of id.
So my question here, is this a good approach (it seems very natural to me), should I use two separate serializers for GET and POST? Should I perform this differently? Could you point me to some best practices/books how to write APIs and client apps?
I had the exact same problem a few months back and would've been more than happy if someone would've told me. I ended up with the exact solution that you proposed to add products to a purchase. I do agree that your proposed POST request it is the most natural way with the minimal required footprint.
In order to correctly process the POST request's data correctly, though, I ended up using two separate Serializers, just as you described. If you're using DRF viewsets, one way to select the correct serializer on GET and POST is to override the get_serializer_class method as described here.
The deserializer for POST requests could look like this:
class PurchasedProductDeserializer(serializers.ModelSerializer):
product = serializers.PrimaryKeyRelatedField(queryset=Product.objects.all())
purchase = serializers.PrimaryKeyRelatedField(queryset=Purchase.objects.all())
class Meta:
model = PurchasedProduct
fields = ('id', 'product', 'purchase', 'quantity')
write_only_fields = ('product', 'purchase', 'quantity')
That deserializer can then be used for input validation and finally to add a product to a purchase (or increase its quantity).
E.g., inside your viewset:
def create(self, request, *args, **kwargs):
# ...
# init your serializer here
serializer = self.get_serializer(data=request.data)
if serializer.is_valid(raise_exception=True):
# now check if the same item is already in the cart
try:
# try to find the product in the list of purchased products
purchased_product = serializer.validated_data['purchase'].purchasedproduct_set.get(product=serializer.validated_data['product'])
# if so, simply increase its quantity, else add the product as a new item to the cart (see except case)
purchased_product.quantity += serializer.validated_data['quantity']
purchased_product.save()
# update the serializer so it knows the id of the existing instance
serializer.instance = purchased_product
except PurchasedProduct.DoesNotExist:
# product is not yet part of the purchase cart, add it now
self.perform_create(serializer)
# ...
# do other stuff here
As for best practices, there's a ton of documentation available on the internet, but if you're looking for books you might wanna look at some of the ones posted here. When you get bored of REST you might even wanna look into GraphQL.
I am trying to set up cakephp to work with the very nice javascriptMVC (http://forum.javascriptmvc.com). JavaScriptMVC requires the JSON-Output in the following format:
[{
'id': 1,
'name' : 'Justin Meyer',
'birthday': '1982-10-20'
},
{
'id': 2,
'name' : 'Brian Moschel',
'birthday': '1983-11-10'
}]
Cake would generate a deeper nested array with a prepended Class Name. I found attempts to solve the problem but theyre not for cakephp 2.x. I know that I can simply generate a new array and json_encode() it via php, but it would be nicer to include a function like this https://gist.github.com/1874366 and another one to deflatten it.
Where would be the best place to put such functions? The AppController doesnt seem to work. Should i put it in beforeRender () or beforeFilter() of the controller? Or does someone maybe even know of an existing solution/plugin for this? This would be the best for me in my current Situation, as Im pretty much pressed for time.
Ok, I'm not 100% sure I understand what you are trying to do so here's a word to the wise just in case: Cake and JMVC are both comprehensive MVC frameworks. if you are attempting to combine them as a single cohesive platform to build your application, I strongly suggest you review your approach / platform / etc.
Also -- I'm not an expert by any means in jmvc, so I'm just going to pretend that processing the response from Cake in jmvc is completely out of the question, for some odd reason. For the record, think of Cake's responses like this:
{ "Model" :
[{
'id': 1,
'name' : 'Justin Meyer',
'birthday': '1982-10-20'
},
{
'id': 2,
'name' : 'Brian Moschel',
'birthday': '1983-11-10'
}]
}
Cake has had comprehensive REST service support, since at least Cake 1.2. The lib you are interested in is HttpSocket. As for json encoding and serving response, Request Handling covers, among other things, responding to all manners of requests, content types, decoding and encoding json, etc. Finally, the built-in Set utility will almost certainly cover whatever array manipulation you need in a line or two.
The functionality you are interested in is pretty basic and hasn't changed too much. I'd bet a lot of the (reasonably simple) solutions you have already found would probably still work, maybe with a little bit of tweaking.
For pretty much any basic service endpoint, you would probably create a controller (not AppController - that is application-wide, hence you can't invoke it directly) method, considering Cake routes the controller/action into your url:
Cake consuming services from a different app would look like this:
http://cakeproject/collect/getInfo
class CollectController extends AppController {
public function getInfo($array = null) {
App::uses('HttpSocket', 'Network/Http');
$http = new HttpSocket();
$http->get('http://jmvcproject/controller/action', $array);
// ...etc.
}
Cake providing services from the same controller / action to a different app would simply be:
public function getInfo($array = null) {
$results = $this->Collect->find('all', $array);
// ...fetch the results
}
Or you could just loop over that array with foreach($this->data as $data) { ... to drop the class name. But if your data will include associated models, etc, Set is probably the most versatile and resilient solution.
Anyway, HTH
django nonrel's documentation states: "you have to manually write code for merging the results of multiple queries (JOINs, select_related(), etc.)".
Can someone point me to any snippets that manually add the related data? #nickjohnson has an excellent post showing how to do this with the straight AppEngine models, but I'm using django-nonrel.
For my particular use I'm trying to get the UserProfiles with their related User models. This should be just two simple queries, then match the data.
However, using django-nonrel, a new query gets fired off for each result in the queryset. How can I get access to the related items in a 'select_related' sort of way?
I've tried this, but it doesn't seem to work as I'd expect. Looking at the rpc stats, it still seems to be firing a query for each item displayed.
all_profiles = UserProfile.objects.all()
user_pks = set()
for profile in all_profiles:
user_pks.add(profile.user_id) # a way to access the pk without triggering the query
users = User.objects.filter(pk__in=user_pks)
for profile in all_profiles:
profile.user = get_matching_model(profile.user_id, users)
def get_matching_model(key, queryset):
"""Generator expression to get the next match for a given key"""
try:
return (model for model in queryset if model.pk == key).next()
except StopIteration:
return None
UPDATE:
Ick... I figured out what my issue was.
I was trying to improve the efficiency of the changelist_view in the django admin. It seemed that the select_related logic above was still producing additional queries for each row in the results set when a foreign key was in my 'display_list'. However, I traced it down to something different. The above logic does not produce multiple queries (but if you more closely mimic Nick Johnson's way it will look a lot prettier).
The issue is that in django.contrib.admin.views.main on line 117 inside the ChangeList method there is the following code: result_list = self.query_set._clone(). So, even though I was properly overriding the queryset in the admin and selecting the related stuff, this method was triggering a clone of the queryset which does NOT keep the attributes on the model that I had added for my 'select related', resulting in an even more inefficient page load than when I started.
Not sure what to do about it yet, but the code that selects related stuff is just fine.
I don't like answering my own question, but the answer might help others.
Here is my solution that will get related items on a queryset based entirely on Nick Johnson's solution linked above.
from collections import defaultdict
def get_with_related(queryset, *attrs):
"""
Adds related attributes to a queryset in a more efficient way
than simply triggering the new query on access at runtime.
attrs must be valid either foreign keys or one to one fields on the queryset model
"""
# Makes a list of the entity and related attribute to grab for all possibilities
fields = [(model, attr) for model in queryset for attr in attrs]
# we'll need to make one query for each related attribute because
# I don't know how to get everything at once. So, we make a list
# of the attribute to fetch and pks to fetch.
ref_keys = defaultdict(list)
for model, attr in fields:
ref_keys[attr].append(get_value_for_datastore(model, attr))
# now make the actual queries for each attribute and store the results
# in a dict of {pk: model} for easy matching later
ref_models = {}
for attr, pk_vals in ref_keys.items():
related_queryset = queryset.model._meta.get_field(attr).rel.to.objects.filter(pk__in=set(pk_vals))
ref_models[attr] = dict((x.pk, x) for x in related_queryset)
# Finally put related items on their models
for model, attr in fields:
setattr(model, attr, ref_models[attr].get(get_value_for_datastore(model, attr)))
return queryset
def get_value_for_datastore(model, attr):
"""
Django's foreign key fields all have attributes 'field_id' where
you can access the pk of the related field without grabbing the
actual value.
"""
return getattr(model, attr + '_id')
To be able to modify the queryset on the admin to make use of the select related we have to jump through a couple hoops. Here is what I've done. The only thing changed on the 'get_results' method of the 'AppEngineRelatedChangeList' is that I removed the self.query_set._clone() and just used self.query_set instead.
class UserProfileAdmin(admin.ModelAdmin):
list_display = ('username', 'user', 'paid')
select_related_fields = ['user']
def get_changelist(self, request, **kwargs):
return AppEngineRelatedChangeList
class AppEngineRelatedChangeList(ChangeList):
def get_query_set(self):
qs = super(AppEngineRelatedChangeList, self).get_query_set()
related_fields = getattr(self.model_admin, 'select_related_fields', [])
return get_with_related(qs, *related_fields)
def get_results(self, request):
paginator = self.model_admin.get_paginator(request, self.query_set, self.list_per_page)
# Get the number of objects, with admin filters applied.
result_count = paginator.count
# Get the total number of objects, with no admin filters applied.
# Perform a slight optimization: Check to see whether any filters were
# given. If not, use paginator.hits to calculate the number of objects,
# because we've already done paginator.hits and the value is cached.
if not self.query_set.query.where:
full_result_count = result_count
else:
full_result_count = self.root_query_set.count()
can_show_all = result_count self.list_per_page
# Get the list of objects to display on this page.
if (self.show_all and can_show_all) or not multi_page:
result_list = self.query_set
else:
try:
result_list = paginator.page(self.page_num+1).object_list
except InvalidPage:
raise IncorrectLookupParameters
self.result_count = result_count
self.full_result_count = full_result_count
self.result_list = result_list
self.can_show_all = can_show_all
self.multi_page = multi_page
self.paginator = paginator
I created a Django app that had its own internal voting system and a model called Vote to track it. I want to refactor the voting system into its own app so I can reuse it. However, the original app is in production and I need to create a data migration that will take all the Votes and transplant them into the separate app.
How can I get two apps to participate in a migration so that I have access to both their models? Unfortunately, the original and separate apps both have a model named Vote now, so I need to be aware of any conflicts.
Have you tried db.rename_table?
I would start by creating a migration in either the new or old app that looks something like this.
class Migration:
def forwards(self, orm):
db.rename_table('old_vote', 'new_vote')
def backwards(self, orm):
db.rename_table('new_vote', 'old_vote')
If that does not work you can migrate each item in a loop with something along these lines:
def forwards(self, orm):
for old in orm['old.vote'].objects.all():
# create a new.Vote with old's data
models = {
'old.vote' = { ... },
'new.vote' = { ... },
}
Note: You must use orm[...] to access any models outside the app currently being migrated. Otherwise, standard orm.Vote.objects.all() notation works.