Cakephp 3 - How to integrate external sources in table? - cakephp

I working on an application that has its own database and gets user information from another serivce (an LDAP is this case, through an API package).
Say I have a tables called Articles, with a column user_id. There is no Users table, instead a user or set of users is retrieved through the external API:
$user = LDAPConnector::getUser($user_id);
$users = LDAPConnector::getUsers([1, 2, 5, 6]);
Of course I want retrieving data from inside a controller to be as simple as possible, ideally still with something like:
$articles = $this->Articles->find()->contain('Users');
foreach ($articles as $article) {
echo $article->user->getFullname();
}
I'm not sure how to approach this.
Where should I place the code in the table object to allow integration with the external API?
And as a bonus question: How to minimise the number of LDAP queries when filling the Entities?
i.e. it seems to be a lot faster by first retrieving the relevant users with a single ->getUsers() and placing them later, even though iterating over the articles and using multiple ->getUser() might be simpler.

The most simple solution would be to use a result formatter to fetch and inject the external data.
The more sophisticated solution would a custom association, and a custom association loader, but given how database-centric associations are, you'd probably also have to come up with a table and possibly a query implementation that handles your LDAP datasource. While it would be rather simple to move this into a custom association, containing the association will look up a matching table, cause the schema to be inspected, etc.
So I'll stick with providing an example for the first option. A result formatter would be pretty simple, something like this:
$this->Articles
->find()
->formatResults(function (\Cake\Collection\CollectionInterface $results) {
$userIds = array_unique($results->extract('user_id')->toArray());
$users = LDAPConnector::getUsers($userIds);
$usersMap = collection($users)->indexBy('id')->toArray();
return $results
->map(function ($article) use ($usersMap) {
if (isset($usersMap[$article['user_id']])) {
$article['user'] = $usersMap[$article['user_id']];
}
return $article;
});
});
The example makes the assumption that the data returned from LDAPConnector::getUsers() is a collection of associative arrays, with an id key that matches the user id. You'd have to adapt this accordingly, depending on what exactly LDAPConnector::getUsers() returns.
That aside, the example should be rather self-explanatory, first obtain a unique list of users IDs found in the queried articles, obtain the LDAP users using those IDs, then inject the users into the articles.
If you wanted to have entities in your results, then create entities from the user data, for example like this:
$userData = $usersMap[$article['user_id']];
$article['user'] = new \App\Model\Entity\User($userData);
For better reusability, put the formatter in a custom finder. In your ArticlesTable class:
public function findWithUsers(\Cake\ORM\Query $query, array $options)
{
return $query->formatResults(/* ... */);
}
Then you can just do $this->Articles->find('withUsers'), just as simple as containing.
See also
Cookbook > Database Access & ORM > Query Builder > Adding Calculated Fields
Cookbook > Database Access & ORM > Retrieving Data & Results Sets > Custom Finder Methods

Related

Yii2 + Redis as Database

I want to use Yii2 and redis as database.
So far, I got Redis ActiveRecord Class for Yii2 from Here.
link1
link2
but, I got a problem. WHY THIS CLASS ADDS ANYTHING AS HASH IN REDIS????
Above that I cant Find in which pattern it Insert data. I add one user and it will add a user under user:xxx namespace and another record under s:user:xxx and so on but none of theme has any fields that i defined in attributes!! only contain IDs.
I know that a Key-value type database and RDBMS are different and also know how can implement relation like records in Redis, but I don't know why it will only save IDs.
I could not find any example of using redis ActiveRecords so far.
There is one in here and its not good enough.
So here is my main wuestion: how can add data to redis Using activeRecords and different data types In YII2?
And if its impossible with ActiveRecords what is the best solution? in this case
ANOTHER QUESTION: is it possible to use a Model instead and write my own model::save() method? and what is the best data validation solution at this rate?
Actually I want to make a telegram bot, so i should get messages and send them in RabitMQ and get data in a worker, do the process and save results to Redis, and finally send response to user through the RabitMQ.
So I need to do a lot of validations AND OF COURSE AUTHENTICATIONS and save and select and range and save to sets an lists and this and that ....
I want a good way to make Model or active record or the proper solution to validation, save and retrieve data to Redis and Yii2.
Redis DB can be declared as a cache component or as a database connection or both.
When it is declared as a cache component (using the yii/redis/cache) it is accessible within that component to store key/value pairs as shown here.
$cache = Yii::$app->cache;
// try retrieving $data from cache
$data = $cache->get($key);
// store $data in cache so that it can be retrieved next time
$cache->set($key, $data);
// one more example:
$access_token = Yii::$app->security->generateRandomString();
$cache->add(
// key
$access_token,
// data (can also be an array)
[
'id' => Yii::$app->user->identity->id
'name' => Yii::$app->user->identity->name
],
// expires
60*60*3
);
Also other components may start using it for caching proposes like session if configured to do so or like the yii\web\UrlManager which by default will try to cache the generated URL rules in whatever valid caching mechanism defined under the config file's cache component as explained here. So it is normal to find some stored data other than yours in that case.
When Redis is declared as a DB connection like in the links you provided which means using the yii\redis\Connection class you can make your model extending its \yii\redis\ActiveRecord class as any other ActiveRecord model in Yii. The only difference I know so far is that you need to define your attributes manually as there is no DB schema to parse for NoSQL databases. Then just define your rules, scenarios, relations, events, ... as any other ActiveRecord model:
class Customer extends \yii\redis\ActiveRecord
{
public function attributes()
{
return ['id', 'name', 'address', 'registration_date'];
}
public function rules()
{
return [
['name', 'required'],
['name', 'string', 'min' => 3, 'max' => 12, 'on' => 'register'],
...
];
}
public function attributeLabels() {...}
...
}
All available methods including save(), validate(), getErrors(), ... could be found here and should be used like any other ActiveRecord class as shown in the official guide.

How does one get properties from related table as properties of it's own table in Laravel 5?

The question might sound a little bit confusing but I don't know how to explain it better in one sentence.
This describes basically what the problem is:
I have a Users table which can contain 2 types of users. I know how I can separate by role. But here's the thing, users with role 1(editor_in_chief) have different attributes than users with role 2(reviewer).
My idea was to create a table named 'reviewer_attributes' and 'editor_in_chief_attributes' and create a one-to-one relation with this table to hold the attributes for the users table.
Maybe you have a better idea, that would be great as well. But for this scenario, I would like to know if it is possible to make a call to the database and to get these users' properties from the other table as properties of the User object.
When using a DB call using relations laravel will give me something like this:
user {
id: 1,
name: "Name",
reviewer_attributes: {
attribute_1: 'attribute_1',
attribute_2: 'attribute_2',
attribute_3: 'attribute_3',
}
}
But this is what I want to object to obtain look like:
user {
id: 1,
name: "Name",
attribute_1: 'attribute_1',
attribute_2: 'attribute_2',
attribute_3: 'attribute_3',
}
I want to achieve this by a single database call instead of setting the properties after the call.
I find this a very interesting topic, I hope you could help!
If I got your problem right, you may call somthing like this:
DB::table('users')
->join('reviewer_attributes', 'users.id', '=', 'reviewer_attributes.user_id')
->find($id);
you may add select to get specific attributes of each table:
DB::table('users')
->join('reviewer_attributes', 'users.id', '=', 'reviewer_attributes.user_id')
->select('users.id', 'users.name', 'reviewer_attributes.*')
->find($id);
Update: You can also use collections to restructure your results returned by Eloquent:
$result = User::with('reviewerAttributes')->find($id);
$result = $result->get('reviewer_attributes')
->merge($result->forget('reviewer_attributes')->all())
->all();
You need export model to Json?
If so, override toArray method
public function toArray()
{
$arr = parent::toArray();
$reviewer_attributes = $this->getReviewerAttributesSomeHow();
return array_merge($arr, $reviewer_attributes);
}

How can I mimic 'select_related' using google-appengine and django-nonrel?

django nonrel's documentation states: "you have to manually write code for merging the results of multiple queries (JOINs, select_related(), etc.)".
Can someone point me to any snippets that manually add the related data? #nickjohnson has an excellent post showing how to do this with the straight AppEngine models, but I'm using django-nonrel.
For my particular use I'm trying to get the UserProfiles with their related User models. This should be just two simple queries, then match the data.
However, using django-nonrel, a new query gets fired off for each result in the queryset. How can I get access to the related items in a 'select_related' sort of way?
I've tried this, but it doesn't seem to work as I'd expect. Looking at the rpc stats, it still seems to be firing a query for each item displayed.
all_profiles = UserProfile.objects.all()
user_pks = set()
for profile in all_profiles:
user_pks.add(profile.user_id) # a way to access the pk without triggering the query
users = User.objects.filter(pk__in=user_pks)
for profile in all_profiles:
profile.user = get_matching_model(profile.user_id, users)
def get_matching_model(key, queryset):
"""Generator expression to get the next match for a given key"""
try:
return (model for model in queryset if model.pk == key).next()
except StopIteration:
return None
UPDATE:
Ick... I figured out what my issue was.
I was trying to improve the efficiency of the changelist_view in the django admin. It seemed that the select_related logic above was still producing additional queries for each row in the results set when a foreign key was in my 'display_list'. However, I traced it down to something different. The above logic does not produce multiple queries (but if you more closely mimic Nick Johnson's way it will look a lot prettier).
The issue is that in django.contrib.admin.views.main on line 117 inside the ChangeList method there is the following code: result_list = self.query_set._clone(). So, even though I was properly overriding the queryset in the admin and selecting the related stuff, this method was triggering a clone of the queryset which does NOT keep the attributes on the model that I had added for my 'select related', resulting in an even more inefficient page load than when I started.
Not sure what to do about it yet, but the code that selects related stuff is just fine.
I don't like answering my own question, but the answer might help others.
Here is my solution that will get related items on a queryset based entirely on Nick Johnson's solution linked above.
from collections import defaultdict
def get_with_related(queryset, *attrs):
"""
Adds related attributes to a queryset in a more efficient way
than simply triggering the new query on access at runtime.
attrs must be valid either foreign keys or one to one fields on the queryset model
"""
# Makes a list of the entity and related attribute to grab for all possibilities
fields = [(model, attr) for model in queryset for attr in attrs]
# we'll need to make one query for each related attribute because
# I don't know how to get everything at once. So, we make a list
# of the attribute to fetch and pks to fetch.
ref_keys = defaultdict(list)
for model, attr in fields:
ref_keys[attr].append(get_value_for_datastore(model, attr))
# now make the actual queries for each attribute and store the results
# in a dict of {pk: model} for easy matching later
ref_models = {}
for attr, pk_vals in ref_keys.items():
related_queryset = queryset.model._meta.get_field(attr).rel.to.objects.filter(pk__in=set(pk_vals))
ref_models[attr] = dict((x.pk, x) for x in related_queryset)
# Finally put related items on their models
for model, attr in fields:
setattr(model, attr, ref_models[attr].get(get_value_for_datastore(model, attr)))
return queryset
def get_value_for_datastore(model, attr):
"""
Django's foreign key fields all have attributes 'field_id' where
you can access the pk of the related field without grabbing the
actual value.
"""
return getattr(model, attr + '_id')
To be able to modify the queryset on the admin to make use of the select related we have to jump through a couple hoops. Here is what I've done. The only thing changed on the 'get_results' method of the 'AppEngineRelatedChangeList' is that I removed the self.query_set._clone() and just used self.query_set instead.
class UserProfileAdmin(admin.ModelAdmin):
list_display = ('username', 'user', 'paid')
select_related_fields = ['user']
def get_changelist(self, request, **kwargs):
return AppEngineRelatedChangeList
class AppEngineRelatedChangeList(ChangeList):
def get_query_set(self):
qs = super(AppEngineRelatedChangeList, self).get_query_set()
related_fields = getattr(self.model_admin, 'select_related_fields', [])
return get_with_related(qs, *related_fields)
def get_results(self, request):
paginator = self.model_admin.get_paginator(request, self.query_set, self.list_per_page)
# Get the number of objects, with admin filters applied.
result_count = paginator.count
# Get the total number of objects, with no admin filters applied.
# Perform a slight optimization: Check to see whether any filters were
# given. If not, use paginator.hits to calculate the number of objects,
# because we've already done paginator.hits and the value is cached.
if not self.query_set.query.where:
full_result_count = result_count
else:
full_result_count = self.root_query_set.count()
can_show_all = result_count self.list_per_page
# Get the list of objects to display on this page.
if (self.show_all and can_show_all) or not multi_page:
result_list = self.query_set
else:
try:
result_list = paginator.page(self.page_num+1).object_list
except InvalidPage:
raise IncorrectLookupParameters
self.result_count = result_count
self.full_result_count = full_result_count
self.result_list = result_list
self.can_show_all = can_show_all
self.multi_page = multi_page
self.paginator = paginator

OOP, MVC - Models and Objects

I'm building a site with CodeIgniter, and my I have a model called Blog_model.
Within Blog_model, there are methods to pull a list of posts for a specific topic, for example, getPopularPosts().
getPopularPosts() queries the posts table for a list of posts with a topic_id matching the one specified and sorts them by popularity. So, that's one query against the entire table of posts (let's assume this will be very large eventually) to find all posts with topic_id x.
Then, foreach result as an individual post id, it creates a new Post object. The Post class constructs a post by setting the field id.
To return the contents of a Post, I assign $post->getPost();, which queries the posts table again to return the entire row for the given id.
This organization (AFAIK) follows a nice object oriented principle. But now, for every posts (again, let's assume thousands, millions, whatever...), I have to first query for a list of ids and then again to get each post's content. If I'm returning 30 posts, that means 31 separate queries.
Alternatively, I could break the object oriented pattern and pull * for each post in posts where topic_id = x. Then, I have one query that returns all 30 posts, but now I don't feel so object oriented.
What to do?
There is no reason to have that many queries. You're basically just looking for X number of posts that are from a particular topic ID... you should return this as one object and then iterate through the result in PHP because it is significantly faster to do it that way once you get to the point of having millions of rows
You should go about it more like this:
class blog_model extends CI_Model {
function __construct(){
parent::__construct();
}
function getPopularPosts($cat_id){
/* Using method chaining here since you sound like you
really want to utilize everything OO CI has to offer */
$posts = $this->db->select('id, title, post_info')
->where('topic_id', $topic_id)
->get('posts');
if($posts->num_rows() > 0){
return $posts;
}else{
return FALSE;
}
}
}
Then your controller would look like this:
class blog extends CI_Controller {
function __construct() {
parent::__construct();
}
function blog_posts($popular_post_id) {
$this->load->model('blog_model');
$posts = $this->blog_model->getPopularPosts($popular_post_id);
if(!empty($posts){
foreach($posts as $post){
echo $post->id;
echo $post->title;
echo $post->post_info;
}
}else{
echo 'There are no posts';
}
}
}
There is no benefit (and actually a big problem) with generating a ton of queries in the fashion that you currently have it set up, vs generating one object from the query and iterating through each of the rows in the controller and doing whatever you need with the data.

Zend Many to Many Relationship

I want to retrieve all the data from 3 tables
users , properties and users_properties.
So I decided I would use the manytomanyRowset. But to my surprise I get the data from the properties and users_properties table but no data from the users table. Why is that? I need some columns from the users table is there a way to tell the manytomanyrowset function that I need the data from the current table as well?
this is my function
public function fetchRegisteredProperties()
{
$userTable = $this->getTable();
require_once APPLICATION_PATH . '/models/DbTable/UsersPropertiesDB.php';
require_once APPLICATION_PATH . '/models/DbTable/PropertiesDB.php';
$propertiesRowset = $table->fetchAll();
$allProperties = array();
foreach ($propertiesRowset as $row) {
$propertiesRowset = $row->findManyToManyRowset(
'Model_DbTable_Properties','Model_DbTable_UsersProperties');
$allProperties = array_merge($tempArray,$propertiesRowset->toArray());
}
return $allProperties;
}
thanks in adavance
I designed and coded the table-relationships features in Zend Framework.
The answer to your question is no, the findManyToManyRowset() method only fetches rows from the related table, it does not merge them into the corresponding Row object. The reason is that a Row object in ZF can save() itself back to the database, and if you add fields it won't know what to do with them.
So you should implement a custom Row object to hold both user fields and the collection of user properties -- store the user properties as a Rowset object.
Then extend __get() and __set() so that it knows how to map fields into the correct array when you read or write object properties. That is, if one tries to read or write a field that isn't part of the user row, it falls back to the user properties Rowset.
Also extend save() to save not only the current row, but also call save() on the Rowset of user properties.

Resources