I have a cakephp application, 2.4, and I'm having issues with the Paginator component. First off, it's not the database, it's definitely the execution of parsing the query results. I have DebugKit installed and can see that my mysql query for the paginated data takes a whole 2 ms. The table has 2.5 million records of messages, and 500,000 users. Obviously proper indexing is in place. But, the controller action is taking 6167.82 ms. So, here's my controller action:
$this->Paginator->settings = array(
'Message' => array(
'fields' => array(
'Recipient.username',
'Recipient.profile_photo',
'Recipient.id',
'Message.*'
),
'joins' => array(array(
'table' => 'users',
'alias' => 'Recipient',
'type' => 'LEFT',
'conditions' => array(
'Recipient.id = `Message`.`recipient_id`'
)
)),
'conditions' => array(
'Message.sender_id' => $this->Auth->user('id'),
'Message.deleted_by_sender' => '0'
),
'limit' => 10,
'order' => 'Message.id DESC',
'recursive' => -1
)
);
$sents = $this->Paginator->paginate( 'Message' );
$this->set( 'sents', $sents );
$this->view = 'index';
I've google this and searched stack overflow. The majority of the responses are for poor mysql optimization which isn't my case. The other half of the responses suggest containable. So, I tried containable. Using contain was actually slower because it tried to grab even more data from the user's field than just the username, photo, and id. Then when cake built the array from the query results it executed nearly 500 ms slower with containable because of the extra user data I'm assuming.
I'm going to now dig into the cake Paginator component and see why it's taking so long to build the response. I'm hoping someone beats me to it and has a good solution to help speed this up.
My web server is running ubuntu 12.04 with 3gb ram, apache and mod_php with apc installed and working for the model and core cache. The database is on a separate server. I also have a redis server persisting other user data and the cake session data. There is plenty of power here to parse 10 records from a mysql query containing about a dozen rows.
EDIT: ANSWER
As suggested first by Ilie Pandia there was something else happening, such as a callback, that was slowing down the pagination. This was actually unrelated to the pagination component. The Recipient model had a behavior that loaded an sdk in the setup callback for a 3rd party service. That service was taking several seconds to respond. This happened when the linkedModel in the query was loaded to filter the results. Hopefully anyone else looking for reasons why cake might be performing poorly will also look at the callbacks on models in the application and plugins.
I see no reason for this to run slow at all.
So this suggests that there are some callback installed (either in the model or the controller) that do additional processing and inflate the action time so much.
That is assuming that there is nothing else in the controller but what your wrote.
You could actually measure the time of the paginate call itself and I think you will find that it is very fast. So the bottle neck is elsewhere in the code.
PS: You could also try to disable DebugKit for a while. Introspection may take very long for some particular cases.
Install DebugKit for your application.
And inspect which query is taking too much time. From there, you should be able to track the bottleneck.
Related
I have an issue with huge WordPress database to find users. I have about 147000 user into my database and 75000 users have same category. When i try to search users by a category for example "Doctors" then Query fails and page got stuck
I need $total_users from WP_User_Query to make pagination but it seems that query fail.
I also tried this WP_User_Query unable to retrieve all user at once
but page load time is still too much.
$query_args = array(
'role' => 'professional',
'order' => 'DESC',
'orderby' => 'ID',
);
Can anyone there to help me to get this resolve?
here is the link : Site link
Thanks
I've set up web services using Drupal's services module. It outputs JSON for me which I am requesting through a Backbone.js front-end application.
I'm having issues with this set-up. If I request data through Backbone.js' fetch method of a model, the model's attributes are all typed as string after fetching, while there are some attributes that should be e.g. integer.
For example:
I have enabled the user resource, which is standard available in the Drupal services module
I can request a user, e.g.:
http://mydevmachine/services/user/8
...which results in the following response (slimmed down version from the real response):
{"uid":"8","name":"itsme","mail":"me#mydomain.nl"}
What I see in the response from the web service above, all values are quoted, however uid is really not a string but an integer in the database.
If I fetch the same user in my Backbone.js model, by setting the uid field of my model to 8 (integer), then call the fetch method. After fetching the uid field is typed as 'string'.
I assume the above leads to my model ending up with a uid attribute of not integer, but string. It also happens with all other web service resources I have created, using my own entities.
I need correct typing of attributes in my model due to sorting issues using Backbone's collection sorting. I.e. sorting a collection of models using a field of type 'integer' leads to different sorting results when sorting the field with the same values although stored as a string.
I'm not sure exactly where to look:
Is the JSON format output by the Drupal services module according to standards?
Is the JSON output format configurable or overridable in the Drupal services module?
Is it perhaps possible to keep the type of a model's attribute after a fetch in Backbone.js?
Should I provide a specific implementation for Backbone's collection comparator function, which handles this situation (seems hackey)?
Should I introduce other solutions, e.g. like posted here: How can I enforce attribute types in a Backbone model? (feels too heavy).
Thanks for any help.
So I finally managed to crack this issue and I found my solution here: How to get numeric types from MySQL using PDO?. I thought I'd document the solution.
Drupal 7 uses PDO. Results fetched using PDO, using Drupal's default PDO settings result in stringified values.
In Drupal's includes/database.inc file you will find this around lines 40-50:
$connection_options['pdo'] += array(
// So we don't have to mess around with cursors and unbuffered queries by default.
PDO::MYSQL_ATTR_USE_BUFFERED_QUERY => TRUE,
// Because MySQL's prepared statements skip the query cache, because it's dumb.
PDO::ATTR_EMULATE_PREPARES => TRUE,
);
The statement here that MySQL's prepared statements skip the query cache is not entirely true, as can be found here: http://dev.mysql.com/doc/refman/5.1/en/query-cache-operation.html. It states MySQL > 5.1.17 prepared statements use the query cache under certain conditions.
I used the info from the other stack overflow question/answers to override the PDO settings for the database connection in Drupal's sites/default/settings.php (please note I only did this for the database I was querying, which is different than Drupal's own database):
'database_name' =>
array (
'default' =>
array (
'database' => 'database_name',
'username' => 'user_name',
'password' => 'user_pass',
'host' => 'localhost',
'port' => '',
'driver' => 'mysql',
'prefix' => '',
'pdo' => array(
PDO::ATTR_STRINGIFY_FETCHES => FALSE,
PDO::ATTR_EMULATE_PREPARES => FALSE
),
),
),
This resulted in integers being integers. Floats/decimals are incorrectly returned by PDO still, but this is different issue. At least my problems are solved now.
I'm using cake 2.1.3 and currently have a page that is getting hundreds of views per second and so I have utelized caching in order to handle the load better. The problem is, that once the cache expires, I get a spike in my server resources as well as hundreds of mysql connections.
I'm wondering if I'm going about this the wrong way and if I should be running a cron to cache the page instead of how I'm currently doing it or if there's another technique I'm not thinking of.
here's what my function looks like in my controller:
public function index() {
$this->layout = 'ajax';
if (isset($this->params['url']['callback'])) {
$callback = $this->params['url']['callback'];
}else{
$callback = 'callback';
}
$this->set('callback',$callback);
$today = date("Y-m-d");
$end_date = strtotime ('+1 day' , strtotime($today)) ;
$end_date = date ( 'Y-m-d' , $end_date);
$start_date = strtotime ('-1 day' , strtotime($today)) ;
$start_date = date ( 'Y-m-d' , $start_date);
$total = Cache::read('popular_stories', 'short');
if (!$total) {
$total = $this->TrackStoryView->find('all', array(
'fields' => array('COUNT(story_id) AS theCount', 'headline', 'url'),
'conditions' => array('date BETWEEN ? AND ?' => array($start_date,$end_date)),
'group' => 'story_id',
'order' => array('theCount DESC'),
'limit' => 20,
));
Cache::write('popular_stories', $total, 'short');
}
$this->set('story', $total);
}
Here's what my Cache config looks like in my bootstrap.php file:
Cache::config('short', array(
'engine' => 'File',
'duration' => '+60 minutes',
'path' => CACHE,
'prefix' => 'cake_short_'
));
This is what's in my view file:
<?php
echo $callback . '('.json_encode($story).')';
?>
I was hoping that once the cached file expires, as soon as the first person accessed it, it would craete a new cached file and serve that up for everyone, however because hundreds of people are hitting it per second, it seems like this method isn't working for me and that maybe I should be caching the view view a cron somehow instead or maybe there's a different way to cache that I'm not utelizing.
It sounds like you have the answer more or less figured out (create the cache automatically, not triggered by a user request).
To do this, look into cake's AppShell class, book talks about it here. You can then link this to a cron job. If you create the file thru Cache::write, cake should be aware that it is a new cache file and read it transparently. You might want to leave the "if cache not found" block in there just in case your cronjob fails.
Shells & Tasks in cake are fun and allow you to free your application from using the request/response model exclusively.
TLDR: It's not ideal to force a user to break the cache for you. Use a chron job or a trigger on data change.
Explanation:
"hundreds of views per second" is the problem. When it expires, there are "hundreds of views" during the time it's trying to create the cache file.
The first person hits it, it starts creating the cache, and in the meantime, another hundred+ people hit it, and it looks, and can't yet find a cache file...etc.
If you can manage, try to manually create the cache when an item(s) is updated, or run a chron job that creates a new cache every X minutes as opposed to having it create for a user.
Cake has lots of cool triggers like afterSave() that you can use to trigger this kind of thing. If that doesn't make sense in your case though, a chron job should be fine for you.
I think the answer lies by working out how long this query takes:
$total = $this->TrackStoryView->find('all', array(
'fields' => array('COUNT(story_id) AS theCount', 'headline', 'url'),
'conditions' => array('date BETWEEN ? AND ?' => array($start_date,$end_date)),
'group' => 'story_id',
'order' => array('theCount DESC'),
'limit' => 20,
));
Lets say it takes 500ms.
You are getting 100 hits a second, so when the cache clears the first request makes the find call and then 50 other people also make the find call before the first request completes.
One alternative solution:
Make the cached content never expire. Set up a cron task that overwrites the cache by calling a different action which runs:
Cache::write('popular_stories', $total, 'short');
To overwrite the cached content.
This way, the 100s of users per second will ALWAYS read from cache
I am using the 'MeioUpload' plugin found here 'https://github.com/jrbasso/MeioUpload' and Cakephp 2.x.
Currently using this for single image uploads, please can anyone give advice on how to handle multiple image uploads using this plugin. Currently the db table storing the images holds filename, dir, mimetype and filesize fields for each image. I want to store more than one image for each of my posts when adding a new post. Any help would be much appreciated, thanks in advance :).
As I mentioned in my comment, you might want to try https://github.com/josegonzalez/upload as MeioUpload is now deprecated, and it's developer is working on that new upload plugin I linked to.
Either way, the following info for MeioUpload holds true for the new plugin, too.
MeioUpload is built to handle one uploaded file per corresponding set of fields. I don't think the example in MeioUpload's ReadMe is ideal, as it seems to imply that you have to have a table of 'images', where as in reality, you can have a table of just about anything, where each record holds one or more uploaded files (be it images, PDF's, MP3's... anything).
So, with that in mind, you have two solutions:
1) If your posts will have a potentially infinite number of images (ie, not a fixed, small number) then you can have Posts and Images in separate tables, and set up a hasMany relationship between them. See http://book.cakephp.org/2.0/en/models/associations-linking-models-together.html
2) If you know that each post will only have a max of say 3 or 4 (or some other relatively small number) of images, then you can implement 3 (or 4, or X) sets of image fields in your Posts table / model, each to handle a separate upload. They'd be named, eg. featured_image_filename, feautred_image_dir, etc; image2_filename, image2_dir, image2_mimetype, etc; image3_filename, image3_dir, etc.
Your acts as would look something like:
var $actsAs = array(
'MeioUpload.MeioUpload' => array(
'featured_image_filename' => array(
'fields' => array(
'dir' => 'featured_image_dir',
'filesize' => 'featured_image_filesize',
'mimetype' => 'featured_image_mimetype'
),
),
'image2_filename' => array(
'fields' => array(
'dir' => 'image2_dir',
'filesize' => 'image2_filesize',
'mimetype' => 'image2_mimetype'
),
),
'image3_filename' => array(
'fields' => array(
'dir' => 'image3_dir',
'filesize' => 'image3_filesize',
'mimetype' => 'image3_mimetype'
),
),
)
);
This second solution is hardly ideal database design, but sometimes when you know there'll never be more than a few images, it's just the easiest way to do it - both in terms of developing, and in terms of an easy to use UI.
Make sense?
I've read that, to be able to rank search results you may query MySQL like this:
SELECT * ,
MATCH (title, body) AGAINST ('$search') AS rating
FROM posts
WHERE MATCH (title, body) AGAINST ('$search')
ORDER BY rating DESC
Is there a way to do this in CakePHP 2.X?
Also, I need to do this while paginating at the same time. So I think I would need to write condition for the paginator, not a direct 'query'.
Thanks for your help!
Use like this it will prevent mysql injection too
array("MATCH(User.current_position) AGAINST(? IN BOOLEAN MODE)" => $srch_arr['text'])
Ok, it took me some time... Since, the key issue was to get a rating on the resulting matches, the complicated part in this query was the specific field:
MATCH (title, body) AGAINST ('$search') AS rating
I figured that I should just write that field in the "field" option, in the pagination array.
The resulting code was the following:
$this->paginate = array(
'limit' => 15,
'fields' => array('*', "MATCH (data) AGAINST ('$q') AS rating"),
'conditions' => "MATCH(SearchIndex.data) AGAINST('$q' IN BOOLEAN MODE)",
'order' => array(
'rating' => 'desc',
),
);
$paginatedResults = $this->paginate('SearchIndex');
And that worked seamlessly!
I think this is the best way to achieve real search results using Cake. Unless someone has a better alternative :)
Searching phrases in between double quotes will give you the results you should expect!
I have used the above database call by Thomas (thank you) and it does work seamlessly.
However the code:
'conditions' => "MATCH(SearchIndex.data) AGAINST('$q' IN BOOLEAN MODE)",
removes the Data Abstraction Layer and opens up your site to SQL injection.
It's probably not quite as good (haven't fully tested it) but try:
'SearchIndex.data LIKE'=>'%'.$search.'%'
I hope this is helpful in someway.