I'm using cakephp and Mysql to develop a shop System where a register all the products I sell, the system was working fine till I started to make some tests, I have introduced more than 30.000 registers of products and from there on, I can't search for my products cause I'm having this error:
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 32 bytes)
From my research if found that this is related to memory and I could increase the memory but also I have realized that these would be a momentum solution.
I have seen that the function called in the controller products is this:
public function search() {
$this->Product->recursive = 0;
$this->set('products', $this->paginate());
$products = $this->Product->find('all');
$this->set(compact('products'));
}
and from this I see that this could be the problem cause it's fetching all the products.
So the help I need is how to improve this, what would be the best approach for this. On my research one of the solution I was expecting to get is probably to have this query but fetching like 10 by time I don't know if it's possible.
Thanks in Advance.
PS
Remove the last two lines.
You're doing the paginated call, which is what you want, but then throwing it away and fetching ALL the records. Which is pretty expensive when you have lots of data.
public function search() {
$this->Product->recursive = 0;
$this->set('products', $this->paginate());
}
If you are using CakePHP version 2 than this is the right way to use paginator, you can also limit the data per page with something else like 10, 20, 30 etc. so that all data can't be load at once.
public function search() {
$this->Paginator->settings = array(
'limit' => 10,
'recursive' => 0
);
$products = $this->Paginator->paginate('Product');
$this->set(compact('products'));
}
Related
I manually set the CakePHP Pagination values in my Usergals Controller like so, so as to Paginate a related Model (TreasuresUsergal) on the view of Usergal. Here is a simplified snippet from the UsergalController:
public function view($id = null) {
$this->loadModel('TreasuresUsergal');
$options['joins'] = array(
array('table' => 'treasures',
'alias' => 'Treasure2',
'type' => 'LEFT',
'conditions' => array(
'Treasure2.id = TreasuresUsergal.treasure_id',
)
)
);
$options['conditions']=array('TreasuresUsergal.usergal_id'=>$id);
$options['fields']=array('Treasure2.*','TreasuresUsergal.ord','TreasuresUsergal.comments');
$options['order']=array('TreasuresUsergal.ord'=>'asc');
$options['limit']=$limit;
$this->Paginator->settings = $options;
$treasures=$this->Paginator->paginate('TreasuresUsergal');
$this->set('treasures',$treasures);
}
So in the above example, $id is the value passed to the view function from the URL. There is a live example of this here:
http://collections.centerofthewest.org/usergals/view/20
As you can see, it works just fine for a single page. However, today I tested the Paginator in the view and discovered the "next" button does not work. The Counter, sorting, and Page numbers all load correctly - but anytime the actual named parameter "page:n" is passed (when n is greater than 1) I get a Not Found page with the following error:
Not Found
Error: The requested address '/usergals/view/20/page:2?url=%2Fusergals%2Fview%2F20' was not found on this server.
I must be missing something simple - I have experimented with the routes a little, but haven't been able to figure it out. Or perhaps I am missing some Paginator options? Or does it think its OutOfBounds when its not?
UPDATE / WORKAROUND
After some messing around, I have devised this workaround. Not as nice as I'd like, but here is the basic idea (error handling, etc can be added)
First, I added a check in beforeFilter to see if page paramter was set. If so, I change it to 'p' parameter and redirect.
I did this here because otherwise I had problems with the Not Found exception (see notes at bottom). So, in beforeFilter:
if (isset($this->params['named']['page'])){
$newurl=$this->params['named'];
$pg=$newurl['page'];
unset($newurl['page']);
$newurl['p']=$pg;
$this->redirect(array('action' => 'view/'.$this->params['pass'][0])+$newurl);
}
Then, in the 'view' function of the same controller, I added this along with the other Paginator options:
if (isset($this->params['named']['p'])) $options['page']=$this->params['named']['p'];
With this, the standard Paginator behavior seems to work fine in the view. Prev, next, etc.
If anyone has a better suggestion, I would love to hear it. I don't like the idea of having to redirect, but it works for now.
It's worth noting that adding this code (even just to experiment) - caused all of my pagination counts to stop working. The query in debug was correct, but the displayed counts were wrong:
try {
$this->Paginator->paginate();
} catch (NotFoundException $e) {
}
I have a cakephp application, 2.4, and I'm having issues with the Paginator component. First off, it's not the database, it's definitely the execution of parsing the query results. I have DebugKit installed and can see that my mysql query for the paginated data takes a whole 2 ms. The table has 2.5 million records of messages, and 500,000 users. Obviously proper indexing is in place. But, the controller action is taking 6167.82 ms. So, here's my controller action:
$this->Paginator->settings = array(
'Message' => array(
'fields' => array(
'Recipient.username',
'Recipient.profile_photo',
'Recipient.id',
'Message.*'
),
'joins' => array(array(
'table' => 'users',
'alias' => 'Recipient',
'type' => 'LEFT',
'conditions' => array(
'Recipient.id = `Message`.`recipient_id`'
)
)),
'conditions' => array(
'Message.sender_id' => $this->Auth->user('id'),
'Message.deleted_by_sender' => '0'
),
'limit' => 10,
'order' => 'Message.id DESC',
'recursive' => -1
)
);
$sents = $this->Paginator->paginate( 'Message' );
$this->set( 'sents', $sents );
$this->view = 'index';
I've google this and searched stack overflow. The majority of the responses are for poor mysql optimization which isn't my case. The other half of the responses suggest containable. So, I tried containable. Using contain was actually slower because it tried to grab even more data from the user's field than just the username, photo, and id. Then when cake built the array from the query results it executed nearly 500 ms slower with containable because of the extra user data I'm assuming.
I'm going to now dig into the cake Paginator component and see why it's taking so long to build the response. I'm hoping someone beats me to it and has a good solution to help speed this up.
My web server is running ubuntu 12.04 with 3gb ram, apache and mod_php with apc installed and working for the model and core cache. The database is on a separate server. I also have a redis server persisting other user data and the cake session data. There is plenty of power here to parse 10 records from a mysql query containing about a dozen rows.
EDIT: ANSWER
As suggested first by Ilie Pandia there was something else happening, such as a callback, that was slowing down the pagination. This was actually unrelated to the pagination component. The Recipient model had a behavior that loaded an sdk in the setup callback for a 3rd party service. That service was taking several seconds to respond. This happened when the linkedModel in the query was loaded to filter the results. Hopefully anyone else looking for reasons why cake might be performing poorly will also look at the callbacks on models in the application and plugins.
I see no reason for this to run slow at all.
So this suggests that there are some callback installed (either in the model or the controller) that do additional processing and inflate the action time so much.
That is assuming that there is nothing else in the controller but what your wrote.
You could actually measure the time of the paginate call itself and I think you will find that it is very fast. So the bottle neck is elsewhere in the code.
PS: You could also try to disable DebugKit for a while. Introspection may take very long for some particular cases.
Install DebugKit for your application.
And inspect which query is taking too much time. From there, you should be able to track the bottleneck.
I'm building a dynamic view (Page) that consists of multiple elements (widgets) called via $this->element('messages_unread'). Some of these elements need data that is not related to the Page model.
In real life words: my users will be able to construct their own Page by choosing from a multitude of elements ("top 5 posts", "10 unread messages", etc...)
I get the data by calling $this->requestAction(array('controller'=>'events','action'=>'archive') from within the element, the url-variables differ per element .
I'm aware of the fact that requestAction() is expensive and I plan on limiting the costs by proper caching.
The actual question:
My problem is Pagination. When I'm in the Page view and call requestAction('/events/archive') the PaginatorHelper in the Page view will be unaware of the Event model and its paginator variables and $this->Paginator->next() etc... will not work.
How can I implement proper Pagination? I've tried to set the model by calling $this->Paginator->options(array('model'=>'Event')) but that doesn't work.
Do I maybe need to return custom defined Pagination variables in the requestAction and thus construct my own?
Or is there another approach that maybe even avoids requestAction()? And keep in mind here that the requested data is unrelated to the Page.
Kind regards,
Bart
[Edit] My temporary solution but still open for comments/solutions:
In the requestedAction Event/archive, return paginator variables along with the data like this:
return array('data'=>$this->paginate(), 'paging' => $this->params['paging']);
I've tinkered a bit more and the following works for me, and the PaginationHelper works:
In the element:
// requestAction returns an array('data'=>... , 'paging'=>...)
$data = $this->requestAction(array('controller'=>'events','action'=>'archive'));
// if the 'paging' variable is populated, merge it with the already present paging variable in $this->params. This will make sure the PaginatorHelper works
if(!isset($this->params['paging'])) $this->params['paging'] = array();
$this->params['paging'] = array_merge( $this->params['paging'] , $data['paging'] );
foreach($data['events'] as $event) {
// loop through data...
}
In the Controller:
public function archive() {
$this->paginate = array(
'limit' => 10
);
if ($this->params['requested'])
return array('events'=>$this->paginate('Event'), 'paging' => $this->params['paging']);
$this->set('events', $this->paginate('Event') );
}
I want to list the posts of a given user. It work but paginate is not accurate.
My code is the following
public function index($userid = null) {
if ($this->Post->exists($userid)) {
$this->set('posts',$this->Post->find('all',array('conditions'=>array('user_id'=>$userid))),
$this->paginate());
} else
{
$this->Post->recursive = 0;
$this->set('posts', $this->paginate());
}
The result give the correct list --> 3 posts, but the paginator display page number 1 and 2
Can you help me?
Thank you
Refer to the documentation
The code in the question is quite confused.
find
The find method only has two parameters:
find(string $type = 'first', array $params = array())
The third parameter (the result of calling paginate) isn't used and will be ignored - but it will setup the view variables for the pagination helper, based on the conditions used in the paginate call - there are no conditions being used.
It is not possible to paginate the result of a find call - to do so restructure the code to call paginate instead of find.
paginate
The paginate method is just a proxy for the paginator component - it can be used in several ways, this one (controller code example):
$this->paginate($conditions)
Is the most appropriate usage for the case in the question i.e. the complete action code should be similar to:
public function index($userId = null) {
$conditions = array();
if ($userId) {
$conditions['Post.user_id'] = $userId;
}
$this->set('posts',$this->paginate($conditions));
}
Note that logically, if a user id is requested that doesn't exist the response should be nothing - not everything.
I'm quite sure that conditions for paginate do now work that way.
If you want to set conditions for paginations you should do it as follows:
$this->paginate = array('conditions' => array('Post.user_id' => $userid)));
$this->set('posts', $this->paginate());
And yes, the result stored in $posts ( in view ) will be proper as you assigned proper find result to it, meanwhile you've paginated post model without any conditions whatsoever.
First off, you're checking to see if the post exists but using the $userid. Are you trying to see "if the user exists, get the posts for that user, or else get posts for ALL users"? As you have it right now, say you have the $userid = 159, but the max Post.id in your database is 28, then the condition is not being met because it is checking to see whether or not there is a Post with the id = 159 that exists, which it doesn't.
Second, your conditions are wrong. You are performing a find and then a paginate which are two separate queries. The conditions are being implemented on the find query but not the paginate but you are only displaying the find results.
public function index($userid = null) {
// setting recursive outside of if statement makes it applicable either way
$this->Post->recursive = 0;
// check if user exists
if ($this->Post->User->exists($userid)) {
// get posts for user
$this->set('posts', $this->paginate('Post', array('Post.user_id' => $userid));
}
else{
// get all posts
$this->set('posts', $this->paginate('Post'));
}
} // end index function
Ok, I have looked and looked but cannot seem to find anything on this anywhere. I have a display of results that are paginated beautifully, but they currently display in ascending order. I'd like for them to display in random order. Here is my current controller code:
public function condos() {
$this->paginate['Unit']=array(
'limit'=>9,
'contain'=>array(
'User'=>array(
'id', 'user_name', 'area_code', 'exchange', 'sln', 'email'),
'Complex'=>array('id','complex_name', 'realname', 'photo1','photo2','photo3','photo4','photo5', 'complex_website')
),
'conditions'=>array(
'Unit.type'=>array('condo', 'rentalco'),
'Unit.active'=>1)
);
$data = $this->paginate('Unit');
$this->set('allcondos', $data);
}
For anyone else finding this - the actual answer is to generate a seed (a float between 0 and 1), and save it to the session before the RAND() sort is necessary (in the controller's beforeFilter()). Then:
$this->paginate['order'] = sprintf('RAND(%f), $this->Session->read('seed'));
This preserves the RAND() seed between calls to the paginator, and preserves the overall order of the results between requests.
This seems to work pretty well on CakePHP 2 for me:
$this->paginate = array(
'order' => 'RAND()'
);
This is using the MySQL RAND() function, which Cake just passes on to the database.
EDIT: Now that I think about it, this is not a good idea, because the order is not maintained between pages. I can't think of a good way off the top of my head to randomize the order and maintain continuity between pages. Maybe if you were to shuffle the items on the page with JavaScript?