we have recently installed Redis Cache for our magento based sites.
we 2 websites on our server both using same redis server, our server is based on linux with centos.
the issue we are facing is that redis is consuming quite a lot of RAM, and it is growing by every day.
we have set some values on our magento local.xml file for redis
<redis_session>
<host>xxxxxxx</host>
<port>xxxx</port>
<password></password>
<timeout>2.5</timeout>
<persistent></persistent>
<db>1</db>
<compression_threshold>2048</compression_threshold>
<compression_lib>gzip</compression_lib>
<log_level>1</log_level>
<max_concurrency>6</max_concurrency>
<break_after_frontend>5</break_after_frontend>
<break_after_adminhtml>30</break_after_adminhtml>
<first_lifetime>86400</first_lifetime>
<bot_first_lifetime>60</bot_first_lifetime>
<bot_lifetime>7200</bot_lifetime>
<disable_locking>0</disable_locking>
<min_lifetime>60</min_lifetime>
<max_lifetime>2592000</max_lifetime>
<automatic_cleaning_factor>1</automatic_cleaning_factor>
</redis_session>
it seems like we do not have an expire set to it, plus there is no memory usage limit either.
i know there are few instruction on internet for setting expire but there is nothing for use with magento in easy way.
all help is apreciated.
Related
I have a per-user DB architecture like so:
There is around 200 user DBs and each has a continuous replication link to the master couch. ( All within the same couch instance) The problem is the CPU usage at any given time is always close to 100%.
The DBs are idle, so no data is getting written/read from them.There's only a few KB of data per DB so I don't think the load is an issue at this point. The master DB size is less than 10 MB.
How can I go about debugging this performance issue?
You should have a look at https://github.com/redgeoff/spiegel - it's a tool to handle many CouchDB replications in a scalable way. Basically it achieves that by listening to the _global_changes endpoint and creating replications only when needed.
In recent CouchDB versions (2.1.0+), the replicator has been improved, but I think for replicating per-user databases it still makes sense to use an external mechanism like Spiegel to manage the number of active replications.
Just as a reminder, there are some security flaws in CouchDB 2.1.0 and you might need to upgrade to 2.1.1. Maybe you've been hacked like this one.
Currently clouds are mushrooming like crazy and people start to deploy everything to the cloud including CMS systems, but so far I have not seen people that have succeeded in deploying popular CMS systems to a load balanced cluster in the cloud. Some performance hurdles seem to prevent standard open-source CMS systems to be deployed to the cloud like this.
CLOUD: A cloud, better load-balanced cluster, has at least one frontend-server, one network-connected(!) database-server and one cloud-storage server. This fits well to Amazon Beanstalk and Google Appengine. (This specifically excludes CMS on a single computer or Linux server with MySQL on the same "CPU".)
To deploy a standard CMS in such a load balanced cluster needs a cloud-ready CMS with the following characteristics:
The CMS must deal with the latency of queries to still be responsive and render pages in less than a second to be cached (or use a precaching strategy)
The filesystem probably must be connected to a remote storage (Amazon S3, Google cloudstorage, etc.)
Currently I know of python/django and Wordpress having middleware modules or plugins that can connect to cloud storages instead of a filesystem, but there might be other cloud-ready CMS implementations (Java, PHP, ?) and systems.
I myself have failed to deploy django-CMS to the cloud, finally due to query latency of the remote DB. So here is my question:
Did you deploy an open-source CMS that still performs well in rendering pages and backend admin? Please post your average page rendering access stats in microseconds for uncached pages.
IMPORTANT: Please describe your configuration, the problems you have encountered, which modules had to be optimized in the CMS to make it work, don't post simple "this works", contribute your experience and knowledge.
Such a CMS probably has to make fewer than 10 queries per page, if more, the queries must be made in parallel, and deal with filesystem access times of 100ms for a stat and query delays of 40ms.
Related:
Slow MySQL Remote Connection
Have you tried Umbraco?
It relies on database, but it keeps layers of cache so you arent doing selects on every request.
http://umbraco.com/azure
It works great on azure too!
I have found an excellent performance test of Wordpress on Appengine. It appears that Google has spent some time to optimize this system for load-balanced cluster and remote DB deployment:
http://www.syseleven.de/blog/4118/google-app-engine-php/
Scaling test from the report.
parallel
hits GAE 1&1 Sys11
1 1,5 2,6 8,5
10 9,8 8,5 69,4
100 14,9 - 146,1
Conclusion from the report the system is slower than on traditional hosting but scales much better.
http://developers.google.com/appengine/articles/wordpress
We have managed to deploy python django-CMS (www.django-cms.org) on GoogleAppEngine with CloudSQL as DB and CloudStore as Filesystem. Cloud store was attached by forking and fixing a django.storage module by Christos Kopanos http://github.com/locandy/django-google-cloud-storage
After that, the second set of problems came up as we discovered we had access times of up to 17s for a single page access. We have investigated this and found that easy-thumbnails 1.4 accessed the normal file system for mod_time requests while writing results to the store (rendering all thumb images on every request). We switched to the development version where that was already fixed.
Then we worked with SmileyChris to fix unnecessary access of mod_times (stat the file) on every request for every image by tracing and posting issues to http://github.com/SmileyChris/easy-thumbnails
This reduced access times from 12-17s to 4-6s per public page on the CMS basically eliminating all storage/"file"-system access. Once that was fixed, easy-thumbnails replaced (per design) file-system accesses with queries to the DB to check on every request if a thumbnail's source image has changed.
One thing for the web-designer: if she uses a image.width statement in the template this forces a ugly slow read on the "filesystem", because image widths are not cached.
Further investigation led to the conclusion that DB accesses are very costly, too and take about 40ms per roundtrip.
Up to now the deployment is unsuccessful mostly due to DB access times in the cloud leading to 4-5s delays on rendering a page before caching it.
Have to say im not an administrator of any sorts and never needed to distribute load on a server before, but now im in a situation where i can see that i might have a problem.
This is the scenario and my problem :
I have a IIS running on a server with a MSSQL, a client can send off a request that will retrieve a datapackage with a request (1 request) to the MSSQL database, that data is then sent back to the client.
This package of data can be of different lenght, but generally <10 MB.
This is all working fine, but im now facing a what-if if i have 10.000 clients pounding on the server simulataniously, i can see my bandwith getting smashed probably and also imagine that both IIS and MSSQL will be dying of exhaustion.
So my question is, i guess the bandwith issue is only about hosting ? but how can i distribute this so IIS and MSSQL will be able to perform without exhausting them ?
Really appriciate an explanation of how this can be achieved, its probably standard knowledge but for me its abit of a mystery, but know it can be done when i look at dropbox and whatelse just a big question how i can do it.
thanks alot
You will need to consider some form of Load Balancing. Since you are using IIS, I'm assuming that you are hosting on Windows Server, which provides a software based Network Load Balancer. See Network Load Balancing Overview
You need to identify the performance bottleneck then plan to reduce them. A sledgehammer approach here might not be the best idea.
Setup performance counters and record a day or two's worth of data. See this link on how to do SQL server performance troubleshooting.
The bandwidth might just be one of the problems. By setting up performance counters and doing a analysis of what is actually happening you will be able to plan a better solution with the right data.
I have an unlimited host in everything, except the MySql Database, which has limited number of connection at the same time. I want to know if there is any way to cache the posts in the server/host, so it doesn't load them every from the Database every time a visitor loads the page.
This would not be a problem without a lot of traffic, but I have a lot of traffic and this crashed they crashed my Database yesterday.
Thank you.
use a caching solution. memcache is one of the most used.
there are also some which does a complete page caching - resin and varnish are quite popular.
in fact there is a varnish plugin fir wordpress at http://wordpress.org/extend/plugins/wordpress-varnish/installation/.
Could somebody please give some tips on how to improve web2py performance (WSGI apache + MySQL)? I have an application that receives Ajax requests from the client every few seconds to access database and return results. The server is a Ubuntu machine with 640 Mb RAM (virtual server on Amazon EC2, no Xserver).
There are 4 WSGI-processes in apache config. A newly started apache2 instance leaves ca 300 Mb free, but after a hundred requests the system is getting slow and there are long delays. Restarting the server helps to free up memory (I set up cron to do it every 30 minutes - but I guess it is bad practice).
Will be grateful for any advance! A more powerful server is not an option yet because of the budget.
Thanks in advance!
Make sure you use connection pools. Makes a big difference.
Also do not use cron. Use a background process. Cron may eat more memory than necessary.
Read 11 Deployment Recipes of the Web2Py book ! There are a lot of ways to improve web2py performance
If you are using background scripts, make sure to commit() or rollback() your transactions. This is not needed on web2py environment. But if you are running outside scripts it will be needed.