The big picture is:
I want to use eventlet in some application that does asynchronous IO while working with Django models externally. Working with Django externally is simple (see Django: How can I use my model classes to interact with my database from outside Django?) but it's not the main problem.
I presume (I haven't tested) that using Django from greenlets is dangerous. Actually, in the case of psycopg2, we have this warning (see http://www.initd.org/psycopg/docs/advanced.html#support-to-coroutine-libraries):
Psycopg connections are not green
thread safe and can’t be used
concurrently by different green
threads
In Django there is one db connection per thread (right?) and as such it may lead to scary scenarios when used. Is it possible to provide a connection object manually somehow? Or make it "greenlet-local"?
My motivation is to use connection pool from eventlet (http://eventlet.net/doc/modules/db_pool.html) so that I could speed up my IO-bound application.
There are some projects out there to make Django work well with greenlet. I would check out psycogreen which uses the coroutines support in Psycopg >= 2.2. There is a good blog post on using gevent, gunicorn, and psycogreen together with Django: http://dbinit.com/blog/going-green/.
Related
To my understanding, we should do database calls async since it will block IOLoop, and I also found some library/tools that working on this.
But recently I just found lots of posts about this question on stackoverflow
Tornado Asynchronous Handler
How does async work in Tornado?
and something from wiki https://github.com/tornadoweb/tornado/wiki/Threading-and-concurrency
Do it synchronously and block the IOLoop. This is most appropriate for
things like memcache and database queries that are under your control
and should always be fast. If it's not fast, make it fast by adding
the appropriate indexes to the database, etc.
So should we do database calls async?
and a related question: as I know, there are two methods to do database calls async
For sync database(mysqldb), we can
executor = ThreadPoolExecutor(4)
result = yield executor.submit(mysqldb_operation)
For async database(motor), we can use it directly.
what's the different of them based on tornado?
I had a similar doubt and went through some research and digging and finally have some grip over this.
Short Answer - Yes, Do it Asynchronously.
Long Answer :
Database calls are generally blocking calls. Tornado is best suited as a framework when we do async/Non-blocking I/O calls to completely utilize its full potential.
Now, You can do Database Calls in Tornado in the following ways, I am giving the How and Why for each method.
Method 1 : Synchronous Database Calls - Start a number of Tornado instances behind a load balancer like for example nginx, and make synchronous database calls(as mentioned in the wiki). In this case, the process with the particular request making the blocking database call will block. So nginx will load balance the other requests by directing them to the other running tornado instances to handle the requests.
NOTE : That wiki is old, and hasn't been updated so that method mentioned is prior to the addition of coroutines in the newer versions of Tornado as mentioned by Ben Darnell, one of the lead developers of Tornado.
Update : The wiki has been updated by Ben recently and you can check it now.
Method 2 : By using multiprocessing or multithreading, concurrent.futures, similar to what you have mentioned:
For sync database(mysqldb), we can
executor = ThreadPoolExecutor(4)
result = yield executor.submit(mysqldb_operation)
Method 3 : By using Async libraries, like for instance if you are using MySql, then you can use libraries like TorMySql, Tornado-MySQL, other links given here-Tornado 3rd-party libraries, go to the MySql section. These libraries are to be used with Tornado's async decorators, tornado.gen.coroutine or tornado.web.asynchronous. Similarly, you need to use other async libraries for different databases.
NOTE : You cannot use blocking libraries otherwise its all the same as Method 1.
CONCLUSION : Method 3 is what I mentioned as my Short Answer. Method 2 is the next best solution, followed by Method 1. This is in accordance with your performance requirements. If you requirement is for not-so-heavy(few to moderate) request processing per second or per minute, than going for the easy and general Method 1 will suffice.
I'm not really sure how to handle the scenario I have in a good code manner.
The basic of the criteria of my work is this:
A WPF application that consumes a WCF service
The service uses per session instancing
The session starts soon after application is started and should live through the application lifetime (with small exceptions)
Some method calls in a session must precede and be finished before others are called
This means I will have to be able to have one instance of a proxy client throughout the whole application. I will also have to be able to handle async calls, so the client won't hang up, but at the same time ensure they are finished.
My technical understanding go WCF is limiting enough to not know if certain scenarios would work as intended. So I'm going to list my uncertainties:
When does a session start and when does it end. Is it based on the creation of clients or could a separate client instance access the same session if the first would go faulted.
What is the best way to handle exceptions through a WCF service
Is ChannelFactory something I should look at to help me put here.
So what I did in the first iteration to try to solve some of these problems.
I used dependency injection to inject the client instance throughout the classes of my WPF application (I'm using MVVM) to ensure the same instance is everywhere.
I made the service reference using the asynchronous generation method to get the Begin and End versions of all methods to ensure the calls would be async
I used the Coroutine (IResult interface) feature of the Caliburn.Micro framework to ensure one async action is finished before the other begins (have no idea if this is a proper usage or if it is a smart move at all).
Problems I still have is of course how to handle a faulted state of the client. I'm assuming right now that I could reinstance the client and either rescue the session or I could actually just set it up again as it was. I now need to reinstance it everyplace I injected it in with the same new instance.
So I though perhaps it would be best to create a ClientManager class that would wrap the Client. That way I could inject this ClientManager and reinstance inside of him if needed. I guess I should expose him outwards to be able to make method calls but it would be great if I could error handle inside him in somehow. I'm just having a hard time testing my methods and I'm never certain it will work properly in integration because I don't understand all of the inner workings of WCF, coroutines and threading.
Is there anyone out there with more experience then me in these matters that could give me a few pointers or at least tell me how WCF works in these situations ( per session ) and what I'm doing wrong and what right.
WCF supports sessions out-of-box, so I would recommend starting with this MSDN article.
At a very high level, first you set SessionMode=SessionMode.Required in your ServiceContract. And then, set the IsInitiating=True and IsTerminating=True properties on your OperationContract's to mark the start and end of each session.
However, note that WCF limits concurrent sessions by default to 16 to prevent DOS attacks, but you can always up the value. Also, you would have realized that the session is valid as long as its host (IIS / Windows Service / other) is not recycled.
On a related note, I have used WCF Durable Services earlier - which are meant to persist the state of your WCF service in a data-store (default is SQL Server). Ofcourse, there is a performance hit here. Suggest reading further to see if this is the right choice for you.
Hope this helps.
Could some explain how to best organize the model entity classes in GWT for use on app engine?
I have been using this ebook as an example to follow http://code.google.com/p/gwt-gae-book/wiki/StoringData, but I am unsure about where to add this code. I do not need help in how to write the classes, I just want to know does this code go in the client or server?
In my application I have one module that handles the UI and that is it so far. My next step is implementing the data functionality features.
I also plan on using twig and appwrench, if possible, in developing my model if that helps.
Thanks in advance for any help in getting this setup.
Am assuming you are new to programming and hence detailed explanation for you:
Everything else except for UI and RPC calls to server will reside in "server" package. Within server side code, you would further like to create modules which interact with each other. Like the layer which receives calls from client and processes them. Another layer which contains core business logic, and next layer which interacts with DB, where your entities/model will reside.
You can look at example for your current problem which separates various layers of code. Only difference being the code is using JSP for it's UI.
I need to build an offline database application on WP7.
App is simple - it's about making orders from our clients, then translate it to main server (MS SQL).
Spend a days read about existing techologies - but I'am still confused. Which is right for that project?
Sync Framework.
Looking good, but as I understand - it provides single tables - no reference beetwen them. All the references I have to build on client side. Sad.
Entity FrameWork on server side.
And I have no clue - what can I use on client side. Is there a way to serialize entity object to Isolate Store, then restore it, and continue work with it? May be I can use Sync FrameWork, but scheme will become strange then - kinda one way.)))
Working with WCF & XML - most simple for me. A lot of code and conversion, but in this case I understand the data flow. In other view - I already have app with pure SQL-queries. I wanna be advanced. ))))
Using ext. databases (siaqodb for example).
Which one? siaqodb suppots "Sync provider", but it doesn't support references beetwen objects - so I have to build them by myself? Any gain? I don't know.
Is there another way to build such apps? Point it please.
If this has to be done offline, then I would generally use something like:
storing the minimal amount of the required data within isolated using a WP7 specific database like Sterling
using either a new REST or a new RIA/WCF service with objects/functions you define in order to provide the required data synchronisation
I think this is your option 3?
I've never really liked automatic data synchronisation. I just find it easier to code the sync and deal with the error cases myself - this is especially the case if your wp7 client app uses quite a small footprint of data in relation to the larger main server db.
Anybody here has actually implemented any logging strategy when application is running as XBAP ? Any suggestion (as code) as to how to implement a simple strategy base on your experience.
My app in desktop mode actually logs to a log file (rolling log) using integrated asop log4net implementation but in xbap I can't log cause it stores the file in cache (app2.0 or something folder) so I check if browser hosted and dont log since i dont even know if it ever logs...(why same codebase)....if there was a way to push this log to a service like a web service or post error to some endpoint...
My xbap is full trust intranet mode.
I would log to isolated storage and provide a way for users to submit the log back to the server using either a simple PUT/POST with HttpWebRequest or, if you're feeling frisky, via a WCF service.
Keep in mind an XBAP only gets 512k of isolated storage so you may actually want to push those event logs back to the server automatically. Also remember that the XBAP can only speak back to it's origin server, so the service that accepts the log files must run under the same domain.
Here's some quick sample code that shows how to setup a TextWriterTraceListener on top of an IsolatedStorageFileStream at which point you can can just use the standard Trace.Write[XXX] methods to do your logging.
IsolatedStorageFileStream traceFileStream = new IsolatedStorageFileStream("Trace.log", FileMode.OpenOrCreate, FileAccess.Write);
TraceListener traceListener = new TextWriterTraceListener(traceFileStream);
Trace.Listeners.Add(traceListener);
UPDATE
Here is a revised answer due to the revision you've made to your question with more details.
Since you mention you're using log4net in your desktop app we can build upon that dependency you are already comfortable working with as it is entirely possible to continue to use log4net in the XBAP version as well. Log4net does not come with an implementation that will solve this problem out of the box, but it is possible to write an implementation of a log4net IAppender which communicates with WCF.
I took a look at the implementation the other answerer linked to by Joachim Kerschbaumer (all credit due) and it looks like a solid implementation. My first concern was that, in a sample, someone might be logging back to the service on every event and perhaps synchronously, but the implementation actually has support for queuing up a certain number of events and sending them back to the server in batch form. Also, when it does send to the service, it does so using an async invocation of an Action delegate which means it will execute on a thread pool thread and not block the UI. Therefore I would say that implementation is quite solid.
Here's the steps I would take from here:
Download Joachim's WCF appender implementation
Add his project's to your solution.
Reference the WCFAppender project from your XBAP
Configure log4net to use the WCF appender. Now, there are several settings for this logger so I suggest checking out his sample app's config. The most important ones however are QueueSize and FlushLevel. You should set QueueSize high enough so that, based on how much you actually are logging, you won't be chattering with the WCF service too much. If you're just configuring warnings/errors then you can probably set this to something low. If you're configuring with informational then you want to set this a little higher. As far as FlushLevel you should probably just set this to ERROR as this will just guarantee that no matter how big the queue is at the time an error occurs everything will be flushed at the moment an error is logged.
The sample appears to use LINQ2SQL to log to a custom DB inside of the WCF service. You will need to replace this implementation to log to whatever data source best suits your needs.
Now, Joachim's sample is written in a way that's intended to be very easy for someone to download, run and understand very quickly. I would definitely change a couple things about it if I were putting it into a production solution:
Separate the WCF contracts into a separate library which you can share between the client and the server. This would allow you to stop using a Visual Studio service reference in the WCFAppender library and just reference the same contract library for the data types. Likewise, since the contracts would no longer be in the service itself, you would reference the contract library from the service.
I don't know that wsHttpBinding is really necessary here. It comes with a couple more knobs and switches than one probably needs for something as simple as this. I would probably go with the simpler basicHttpBinding and if you wanted to make sure the log data was encrypted over the wire I would just make sure to use HTTPS.
My approach has been to log to a remote service, keyed by a unique user ID or GUID. The overhead isn't very high with the usual async calls.
You can cache messages locally, too, either in RAM or in isolated storage -- perhaps as a backup in case the network isn't accessible.
Be sure to watch for duplicate events within a certain time window. You don't want to log 1,000 copies of the same Exception over a period of a few seconds.
Also, I like to log more than just errors. You can also log performance data, such as how long certain functions take to execute (particularly out-of-process calls), or more detailed data in response to the user explicitly entering into a "debug and report" mode. Checking for calls that take longer than a certain threshold is also useful to help catch regressions and preempt user complaints.
If you are running your XBAP under partial trust, you are only allowed to write to the IsolatedStorage on the client machine. And it's just 512 KB, which you would probably want to use in a more valuable way (than for logging), like for storing user's preferences.
You are not allowed to do any Remoting stuff as well under partial trust, so you can't use log4net RemotingAppender.
Finally, under partial trust XBAP you have WebPermission to talk to the server of your app origin only. I would recommend using a WCF service, like described in this article. We use similar configuration in my current project and it works fine.
Then, basically, on the WCF server side you can do logging to any place appropriate: file, database, etc. You may also want to keep your log4net logging code and try to use one of the wcf log appenders available on the internets (this or this).