I am interested in make better programs with more responsive design and capabilities. Nowadays, when I create my programs that access data remotely, my interface freezes and there is no animated GIF to work on that condition.
I was told by David Hefferman that animated GIF that are created in the VCL do not respond even in threads because the VCL is in the main thread, and the same goes to databases.
My doubt here is how to work with threads, specifically in databases, so I have lots of questions about it.
Do I have to implement my entire database in thread functions and procedures?
If that is correct, then I can't use database by dropping components to the Form, right?
But what about the user input and grids? Will they work correctly with those threads or will I have to user regular TEdit instead of TDBEdit to then send it's content to a insert/update sql command?
The main objective in here is to create Delphi application that access remote databases like MySQL using Zeos but not freezing for every drop of consult made in the server. At least the smaller ones. It would be very ugly if the system were to download a list of records to a table and the user could still input things. For those cases I would like very much that my animated GIF (or other solutions) could work.
Thank you for any help at all!
In my experience, the best approach is to drop your database components on a Data module and then create this data module dynamically in each thread. Database components typically work fine if they are created and initialized in the thread that is using them.
There are, however, caveats - if you are connecting to a Firebird database, you should make sure that only one thread at the time is establishing a connection. (Use a critical section around the code that connects to the database.) This holds for Firebird 1.5, 2.0 and 2.1 but may not be necessary anymore for Firebird 2.5 (I didn't yet have opportunity to test it).
EDIT (in answer to EASI's comment): Yes, connecting to a database can take some time. If you frequently need to execute short operations, it is best to keep threads connected and running for a longer period of time.
I can think of two ways to do that. 1) Keep threads alive and connected and run a message loop inside. This loop would receive commands from the main thread, process them and return a result. 2) Keep threads initialized and connected in a thread pool and activate them when you need to perform a database operation.
Basically, both approaches are the same, the difference is in the level which handles 'receive and process command' loop.
The second approach can be easily implemented in the OmniThreadLibrary by using the IOmniConnectionPool.SetThreadDataFactory mechanism. See Adding connection pool mechanism to OmniThreadLibrary and demo 24_ConnectionPool for more information. Alternatively, you can use the high-level abstraction Background worker where you can establish database connection on a per-thread basis in a Task initialization block.
Related
If I want to query (read) from an SQLite database using Haskell's HDBC across multiple threads, can I use a single connection, or should each thread have its own connection? Thanks.
I searched through the hdbc-sqlite code and found this comment:
Logic for handling counts of changes: look at the total changes before
and after the query. If they differ, then look at the local changes.
(The local change counter appears to not be updated unless really
running a query that makes a change, according to the docs.)
This is OK thread-wise because SQLite doesn't support using a given
dbh in more than one thread anyway.
The official Sqlite documentation has a whole page about this topic
The FAQ says:
(5) Can multiple applications or multiple instances of the same
application access a single database file at the same time?
Multiple processes can have the same database open at the same time.
Multiple processes can be doing a SELECT at the same time. But only
one process can be making changes to the database at any moment in
time, however.
This information would exclude both of your approaches. Perhaps you can write some tests for this to show that this information are wrong.
I am new to the community and looking forward to being a contributing member. I wanted to throw this out there and see if anyone had an advice:
I am currently in the middle of developing a MVC 3 app that controls various SQL Jobs. It basically allows user to schedule jobs to be completed in the future, but also also allows them to run jobs on demand.
I was thinking of having a thread run in the web app that pulls entity information into an XML file, and writing a window service to monitor this file to perform the requested jobs. Does this sound like a good method? Has anyone done something like this before and used a different approach? Any advice would be great. I will keep the forum posted on progress and practices.
Thanks
I can see you running into some issues using a file for complex communication between processes - files can generally only be written by one process at a time, so what happens if the worker process tries to remove a task at the same time as the web process tries to add a task?
A better approach would be to store the tasks in a database that is accessible to both processes - a database can be written to by multiple processes, and it is easy to select all tasks that have a scheduled date in the past.
Using a database you don't get to use FileSystemWatcher, which I suspect is one of the main reasons you want to use a file. If you really need the job to run instantly there are various sorts of messaging you could use, but for most purposes you can just check the queue table on a timer.
I am implementing a small database like MySQL.. Its a part of a larger project..
Right now i have designed the core database, by which i mean i have implemented a parser and i can now execute some basic sql queries on my database.. it can store, update, delete and retrieve data from files.. As of now its fine.. however i want to implement this on network..
I want more than one user to be able to access my database server and execute queries on it at the same time... I am working under Linux so there is no issue of portability right now..
I know i need to use Sockets which is fine.. I also know that i need to use a concept like Thread Pool where i will be required to create a maximum number of threads initially and then for each client request wake up a thread and assign it to the client..
As for now what i am unable to figure out is how all this is actually going to be bundled together.. Where should i implement multithreading.. on client side / server side.? how is my parser going to be configured to take input from each of the clients separately?(mostly via files i think?)
If anyone has idea about how i can implement this pls do tell me bcos i am stuck here in this project...
Thanks.. :)
If you haven't already, take a look at Beej's Guide to Network Programming to get your hands dirty in some socket programming.
Next I would take his example of a stream client and server and just use that as a single threaded query system. Once you have got this down, you'll need to choose if you're going to actually use threads or use select(). My gut says your on disk database doesn't yet support parallel writes (maybe reads), so likely a single server thread servicing requests is your best bet for starters!
In the multiple client model, you could use a simple per-socket hashtable of client information and return any results immediately when you process their query. Once you get into threading with the networking and db queries, it can get pretty complicated. So work up from the single client, add polling for multiple clients, and then start reading up on and tackling threaded (probably with pthreads) client-server models.
Server side, as it is the only person who can understand the information. You need to design locks or come up with your own model to make sure that the modification/editing doesn't affect those getting served.
As an alternative to multithreading, you might consider event-based single threaded approach (e.g. using poll or epoll). An example of a very fast (non-SQL) database which uses exactly this approach is redis.
This design has two obvious disadvantages: you only ever use a single CPU core, and a lengthy query will block other clients for a noticeable time. However, if queries are reasonably fast, nobody will notice.
On the other hand, the single thread design has the advantage of automatically serializing requests. There are no ambiguities, no locking needs. No write can come in between a read (or another write), it just can't happen.
If you don't have something like a robust, working MVCC built into your database (or are at least working on it), knowing that you need not worry can be a huge advantage. Concurrent reads are not so much an issue, but concurrent reads and writes are.
Alternatively, you might consider doing the input/output and syntax checking in one thread, and running the actual queries in another (query passed via a queue). That, too, will remove the synchronisation woes, and it will at least offer some latency hiding and some multi-core.
I tried sqlite,
by using multi-thread, only one thread can update db at the same time..
I need multi-thread updating the db at same time.
Is there are any DB can do the job?
ps: I use delphi6.
I found that sqlite can support multi-threading,
But in my test of asgsqlite, when one thread inserting, others will fail to insert.
I'm still in testing.
SQLite can be used in multi-threaded environments.
Check out this link.
Firebird can be used in an embedded version, but it's no problem to use the standard (server) installation locally as well. Very small, easy to deploy, concurrent access. Works good with Delphi, you should look into it as an option.
See also the StackOverflow question "Which embedded database to use in a Delphi application?"
Sqlite locks the entire database when updating (unless this has changed since I last used it). A second thread cannot update the database at the same time (even using entirely separate tables). However there is a timeout parameter that tells the second thread to retry for x milliseconds before failing. I think ASqlite surfaces this parameter in the database component (I think I actually wrote that bit of code, all 3 lines, but it was a couple of years ago).
Setting the timeout to a larger value than 0 will allow multiple threads to update the database. However there may be performance implications.
since version 3.3.1, SQLite's threading requirements have been greatly relaxed. in most cases, it means that it simply works. if you really need more concurrency than that, it might be better to use a DB server.
SQL Server 2008 Express supports concurrency, as well as most other features of SQL Server. And it's free.
Why do you need multiple threads to update it at the same time? I'm sure sqlite will ensure that the updates get done correctly, even if that means one thread waiting for the other one to finish; this is transparent to the application.
Indeed, having several threads updating concurrently would, in all likelihood, not be beneficial to performance. That's to say, it might LOOK like several threads were updating concurrently, but actually the result would be that the updates get done slower than if they weren't (due to the fact that they need to hold many page locks etc to avoid problems).
DBISAM from ElevateSoft works very nicely in multi-threaded mode, and has auto-session naming to make this easy. Be sure to follow the page in the help on how to make it all safe, and job done.
I'm actually at the moment doing performance testing with a multi-threaded Java process on Sybase ASE. The process parses a 1GB file and does inserts into a table.
I was afraid at first, because many of the senior programmers warned me about "table locking" and how dangerous it is to do concurrent access to DB. But I went ahead and did testing (because I wanted to find out for myself).
I created and compared a single threaded process to a process using 4 threads. I only received a 20% reduction in total execution time. I retried the the process using different thread counts and batch insert sizes. The maximum I could squeeze was 20%.
We are going to be switching to Oracle soon, so I'll share how Oracle handles concurrent inserts when that happens.
I know this has been asked before here there and everywhere but i can't get a clear explanation so i'm going to pitch it again. So what is all of the fuss about using a singleton to control the db connection in your web app? Some like it some hate it i don't understand it. From what I've read, "it's to ensure that there is always only one active connection to your DB". I mean why is that a good thing? 1 active DB connection on a data driven web app processing multiple requests per second spells trouble doesn't it? For whatever reason nobody can properly explain this. I've been all over the web. I know i'm thick.
Assuming Java here, but is relevant to most other technologies as well.
I'm not sure whether you've confused the use of a plain singleton with a service locator. Both of them are design patterns. The service locator pattern is used by applications to ensure that there is a single class entrusted with the responsibility of obtaining and providing access to databases, files, JMS queues, etc.
Most service locators are implemented as singletons, since there is no need for multiple service locators to do the same job. Besides, it is useful to cache information obtained from the first lookup that can be later used by other clients of the service locator.
By the way, the argument about
"it's to ensure that there is always
only one active connection to your DB"
is false and misleading. It is quite possible that the connection can be closed/reclaimed if left inactive for quite a long period of time. So caching a connection to the database is frowned upon. There is one deviation from this argument; "re-using" the connection obtained from the connection pool is encouraged as long as you do so with the same context, i.e. within the same HTTP request, or user request (whichever is applicable). This done obviously, from the point of view of performance, since establishing new connections can prove to be an expensive operation.
High-performance (or even medium-performance) web apps use database connection pooling, so one DB connection can be shared among many web requests. The singleton is usually the object which manages this pool. I think the motivation for using a singleton is to idiot-proof against maintenance programmers that might otherwise instantiate many of these objects needlessly.
"it's to ensure that there is always only one active connection to your DB." I think that would be better stated as to ensure each CLIENT has only one active connection to your DB. The reason why this is incredibly important is because you want to prevent deadlocks. If I have TWO open database connections (as a client) I might be updating on one connection, then I might try to update the same row in another connection. This will a deadlock which the database cannot detect. So, the idea of the singleton is basically to make sure that there is ONE object who is charge of handing out database connections to each client. Basically. You don't HAVE to have a singleton for this, but most people will tell you it just makes sense that the system only has one.
You're right--usually this isn't what you want.
However, there are plenty of cases where you need to throttle yourself down to a single connection. By serializing your access to the database through a singleton, you can address other issues or constraints like load, bandwidth, etc.
I've done something similar in the past for a bulk processing app. Instead, though, I used a semaphore to synchronize access to the database so I could allow n concurrent db operations.
One might want to use a singleton due to database server constraints, for example, a server might limit the number of connections.
My main conscious reason is that you know what connections can be managed/closed etc., just makes things a bit more organised when you don't have unnecessary, redundant connections.
I don't think it's a simple answer. For instance on ASP.NET, the platform implements connection pooling by default, so it will automatically adjust a "pool" of connections and re-use them so you're not constantly creating and destroying expensive objects.
However, let's say you were writing a data collection application that monitored 200 separate input sources. Every time one of those inputs changed, you fire off a thread that records the event to the database. I would say that could be a bad design if there's a chance that even a fraction of those could fire off at the same time. Suddenly having 20 or 40 active database connections is inefficient. It might be better to queue the updates, and as long as there are updates left in the queue, a singleton connection picks them off the queue and executes them on the server. It's more efficient because you only have to negotiate the connection and authentication once. Once there's no activity for a while you could choose to close down the connection. This kind of behavior would be hard to implement without a central resource manager like a singleton.
"only one active connection" is a very narrow statement for illustration. It could just as well be a singleton managing a pool of connection. The point of a singleton for database connections is that you don't want every consumer making it's own connection or set of connections.
I think you might want to be more specific about, "using a singleton to control the db connection in your web app." Ideally, a java.sql.Connection object will not be thread safe, but your javax.sql.DataSource may want to pool connections, so you should go to a single instance of it to share the pooling.
you are more looking for one connection per request, not one connection for the entire application. you can still control access to it through a singleton though (storing the connection in the HttpContext.Items collection).
It guarantees that each client using your site only gets one connection to the db.
You really do not want a new connection being made everytime a user does an action that will create a db query. Not only for performance reasons with the connection handshaking involved, but to decrease load on the db server.
DB connections are a precious commodity, and this technique helps minimize the amount used at any given time.