I would like to get the opinions on situation I'm facing regarding the connection pools.
I'm SW developer, working on some multitenant application. We've one DB and each tenant (client) has it's own schema. For each connected tenant, the solo process is started and it gets a solo DB connection. In the near future, I would need to use it in 300+ simultaneous tenant environment.
From what I read, using a lot of connections (100+) to postgres is not advised. The one solution is to use the connection pool. The other one, to use more DB servers.
I was thinking about the connection pooler (pgBouncer, pgPool). But in current state of our application, it is a bit of problematic. Here is the list of the "problematic" items and proposed solution:
Single connection to the server for whole lifetime of the process - This is because, we're using the temp tables and prepared statements heavily. The temp tables duration are variable but in most circumstances, they span multiple transactions.
Because connection pooler will return the "free" connection I cannot be sure if the given temp table was created in the returned connection. But I think I could workaround it creating "temp tables" in predefined schema (but then I would need to have some background task to clean the orphaned temp tables from the processes that aren't cleanly closed or crashed - the postgres temp tables are dropped on connection close automatically). For prepared statements I haven't found the workaround.
Use of "set search_path="custom_schema",public;" - This is done on application start for each tenant, so the correct tables are used.
This could be fixed by issuing the set search_path=... command on each transaction. It should be cheap/fast.
Use of triggers that depends on the temp tables - This is used for automatic logging of some stuff. It use the temp tables that are automatically created on the application start.
Don't have the solution yet as cannot use the "custom table approach" mentioned above because of the table name should be constant (multiple tenants would then create the same table more than once, which is bad).
So, I don't know if I should start to think about redesigning or not. If I could just add more DB servers and everything would run, than no need to redesign it.
What do you think?
Thank You
These things are not a problem in pgbouncer with pool_mode = session.
Related
We use global temporary tables in our program, written in C++ using ATL as a DB interface. Sessions are established with the SQL OLE DB provider.
This global temporary tables are held for a long time, maybe for the complete time of a session. Such temporary tables are explicitly deleted by us, when the specific action/activity ends. So we always clean up the tables.
Now we see an effect on people that are using a slow or unstable VPN connection that the global temporary is deleted. A query that should read some content returns an error
##tblTemp... is not a valid object name
For me it is an indicator that SQL Server terminated the session.
But how can it be? Our program has internal functions that access the server at least every 5 minutes (even if the user is inactive). Usually the SQL Server is accessed much more frequently. But the program may be minimized in the background.
What timeout is responsible that SQL Server terminates a session and deletes the temporary tables?
I see the Remote Query Timeout in the server settings. But this seams to be wrong for me, because we have no open query here... also the queries to the table are real simple. Insert an record, delete an record.
Questions:
Where do I find the settings for this session timeout?
Is there a way for the client to find out that the session got terminated? Strange for me the SQL query itself was transferred to SQL Server and finally failed because the temporary table did no longer exist. We got other error on the client.
Is there a way to protocol this on the server?
EDIT:
Here more details how we work with this tables.
The tables are created in my main thread. This thread has a SQL session that is created at start of the program and ends with the program end.
Other threads use the temporary tables. We pass the names through it.
So due to the fact that the creating SQL session is still alive and doesn't show an error when executing a statement that uses the temporary table, it tells me that the session is still alive. But my problem is the object seams to be deleted.
Again: We only have this problem on machines with a slow / bad VPN connection to the server!
To quote the manual:
Global temporary tables are automatically dropped when the session that created the table ends and all other tasks have stopped referencing them. The association between a task and a table is maintained only for the life of a single Transact-SQL statement. This means that a global temporary table is dropped at the completion of the last Transact-SQL statement that was actively referencing the table when the creating session ended.
source
So it's not about the server being accessed every few minutes, but the specific object being referenced.
If the session that created the global temporary table is ended (e.g. a timeout) and nothing else is actively referencing the same table, it is dropped!
I'm managing an online book store website. For sake of high availability I've setup two Tomcat instances for running the website application, they are the exactly same program, and they are sharing the same database which located in another server.
My question is that how can I avoid conflicts or dirty data when the two applications do the same updates/inserts at the same time to the database.
For example: update t_sale set total='${num}' where category='cs', if there are two processes execute the above sql simultaneously would cause data lost.
If by "database" you are talking about a well designed schema that is running on an RDBMS such as Oracle, DB2, or SQL Server, then the database itself will prevent what you call "conflicts" by locking parts of the database during each update transaction.
You can prevent "dirty data" from getting into the database by adding features such as check clauses and primary-foreign key structures in the database itself.
Scenario C in This Microsoft Doc describes how temp tables scoped to a connection can be replaced with Memory-Optimized Tables. The scheme uses a Filter Security Policy which calls a function to determine if ##spid matches the SpidFilter column in the Memory-Optimized table.
Will this work with .NET connection pooling? I would expect ##spid will return the same number as a connection is re-used over and over again. .NET clears the session scoped temp tables by calling sp_reset_connection, but that will not clear Memory-Optimized tables, or change ##spid. Maybe sys.dm_exec_sessions's session_id could be added to make it work in a connection pooling environment?
With the help of Microsoft Support, I was able to get the necessary details about ASP.NET Connection Pooling to answer this concern. It is true that ASP.NET threads will share the same SPID, but never at the same time. A thread gets assigned a connection only after the connection is no longer being used by the previous thread. Connection Pooling does not reduce the number of connections needed, it only reduces the number of times connections need to be opened and closed.
This is good documentation on Connection Pooling, although it does not make that distinction. https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/sql-server-connection-pooling
Notice that Scenario C has special note: "Replace the CREATE TABLE #tempSessionC statements in your code with DELETE FROM dbo.soSessionC, to ensure a session is not exposed to table contents inserted by a previous session with the same session_id." – i-one
Since only one thread will be using the connection at a time, this is sufficient. If the table is not also deleted after being used, it will continue consuming memory (especially precious in Azure) until another thread happens to use a connection with the same SPID.
I have two applications.
One inserts data into database continuously like it is having an infinity loop.
When the second application inserts data to same database and table what will happen.
If it waits till the other application to complete inserting which will handle this?
Or it will say it is busy?
Or code throws an exception?
SQL servers have something called a connection pool which means that more than once connection to the database can be made at any particular time, and that's where the easy bit ends.
If you were to for example connect to the database on two applications at the same time and insert data in to different tables from each application then the two could happily happen at the same time without issue.
If however those applications wanted to do something like edit the same row then there's an issue with "locking" ...
Essentially any operation on a SQL database requires "acquiring a lock" on a "set" or "row" or "cell" depending on the configuration of the server its hard to say what might happen in your case.
So the simple answer is:
Yes, SQL can make stuff happen (like inserts) at the same time but with some clauses.
And long answer ...
requires in depth knowledge of locking and your database and server configuration.
I have develop application in which i have created different logins for every client.Our applications is having so many clients like job portals or facebook and every client having huge amount of data .If i use single database then one table get huge amount of data for all client
I find out one solution for that and solution is to create separate database for every client but as there are so many client then we need to create so many databases so that not correct solution
Please can you tell me right way to implement this by using sql server 2008 r2
Thanks
You could try having one schema per client, and that client's logon has that schema as their default and is the only schema that they have access to. However you'll have a lot of schemas so it may not be much help! (Also, iof you're using something like EF to access the db it won't work.)
Single database good:
Easy management
Single database bad:
Possible performance problems (although not until you get into
billions of rows; one DB I designed had a table with more than 21B
rows after 3 months; lucky I made the IDENTITY column a BigInt!)
Security issues/complexity: how do you stop one client accessing
another's data?
Single point of failure for all clients
Multiple database good
Security is easier
Single point of failure per client (assuming multiple DB Servers to
spread that load also)
More flexibility in applying updates: some clients are OK with
Wednesday, some with Thursday
I'm sure that there are other issues as well. Really it's up to your requirements and how they can best be met,
Multiple db bad:
More management required
Given a DB has overhead, your overhead resource usage goes up