We have a bug where data is written to a Postgres database, using a stored procedure. The commit result indicates success. Approximately 500-700 milliseconds later we try to read the recently-written data (using a stored procedure again). However, the data does not exist in the database.
We use two database sessions- one for read and one for write.
We are using libpq C++ library.
The writes are performed asynchronously in a separate thread. We use a PostGres "pipeline" for writes.
The reads are performed synchronously in the application thread.
We are currently restricted to this architecture due to the way the generic database code was written (unless this is the only way to fix the problem).
Could someone please suggest what might be happening/how to fix?
We changed from stored procedure to "inline" and it made no difference.
Related
I have a long running stored procedure that is executed from IIS. On average this stored procedure takes between two and five minutes to complete because it is searching through a large dataset. (although it has take around 20 minutes in some cases)
Most of the time the stored procedure works fine but every now and then the SPIDS go into a sleeping state and never recover. The only solution I have found is to restart the SQL Server and re-run the stored procedure
The are no table inserts in the proc (only table variable inserts), and the other statements are selects on a large table.
I'm stuck for where to start debugging this issue. Any hints one what it might be or suggestions on tools that would help me find the issue would be most helpful
EDIT: More info added:
The actual issue is the proc doesn't return the resultset. My first thought was to look at the spids, they were sleeping but the cputime was still increasing
It's a .Net app so .Net Core 3.1 with ASP.NET Core and a Blazor UI. The libary used for db connection is System.data.SqlClient I believe System.data.SqlClient uses it's own custom driver. Calling code below:
The stored procedure doesn't return multiple result sets, however obviously different instances of the proc run at the same time.
No limits to connection pooling in IIS
#RichardWatts when you say " re-run the stored procedure" you mean that the same stored proc with the same parameter and data works once you restart SQL Server ?
If so look over your loc (sp_loc} inside your table probably another process loc some data and doesnt release it properly, specialy if you have transaction accessing the same tables.
What is your your isolation level on your connexion ? If you can, try to change it to READ UNCOMMITTED to see if that solve your problem.
as an alternate you can also add a WITH (NOLOCK) or (READUNCOMMITTED) to your sql command.
Know that you will need to hold query with a read uncommited or nolock if you have some modification on the structure of your table or index re construction for example or they will in turn block its execution
Nevertheless be cautious this solution depend on your environment, specially if your tables gots lots of update, delete, insert,... this kind of isolation can lead to a Dirty read and doesnt adress the root cause of your problem wich I would bet is uncomited transaction (good article that explain it)
Make also a DBCC CHECKTABLE just to be sure on this side
I have SQL Server 2017 Express database that is accessed by up to 6 tablets that connect via an Angular 7 app using REST web services.
I have a stored procedure that allows a new user to be inserted into a specific database table. The insert will always only insert 1 record at a time, but with 6 clients, the stored procedure could be called by each client almost simultaneously.
The last step of the process is to print an agreement form to a specific printer. Initially this was going to be handled on the client side, but the tablets do not have the ability to print to a network printer, so that functionality now needs to reside on the server side.
With this new requirement, the agreement form is an RTF document that is being read, placeholder values replaced with values from the insert statement, written to a temporary file and then printed to the network printer via the application (Wordpad most likely) that is associated with the RTF file format.
There is also an MS Access front-end app that uses linked servers to connect to the database, but doesn't have the ability to create new users, but will need to be able to initiate the "print agreement" operation in the case of an agreement not being printed due to printer issue, network issue, etc.
I have the C# code written to perform the action of reading/modifying/writing/printing of the form that uses the UseShellExecute StartInfo property with the Process.Start method.
Since the file read/modify/write/print process takes a few seconds, I am concerned about having the stored procedure for adding the registration blocking for that length of time.
I am pretty sure that I am going to need a CLR stored procedure so that the MS Access front-end can initiate the print operation, so what I have come up with is either the "Add_Registration" stored procedure (Transact-SQL) will call the CLR stored procedure to do the read/modify/write/print operation, or an insert trigger (either CLR or Transact-SQL) on the table that calls the CLR stored procedure to read/modify/write/print.
I could avoid the call from the trigger to the stored procedure by duplicating the code in both the CLR trigger and the CLR stored procedure if there is a compelling reason to do so, but was trying to avoid having duplicate code if possible.
The solutions that I am currently considering are as follows, but am unsure of how SQL Server handles various scenarios:
A CLR or Transact-SQL Insert trigger on the registration table that calls a CLR stored procedure that does the reading/modifying/writing/printing process.
A CLR stored procedure that does the reading/modifying/writing/printing process, being called from the current add_registration Transact-SQL stored procedure
The questions I keep coming back to are:
How are Insert CLR triggers executed if multiple inserts are done at the same or nearly the same time (only 1 per operation), are they queued up an then processed synchronously or are they executed immediately?
Same question as #1 except with a Transact-SQL trigger
How are CLR stored procedures handled if they are called by multiple clients at the same or nearly the same time, are they queued up an then processed synchronously, or is each call to the stored procedure executed immediately?
Same question as #3 except with a Transact-SQL stored procedure
If a CLR stored procedure is called from a Transact-SQL trigger, is the trigger blocked until the stored procedure returns or is the call to the stored procedure spawned out to it's own process (or similar method) with the trigger returning immediately?
Same question as #5 except with a CLR trigger calling the CLR stored procedure
I am looking for any other suggestions and/or clarifications on how SQL Server handles these scenarios.
There is no queuing unless you implement it yourself, and there are a few ways to do that. So for multiple concurrent sessions, they are all acting independently. Of course, when it comes to writing to the DB (INSERT / UPDATE / DELETE / etc) then they clearly operate in the order in which they submit their requests.
I'm not sure why you are including the trigger in any of this, but as it relates to concurrency, triggers execute within a system-generated transaction initiated by the DML statement that fired the trigger. This doesn't mean that you will have single-threaded INSERTs, but if the trigger is calling a SQLCLR stored procedure that takes a second or two to complete (and the trigger cannot continue until the stored procedure completes / exits) then there are locks being held on the table for the duration of that operation. Those locks might not prevent other sessions from inserting at the same time, but you might have other operations attempting to modify that table that require a conflicting lock that will need to wait until the insert + trigger + SQLCLR proc operation completes. Of course, that might only be a problem if you have frequent inserts, but that depends on how often you expect new users, and that might not be frequent enough to worry about.
I'm also not sure why you need to print anything in that moment. It might be much simpler on a few levels if you simply have a flag / BIT column indicating whether or not the agreement has been printed, defaulted to 0. Then, you can have an external console app, scheduled via SQL Server Agent or Windows Scheduled Tasks, executed once every few minutes, that reads from the Users table WHERE HasPrintedAgreement = 0. Each row has the necessary fields for the replacement values, it prints each one, and upon printing, it updates that UserID setting HasPrintedAgreement = 1. You can even schedule this console app to execute once every minute if you always want the agreements immediately.
I am not a SQL developer, but I have a bit of SQL that is longer and more complex than my usual query/update. It is not a stored proc (policy thing).
(The code reads some state into variables, selects some data into temp tables, and performs some updates and deletes. There are a few loops to perform the deletes in small batches, as not to fill the transaction log.)
If I were writing code in Java I could create some test data and step through the method to watch the data being manipulated. Are there any tools that DB developers use to debug their code in a similar fashion?
You haven't provided any details on how your SQL is being processed, eg:
is your application submitting SQL batches to ASE? perhaps as a looping construct submitting prepared statements? [if so, you'll likely have a better chance of finding a debugger for your application; otherwise, add some print/select statements, perhaps based on a debug variable being set]
is your application submitting a (large) SQL batch to ASE? [if so, you may be able to use ASE's sqldbgr utility to step through your SQL code; you can find more details about ASE's sqldbgr in the ASE Utility Guide]
We have a very strange problem:
We have a stored procedure which returns a set of data. This procedure does a rather complex join and aggregation, so it takes 700 Milliseconds to execute.
When called directly in SQL Studio, the procedure ALWAYS returns correcty and always takes about 700 ms of time.
However, when called via a Client software (C# or Excel), then it works for a while, but then suddenly, the procedure takes 30 seconds (!) and gets a timeout. The number of disk reads is normal, but CPU is going up.
This behaviour goes on for 1-2 hours, and then it is normal again ! During this time, when the procedure show this behaviour, you can still execute the procedure in SQL Studio. Also, other and more simple procedures also get normally executed when called in the client.
I checked the calls with the profiler. When calling via a client, the call is marked as "RPC", when called in SQL Studio, it is "SQL"
So probably the procedure executes normal, but the data transmission via RPC somehow is hanging. But this is only a guess.
Does anyone have a idea, or can point us in some direction? I don't know where to look.
Also the Event log is empty.
Problem solved -- it was a fairly complex function, and compiling the execution plan took a lot of time. I simplified the procedure by removing a lot of conditional branches (if, case) and splitting the procedure into multiple sub-procedures.
.NET applications uses predefined (default) command timeout values.
This duration is most probably not enough for some of your commands and queries.
You may try to define your custom command timeout in your application.
I think you use a C# application as mentioned. Following definition in a constructor of your database context (EF code-first approach) may solve your issue.
public MyDBContext()
: base("MyDB")
{
((IObjectContextAdapter)this).ObjectContext.CommandTimeout = 300;
}
I've been using this definition in my C# based applications together with EF that executes sql server stored procedures.
Good luck!
I am taking an operating systems class where we just learned about the 'readers and writers' problem: how do you deal with multiple processes that want to read and write from the same memory (at the same time)? I'm also dealing with a version of this problem at work: I am writing an application that requires multiple users to read and write to a shared SQL server database. Because the 'readers and writers' problem seems so well understood and discussed, I'm assuming that Microsoft has solved it for me. Meaning, that I don't need to worry about setting permissions or configuring SQL server to ensure that people are not reading and writing to a database at the same time. Specifically, can I assume that, with SQL server 2005, by default:
No process reads while another process is writing
No two processes write at the same time
A writer will take an exclusive X lock on at least the row(s) they are modifying and will hold this until their transaction commits. X locks are incompatible with other X locks so two writers can not concurrently modify the same row.
A reader will (at default read committed isolation level) take a shared lock and release this as soon as the data is read. This is incompatible with an X lock so readers must wait for writing transactions to finish before reading modified data. SQL Server also has snapshot isolation in which readers are not blocked by writers but instead read an earlier version of the row.
Classic SQL Servers like MS-SQL use a pessimistic approach and lock rows, tables or pages until a writing operation is done. You really don't have to cope with that, because -- as you said -- the creator already solved that problem. Have a look at this article for some first information, any database book will cover the problem in depth. If you are interested in this topic, I would suggest reading "Database Systems" by Connolly and Begg, for example.