I'm trying to write a program that is able to import and export a specific database, plus users and logins, from a Microsoft SQL Server database. I have an abstraction layer between my code and ODBC that much of our other software uses. The abstraction layer normally runs with autocommit off and handles transactions on its own, but since the BACKUP command doesn't like being run in any transaction, I"m using another method of the layer, called executeDirect, that runs it with autocommit on.
The method uses the SQLExecDirect function to run the BACKUP command. When it's done, the function returns SUCCESS_WITH_INFO, since the BACKUP command likes to give three lines of output. The code then attempts to get the output with SQLGetDiagField and is able to gather the first line from record #1, but there is no record #2.
The last thing the method wants to do is reset the connection to autocommit off, but when it attempts that, an error occurs, saying "Connection is busy with results from another command", SQL State is "HY000". So, obviously, the connection wants to remit the other two lines of output, but I don't know how to let it do so.
The BACKUP DATABASE command works as a batch, requiring a call to SQLMoreResults to continue with the backup.
Related
I am dealing with my problem on some Windows Server 2019 (Core) with one running SQL Server 2019 CU4 instance each.
What we try to do
We are currently building a data warehouse with distributed databases. The individual layers of the DWH are located on one database server each. The data exchange between the layers/servers takes place via SSIS ETLs, which use Linked Servers to reach the other layers and drag and drop data. Each layer also has its own SSIS service instance and executes the corresponding SSIS packets.
The SSIS packages are called by SQL Server Agent jobs. We have a job that executes the SSIS packets (#1), which in turn calls another job (#2) as the last step, which after a short wait time executes the calling job (#1). Thus, controlled by schedules, a loop is created and data is continuously transferred with ETLs.
I hope this was not too much unnecessary background
The error
Basically the job is running and there are numerous successful executions. However, we are observing interruptions at job #1 without helpful information regarding the error. This means that the job history log refers to the SSIS log, which again only contains an "unexpected termination". In the SSIS log, we only see behavior that indicates that the ETL packet active at that time stopped after validation. Depending on the log level, nothing is logged at all, not even the execution of single packages of the project. The package where this error occurs is different and not limited to a specific one.
What I have already tried
Re-create the jobs and SSIS Enviroments by hand (scripted before)
Using the 32Bit Runtime
Upgrade the SSIS project/package version to
2019
Increase the log level to "verbose"
Patching the SQL Server to CU4
Save ssis dump files (couldn't find them or they weren't created)
Search Windows and SQL Server Logfiles
Does anyone have some suggestions or some ideas how to become more error specific informations?
Thank you very much and take care :)
UPDATE We have an error message (OLE DB 0xC0202009 and 0X80004005)!
In order to exclude the use of environments as a cause, I manually set the parameters in the SSIS job step instead of overwriting them by selecting an environment.
Long story short: Today it turns out that the parameter for an OLE DB Connection String is not passed correctly.
The following is specified as a parameter in the job step:
However, the following connection string is specified in the context of the error message:
Please note that some arguments are added twice to the parameter (red).
What could have caused that?
I have a SQL Server service and multiple Windows Service doing some backgrounds work on same server.
One of them (I'm calling it "A") have a routine executing "single_user/offline" and "online/multi_user" to active databases to do some backup operations at midnight. The another ones executes multiple queries over that databases (I'm calling it "B").
The problem is the following:
1.- Windows Service "A" executes SET ONLINE.
2.- Windows Service "B" executes a random SELECT.
3.- Windows Service "A" tries to execute SET MULTI_USER. This execution is dropped because there is an active connection made from Windows Service "B".
I've tried executing SET ONLINE and SET MULTI_USER on same CommandText of the SqlCommand, but this doesn't denies the incoming query from Windows Service "B", breaking my process and keeping the database locked (because the SINGLE_USER).
How can I make an ONLINE and MULTI_USER commands at same time on the Windows Service "A" to make Windows Service "B" being cancelled or wait the process finished? (It's not a problem that Windows Service "B" being cancelled)
Could be sp_dettach_db or sp_attach_db useful?
It sounds like Service A is closing it's connection after bringing the database into Single User mode. This frees up Service B to become the single user, at which point Service A can no longer change the mode until it can grab the single connection back, which it won't be able to do as long as Service B, or any other client for that matter, has it.
I can think of a couple of things you could do here:
Once your offline operations are complete, begin polling until Service A can become the single user again
Find the SPID of the connection from Service B and kill it.
See the limitations and restrictions section about single user mode at this link for more info.
Thanks to all, but I reanalyzed the problem I was having from the beginning, and concluded that the action "SET SINGLE_USER" was not necessary, since the process does not require actions in that mode, so finally with the action "SET OFFLINE" and "SET ONLINE" could prevent intermediate connection problems.
I use RODBC to send queries to an SQL-Server. Sometimes they take too much time to run, so I need to cancel them.
Clicking the red "stop" button in RStudio yields this error message:
R is not responding to your request to interrupt processing so to stop
the current operation you may need to terminate R entirely.
Terminating R will cause your R session to immediately abort. Active
computations will be interrupted and unsaved source file changes and
workspace objects will be discarded.
Do you want to terminate R now?
And if I click yes my session is indeed terminated. (note: using Rgui instead of RStudio doesn't make things better)
However:
when I use another software (named "Query ExPlus") to connect to this same SQL-Server, I have a similar stop button, and clicking it instantly interrupts the query, without any crash.
when I connect to a PostgreSQL database using the RPostgres package I can also interrupt the query at any time.
These two points lead me to think that there should be a way to solve my problem. What can I do?
So far my workaround is:
library(RODBC)
library(R.utils)
withTimeout(mydf <- sqlQuery(myconnection, myquery), timeout=120)
Note: I don't have permission to kill queries from the database side.
I've just stumbled upon the odbc package. It allows to interrupt a query at any time.
Basic usage goes like this:
library(DBI)
myconnection <- dbConnect(odbc::odbc(),
driver = "SQL Server",
server = "my_server_IP_address",
database = "my_DB_name",
uid = "my_user_id",
pwd = "my_password")
dbGetQuery(myconnection, myquery)
I don't have a deep understanding of what happens behind the scenes, but for what I've seen so far in my personal use this package has other advantages over RODBC:
really faster
get the column types from the DB instead of guessing them (see here)
no stringsAsFactors and as.is arguments necessary
Most SQL Server users use SQL Server Management Studio (which is free and can be downloaded from Microsoft) to connect to SQL Server or execute commands from the command line via a tool called SQLCMD.
If you can determine the session id that the SQL Command is being run in you can kill the session which would stop any executing command(s). SQL Server will still need time (could be a 'long' time) to rollback any changes made during the execution of the command.
Terminating a session (depending on the software) can take a while to communicate to SQL Server that the session has been terminated. When I connected to DB2 from SQL Server using linked servers DB2 would buffer the terminate command and it would frequently take up to an hour for DB2 to realize the session had been terminated.
To determine what the session you are running in you can try:
select ##spid;
once you have the spid (lets say 86)
you can then issue (depending on if you have permission to do so)
kill 86;
but as Microsoft notes:
Terminates a user process that is based on the session ID or unit of work (UOW). If the specified session ID or UOW has a lot of work to undo, the KILL statement may take some time to complete, particularly when it involves rolling back a long transaction.
Try to close your "tab query" on SQL Server Management Studio
Then it will appear pop-up,
This Query is currently executing. Do you want to cancel this query ?
Cancel anyway, choose "yes".
try to set your connection prior to query:
sql = odbcConnect('Database name')
Then use same line to run your query:
mydf <- sqlQuery(sql, " myquery ")
Note: The running time is dependant on both database and R server but setting up the connection this way should resolve termination problem.
I am doing restores across servers via dynamic sql and linked servers as follows:
Exec ('USE MASTER; RESTORE DATABASE <dbname> FROM DISK = <path> WITH REPLACE, RECOVERY') AT <target server>
The database is set to single user mode before the above statement runs. However, on intermittent occasions, when trying to get the database back to multi user mode I get the a query timeout from the the target server and error:
ALTER DATABASE is not permitted while a database is in the Restoring state
I have looked around for this error but apparently I'm should not be getting it when using the RECOVERY option.
Any ideas please?
I managed to figure this out today. The culprit behind the issue was the default remote query time out for Sql Server which is 600s (10mins). More on this in the following link https://msdn.microsoft.com/en-us/library/ms189040(v=sql.105).aspx
My restore usually takes around 10mins to complete, sometimes a bit less so it succeeds and sometimes a bit more when it fails. Hence the issue was manifesting itself intermittently.
I increased the remote query time out on the source server to 36000s and it restored successfully. Just to prove the point, I decreased the remote query time out to a small figure like 60s and got the same symptom again.
Michelle - I set the database in single user to kill any running queries before the restore and to make sure there are no conflicts whilst the database is restored. This would eliminate the Exclusive Access could not be obtained because the database is in use error. Alternatively the database can be put to offline before the restore is initiated and back to online afterwards.
I had a package that worked perfectly until i decided to put some of its tasks inside a sequence container (More on why I wanted to do that - How to make a SSIS transaction in my case?).
Now, i keep on getting an error -
[Execute SQL Task] Error: Failed to acquire connection "MyDatabase". Connection may not be configured correctly or you may not have the right permissions on this connection.
Why could this be happening and how do I fix it ?
I started writing my own examples to reply to your question. Then I remember that I met Matt Mason when I talked at a SQL Saturday in New Hampshire. He is the Microsoft Program Manager for SSIS.
While I spent 3 years between 2009 and 2011 writing nothing else but ETL code, I figured Matt had an article out there.
http://www.mattmasson.com/2011/12/design-pattern-avoiding-transactions/
Here is a high level summary of the approaches and the error you found.
[ERROR]
The error you found is related to MSDTC having issues. This must be configured and working correctly without any issues. Common issues are firewalls. Check out this post.
http://social.msdn.microsoft.com/Forums/sqlserver/en-US/3a5c847e-9c7e-4628-b857-4e6edaa7936c/sql-task-transaction-required?forum=sqlintegrationservices
[SOLUTION 1] - Use transactions at the package, task or container level.
Some data providers do not support MSDTC. Some tasks do not support transactions. This may be slow in performance since you are adding a new layer to support two phase commits.
http://technet.microsoft.com/en-us/library/aa213066(v=sql.80).aspx
[SOLUTION 2] - Use the following tasks.
A - BEGIN TRAN (EXECUTE SQL)
B - YOUR DATA FLOW
C - TEST THE RETURN CODE
1 - GOOD = COMMIT (EXECUTE SQL)
2 - FAILURE = ROLLBACK (EXECUTE SQL)
You must have the RetainSameConnection property set to True on the connection.
This forces all calls thru one session or SPID. All transaction management is now on the server.
[SOLUTION 3] - Write all you code so that it is restartable. This does not mean you go out and use check points.
One solution is to always use UPSERTS. Insert new data. Update old data. Deletes are only a flag in a table. This pattern allows a failed job to be executed many times with the same final state being achieved.
Another solution is to handle all error rows by placing them into a hospital table for manual inspection, correction, and insertion.
Why not use a database snapshot (keeps track of just changed records)? Take a snapshot before the ETL job. If an error occurs, restore the database from the snapshot. Last step is to remove the snapshot from the system to clean up house.
In short, I hope this is enough ideas to help you out.
While the transaction option is nice, it does have some down falls. If you need an example, just ping me.
Sincerely
J
What package protection level are you using? Don't Save Sensitive? Encrypt Sensitive with User Key? I'd recommend changing it to use Encrypt Sensitive with Password and enter a password. The password won't disappear.
Have you tried testing the connection to the database in the connection manager?