I have an SSIS package that is using an ADO NET Source component to select a number of rows from a database and inserting these records into another table.
The default command timeout for this component was 30 seconds, however the query can run for much longer than this 30 second period. Running some tests it seems that this timeout period is only to return the very first row. If it manages to do this then the query can run for 1000s of seconds if needed.
If it fails to return a row in the first 30 seconds then the pre-execute part of the component fails. Which I read is the period between starting to execute the query and the first row being returned.
This seems to conflict with anything I can find on google about this property. They seem to say that the command timeout period is to completely execute the query. That is not what I am seeing.
Can anyone clarify what this command timeout period is actually for?
EDIT:
Links to screenshots of the SSIS component and the property (CommandTimeout) that I am querying about.
Properties
Component
My question isn't about reducing the query time to be within this 30 second timeout period but why executions that take longer than 30 seconds are not timing out.
The 1000 second run of this package was a one of run that moved 17GB~ of data across servers, this isn't a normal run of the package but I was just using it to show that this package is clearly exceeding this 30 second command timeout.
It is a client side setting that the client uses to determine if a query is not responding. The client will then attempt to send a command to the server to cancel the query.
You are correct that the information available is a bit vague, even what is on MSDN. From my personal observations I agree with your findings that if the client starts to receive data from the server before the timeout then it can run for as long as it takes.
I think of it this way. When the first row of data is ready to be returned then the query itself is finished and only the movement of data is left to do. So, by definition the query did not time out.
Related
I have a scheduled SSIS job which runs daily. The job runs a select query on a SQL 2008 R2 server and then generates a text file. Generally, the job takes less than a minute to complete but sometimes it just takes hours to complete.
Symptoms :
1. The job gets stuck at "Pre-Execute phase is beginning" for just over 10 hours.
2. Then it starts processing the output file.
3. The job then immediately gets stuck at "Rows were provided to a data flow component as input" for another 10 hours.
I don't know what needs to be changed at package level because it works fine 90% of the time. Also, I tried to find a pattern and don't see any. The data being extracted remains almost same. There is no other connection to the table during the job run.
I have a SQL Server agent job running every 5 minutes with SSIS package from SSIS Catalog, that package does:
DELETE all existing data ON OLTP_DB
Extract data from Production DB
DELETE all existing data on OLAP_DB and then
Extract data transformed from OLTP_DB into OLAP_DB ...
PROBLEM
That job I mentioned above is hanging randomly for some reason that I don't know,
I just realize using the activity monitor, every time it hangs it shows something like:
and if I try to run any query against that database it does not response just say executing.... and nothing happen until I stop the job.
The average running time for that job is 5 or 6 minutes, but when it hangs it can stay running for days if I don´t stop it. :(
WHAT I HAVE DONE
Set delayValidation : True
Improved my queries
No transactions running
No locking or blocking (I guess)
Rebuild and organize index
Ran DBCC FREEPROCCACHE
Ran DBCC FREESESSIONCACHE
ETC.....
My settings:
Recovery Mode Simple
SSIS OLE DB Destination
1-Keep Identity (checked)
2-Keep Nulls (checked)
3-Table lock (checked)
4-Check constraints (unchecked)
rows per batch (blank)
Maximum insert commit size(2147483647)
Note:
I have another job running a ssis package as well (small) in the same instance but different databases and when the main ETL mentioned above hangs then this small one sometimes, that is why I think the problem is with the instance (I guess).
I'm open to provide more information as need it.
Any assistance or help would be really appreciated!
As Joeren Mostert said, it's showing CXPACKET which means that it's executing some work in parallel. (cxpacket)
It's also showing ASYNC_NETWORK_IO (async_network_io) which means it's also transfering data to the network.
There could be many reasons. Just a few more hints:
- Have you checked if network connection is slow? - What is the size of the data being transfered vs the speed of the network? - Is there an antivirus running that could slow the data transfer?
My guess is that there is lots of data to transfer and that it's taking a long time. I would think either I/O or network but since you have an asyn_network_io that takes most of the cumulative wait time, I would go for network.
As #Jeroen Mostert and #Danielle Paquette-Harvey Said, By doing right click using the activity monitor I could figure out that I had an object that was executing in parallel (for some reason in the past), to fix the problem I remove the parallel structure and put everything to run in one batch.
Now it is working like a charm!!
Before:
After:
I have used the ADO-function from another website to make a query from Excel VBA 2016 on my SQL Server Express database on the same workstation. The SQL database is continuously updated with new records from a feed provider.
The query runs very smoothly and takes around 70 milliseconds to execute. I call the query every 3 seconds from a routine in Excel VBA. Although once or twice a minute the query takes 6000-12000 milliseconds. It seems the query is "hanging" at these moments for some reason.
When I turn off the external feed, the issue is not there!
I also have used Express Profiler to get insight in the cause. I have added the output below this post. Lines 33 and 173 are lines with a long interval. As you can see this duration is much longer than the other (identical queries).
I have set the READ_COMMIT_SNAPSHOT ON in options.
I have also set ALLOW_SNAPSHOT_ISOLATION ON in options.
I did not close down any connections, although SQL did give a message that it was closing all connections.
Although after testing the problem is still there.
After reading this:
https://learn.microsoft.com/en-us/sql/t-sql/statements/set-transaction-isolation-level-transact-sql I have tried using the command SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED but, after testing, the problem is still there.
Does anyone know a solution to this problem?
I am writing some code that is importing a large amount of data into three tables currently around 6 million rows across the three tables. I am wanting to do this in a transaction so if there are any issues or the user cancels the import nothing is imported. This works fine on my own development machine however on a slower amazon ec2 instance and micro sql instance I am getting the following exception:
System.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding
Now I know that the commit is finishing eventually because the data is present in the tables when I look, so my question is; can this be easily avoided without adding the connection timeout property to my connection string (I only want this one operation to not timeout) or is this a really hard/dangerous thing to be doing?
I am not sure if maybe I should import into holding tables and then call stored procedures to move the data when I am ready because I would assume this will result in a shorter transaction)
I am using Ms Sql server 2012.
Do comment if I need to add more data.
Many thanks for your help
Check what is the SP getting timedout .. if you have any third party tool like Redgate or Avicode you can figure it out ..or use Profiler to figure it out.. then see the execution plan for the SP or query .. If you find any Key lookups or RID lookups then resolve it first and try again..
I have a reporting solution with several reports. Up to now, I have been able to add a dataset based on a SPROC with no problems. However, when I try to add the lastest dataset, and use a SPROC for its query type, when I click on Refresh Fields I get the following error:
Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
I have tested the database connection in Data Source properties>Edit>Test Connection, and it's working fine.
I have increased the timeout to 100 in the following areas:
The connection string for the datasource, which is - Connect
Timeout=100
Tools>Options>Database Tools>Query and View
Designers. Cancel long running query is set to 100.
Tools>Options>Database Tools>Table and Database Designers>Checked
'Override Connection String time-out value for table designer
updates. Transaction time-out after is set to 100
The SPROC runs fine in the SQL database. It takes about 55 seconds.
Any other ideas?
Thanks.
UPDATE: I now can't add any dataset with a SPROC. Even thought the SPROCs are all working fine in SQL!!!!!!
If you are using Report Builder, you can increase timeout also in your DataSet.
I have also face same problem for adding the newly added column in stored procedure.
From the following way overcome this issue.
Alter the stored procedure as comment all the query except that final select command.
Now that new column has been added, then un-comment that quires in sp.
The thing to remember with your report is that when it is ran, it will attempt to run ALL the datasets just to make sure the are runnable, and the data they are requesting can be returned. So by running the each proc seperately you are in fact not duplicating with SSRS is trying to do...and to be honest don't bother.
What you could try is running sp_who while the report is running, or even just manually go through the procedures to see what table they have in common. Since your proc takes 52 seconds to return its dataset I'm going to assume its doing some heavy lifting. Without the queries nobody will be able to tell what the exact problem is.
I suggest using NO LOCK to see if that resolves your issue. If it does then your procs are fighting for data and blocking each other...possibly in a endless loop. Using NO LOCK is NOT a fix. Read what it does and judge for yourself however.
My solution was to go to the Dataset Properties for the given problem dataset, paste the query in the Query field, click Refresh Fields, and click Ok.