I'm working on a project for a client that involves a pretty huge ETL process to move data from MSSQL Server to Postgres. We are using SSIS 2014 with the ODBC drivers provided on postgresql.org, and have setup an ODBC DSN. We are only using built in OLEDB Sources and ODBC Destinations, and we are running into an issue that I have not been able to find referenced anywhere else online.
The exact issue is that SSIS seems to open multiple connections for each ODBC destination data flow component, even with connection pooling enabled. This can result in 50+ idle connections being opened, which are not killed until the process is completed. Previewing data from Visual Studio causes connection leaks as well which will only be killed upon restarting Visual Studio. We have temporarily resolved this by increasing the maximum connections to 1000, but we are hoping to fix the underlying problem if possible.
I've done a decent amount of experimentation, and the issue seems to be SSIS related as opposed to an error in the driver.
Has anyone else run into a similar issue and know how to resolve this?
EDIT: Thought this wasn't going to be a big issue, but now I realize that ODBC connections are leaking when the SSIS package is run from Integration Services as well. I've played with it a bit by making some empty packages and adding only a single ODBC source and also a package which only accesses the database from C# script task; only the ODBC source/destinations are causing leaking and not script components, so it seems like a bug in SSIS and not my script tasks or the postgres driver :O
Anyone have any idea how to resolve this besides rewriting the whole package to not use ODBC sources/destinations (or some other weird thing like killing all connections afterwards with a shell script)?
Please find the statement timeout
must give statement timeout in secs on odbc destination properties, default value 0 is there it means infinity in SSIS
Related
I have packages deployed on a sql server 2008R2 and recently, we migrated to a new server machine, deployed with sql server 2012. I configured packages to project deployment mode and for 10 days, all packages are working smoothly, with the execution times in the same range of older server.
Since last two days, packages started to fail. I checked in detail and found that, they are taking longer time than usual, and fail due to "Protocol error in TDS stream, communication link failure and remote host forcibly closed the connection".
When I tried to run the package through ssdt, they can run successfully, but I see data transfer movement slower than I used to see, and so package execution time is much longer.
I am not sure, what has changed. I have searched the internet for the possible reason and checked the server memory and packet size, and tried match with the older server, which did not solve the problem. I suspect, SSIS logging may have causes this, but not sure how to check it?
Please help to identify the cause of this problem.
**Edit: I enabled logging in ssdt and could see that majority of time is used in rows transfer steps only. Since my package have look ups, I thought that look ups might make it slower somehow. So copied the main query to ssms and run as a normal query on this server.
About 13L rows were returned in 12 minutes. Then I run the same query on the old server, there it returned 13L rows in less than a minute. So, possibly it proves the problem somehow is related with data transfer and not specific to packages itself.
Can Someone help please.**
Just check the solution connection, it should be ‘RetainSameConnection’ property to 'true'. This can be done both in the SSIS package under connection manager properties and in the job step properties (Configuration > Connection Managers).
Link: http://www.sqlerudition.com/what-is-the-retainsameconnection-property-of-oledb-connection-in-ssis/
First of all, here is our setup:
SQL Server 2014 SP2 Standard Edition 64-bit
128GB of RAM (maximum allowed by Standard Edition), of which 120GB is allocated to SQL Server
The server currently hosts ~5000 databases which are all similar (same tables, stored proc, etc.) for a total of 690GB of data (mdf files only)
Now what happens:
Every now and then, after the server has been up for some time, we receive this error when executing queries on some databases:
.NET Framework execution was aborted by escalation policy because of out of memory
This error happens more often when we perform an update of all client databases (when we launch a feature) using Red Gate SQL Multi Script. Of the ~5000 DBS, we have the error on 70 of them. Running the update script again, the error happens on a portion, and so on until we have all databases updated correctly. This is just annoying.
We have this error since a long time. Our server had 64GB of RAM, so we just added more memory to max out SQL Server Standard Editor, but still, the error came back a few days later. We think the error might be a symptom of something else.
A few things that might help to get an answer:
Our version of SQL Server is 64-bit, so we think we don't have to deal with Virtual Address Space Reservation
The error also happens when running from a client app written in PHP/Linux, so we're not talking about .NET framework of the client code
In our databases, the only usage of .NET framework we make is GROUP_CONCAT, a.NET CLR Assembly with User Defined Functions which help us simulate MySQL GROUP_CONCAT aggregate. We have a copy of the assembly in each of our 5000 client databases.
We already tried to lower the max server memory setting (to 96GB), but we were still getting those errors
If more info is needed I will update my question.
It's been 4 months since I tried a fix, and I did not experienced the bug again. Still, I don't have the exact explanation for the bug, but here is what I tried and it seems to work:
My guess was that having the same .NET CLR Assembly in each of our 5000+ databases might be the problem and increased memory usage for .NET in some way.
I created a new database named something like DotNetClrUtils
I copied the .NET CLR Assembly for GROUP_CONCAT in this database
I changed all usage of GROUP_CONCAT in all our client code and stored procedures to reference the single instance in DotNetClrUtils database (calling it like this: DotNetClrUtils.dbo.GROUP_CONCAT_D(col, ','))
That's all, and now this problem is gone!
I have an SSIS package that moves data from SQL server to a Postgres 9.5 database. I'm using the latest Postgres ODBC drivers to connect to the postgres server. The bulk of the operations are inserts and updates and upserts. When I run this package from my development machine (Win 10 64-bit) in Visual Studio 2012 it's quite speedy. It can transfer about 80k rows in around 40 seconds.
When I deploy it to the server (an SQL Server 2012 instance) and run it using SQL management studio it executes painfully slow. It took around 20 seconds to transfer less than 10 rows, and takes forever to work on the full data set. I never let if finish because it just takes too long.
Both my development machine and the server have the exact same postgres driver installed and identically configured ODBC sources.
Edit: I should also note that I have other packages deployed to that server that run just fine, though these packages don't touch postgres or use ODBC for anything.
If all else is equal:
Same drivers
Same configurations - ODBC (i.e. ssl, same boxes ticked), package parameters
Same tables - with same amount of data in them and indexes
Same bit-ness (64x)
I would look towards resource differences. I.e.
is the IO the same. Assuming this is writing to the same database (and files), then this should not be an issue.
Memory - is the ram constrained on the dev instance? If the dataflow has large buffers this could really slow things down.
CPU - is the dev server more limited than your dev machine
Network - is the dev server sitting in a different subnet with differing QoS. I would guess this is not an issue since other packages are not affected. Unless there is something weird when connecting to Postgres.
Another way to zero in on the issue would be to run different versions of the package, stripping out parts of the dataflow. i.e. remove the odbc / ado.net destination and run the package. If it finishes quickly, you know the issue is there. If it is still slow, keep going upstream until you've identified the component that's slow.
Lastly, I would consider using psql over odbc. psql is a postgres utility, like SQL's bcp, which would allow you to bulk copy data to postgres. The odbc driver will only allow row by row inserts which tend to be sluggish. Writing the data to a tab delimited file and then using psql to bulk copy the data into postgres is actually significantly faster (80k rows might take 5s all told).
I've done a migration from SQL 2008 to SQL 2014. Unfortunately, one of the SSIS package which takes only 6 hours to run on 2008 is now taking 8 hours on 2014.
Can somebody told me why this is happening and how can I solve this problem? Is it something to do with setting?
I appreciate any idea/help from you guys. Thanks in advance.
Could be some problems:
Check the operating system is the same data SQL 2008.
Check the memory SQL SERVER:
Right-click: Server properties -> Memory -> Maximum Server Memory
Sometimes the virtual team, lowers the CPU consumption for the benefit of another machine
(If this is a virtual machine).
What about logging?
In 2012, the concept of project deployments was born. In addition to that concept, a centralized SSIS database was created by default when as Integration Services server was installed. Are you deploying the packages to a server to be run? If so, then logging might slow you down. http://msdn.microsoft.com/en-us/library/hh231191.aspx especially if the default it set to verbose and/or you're doing your own custom logging ( for each event, two executions happen).
Your SSIS server may be drowning from the default logging in addition to the standard workload of the data movements in the package. Try turning logging down or off. Basic works well for us. While the package is executing, monitor any resources that are running too high. That could give you some hints about potential bottle necks and where else to look.
I'm running Sql Server Management Studio 2008 on a decent machine. Even if it is the only thing open with no other connections to the database, anything that has to do with the Database Diagram or simple schema changes in a designer take up to 10 minutes to complete and SQL Management Studio is unresponsive during that time. The same SQL code takes less than a second. This entirely defeats the purpose of the designers and diagramers.
------------------
System Information
------------------
Operating System: Windows Vista™ Ultimate (6.0, Build 6001) Service Pack 1 (6001.vistasp1_gdr.080917-1612)
Processor: Intel(R) Core(TM)2 Quad CPU Q6700 # 2.66GHz (4 CPUs), ~2.7GHz
Memory: 6142MB RAM
Please tell me this isn't a WOW64 problem; if it is, I love MS, but step up your 64-bit support in development tools.
Is there anything I can do to get the performance anywhere near acceptable?
Edit:
I've got version 10.0.1600.22 of SQL Server Management Studio installed. Is this not the latest release? I'm sure I installed it from an MSDN CD and I pretty much rely on Windows Update these days. Is there any place I can quickly see what the latest release version number is for tools like this?
Edit:
Every time I go to open a database diagram I get the message "This database does not have one or more of the support objects required to use database diagramming. Do you wish to create them?" I say yes every time. Is this part of the problem? Also, if I press the copy icon, I get the message "Current thread must be set to single thread apartment (STA) mode before OLE calls can be made." Database corruption?
I'm running in a similar environment and not having that problem.
As with any performance problem, you'll have to analyze it a bit - just saying "it takes 10 minutes" give no information on the reason it takes so long, so no information you can use to solve the problem.
Here are some tools to play around with. I'd have mentioned them originally, but "play around" is all I've learned to do with them. I'd recommend you try learning a little about them, which I have not done. http://technet.microsoft.com is a good source on performance issues.
Start with Task Manager, believe it or not. It's been enhanced in Vista and Server 2008, and now has a better Performance tab, and a Services tab. Be sure to click "Show processes from all users", or you'll miss nasty things done by services.
The bottom of the Performance tab has a "Resource Monitor" button. Click it, watch it, learn what it can do for you.
The Resource Monitor is actually part of a larger "Reliability and Performance Monitor" tool in Administrative Tools. Try it. It even includes the new version of perfmon, which will be more useful when you have a better idea what counters to look at.
I will also suggest the Process Explorer and Process Monitor tools from Sysinternals. See http://technet.microsoft.com/en-us/sysinternals/default.aspx.
Do your simple schema changes possibly mean that you're reordering the columns of a table?
In that case, what SQL Management Studio does behind the scenes is create a new table, move all the data from the old table to the newly created table, and then drop the old table.
Thus, if you reorder columns on a table with lots of data, lots of indices or both, you CAN incur a massive amount of "reorganization" work without really realizing it.
Marc
Can you try connecting your SQL Management Studio to a different instance of SQL Server or, better, an instance on a remote machine (and try to make similar changes)?
Are there any entries in the System or Application Event Logs (or SQL logs for that matter)? Have you tried uninstalling and reinstalling SQL Server on your machine? What version of SQL Server (database) are you running?
Lastly, can you open the Activity Monitor successfully? Right click on the server (machine name) - top of the three in the object explorer window - and click on 'Activity Monitor'.
Do you have problems with other software on your machine or only with SQL Server & Management Studio?
When you open SSMS it attempts to validate itself with Microsoft. You can speed this process by performing the second of the recommendations at the following link.
http://www.sql-server-performance.com/faq/sql_server_management_studio_load_time_p1.aspx
Also, are you using the registered servers feature? If so SSMS will attempt to validate all of these.
It seems as though it was a network configuration problem. Never trust a developer (myself) to setup a haphazard domain at his office.
I had my DNS server on my computer pointed to my ISP's (default because the wireless router we're using provided by the ISP doesn't allow me to override the DNS server to my own) instead of my DNS server here, so I have to remember to configure it manually on each computer, which I forgot for this particular computer.
I only discovered it when I tried to connect for the first time to a remote SQL Server instance form this PC. It was trying to resolve to an actual sub-domain of mycompany.com instead of my DNS server's authority of COMPUTERNAME.corp.mycompany.com
I can't say why this was an issue for the designers in SQL Server but not anything else, but my only hypothesis is that when I established a connection to my own computer locally using the computer name instead of "." or "localhost", SQL queries executed immediately, knowing it was local, but the designers still waited for a timeout from the external IP address before trying the local one.
Whatever the explanation is, changing my DNS server for my network card on the local machine to my DNS server's IP made it all work very quickly.
I had a similar issue with mine. Turned out to be some interference with the biometrics login service running on my laptop. Disabled the service and now it works.