We a project to rewrite a legacy system built on SQL Server. The system is making use of various databases (+-10) and the related SQL code in them (sp, functions,...). The challenge for me at the moment is to get these databases in source control (TFS).
Have tried to make use of a Visual Studio Database project, looked very promising at the beginning but after we added most of the databases, the VS project got unusably slow or VS 2017 just crashes.
We split the databases into different solutions, this at least stopped VS from crashing, but the speed is still a very big issue.
Is there a way to speed things up? Have searched, but could not find any way of disabling some of the VS synchronization checks, that I assume in the cause for the slow speed.
My worry is that the compare needed from the DBA team later in the process to build the dacpac or just change SQL will take too long. Looking for a better solution that is suitable for production deployments?
Related
I'm trying to update a SQL Server project in Visual Studio 2019 by using the SSDT schema comparison. My source is a running database server, the destination is the VS SQL Server project.
When the comparison is done and I click "Update", I get the message
Source schema drift detected. Press Compare to refresh the comparison
No matter how many times I refresh the comparison, I always get the same result.
I have tried various connection tweaks (read-only intent, asynchronous processing, multiple active result sets) in the hopes that I can make the comparison run faster and update the project before the drift happens, but to no avail. I have also tried reducing the types of objects included in the comparison, but have not been able to reduce it enough to prevent drift from being detected.
I think the biggest issue I have is that aside from the "schema drift detected" message, I feel like I'm shooting in the dark. By that I mean that I have no idea what is causing SSDT to detect drift, and therefore I can't work around it.
I tried running the SQL Profiler to capture what SSDT is doing so I could find where SSDT is detecting drift. However, I haven't been able to find any query that gives different results when run multiple times within a short period.
So in conclusion, my questions are:
What does SSDT look at to determine when the database schema has drifted?
How can I update my SQL Server project when it always detects schema drift?
I also struggled for months to find the cause of the same error. I was already thinking about flashing Windows 10 on my laptop. I won't list the dead ends anymore. In my final desperation, I copied the SQL Server database and VS project to another machine, and there the comparison worked without a bone. The suspicion arose that maybe the error is not in VS, but rather that my SQL server is confusing VS.
I have a SQL Server 2012. I put the latest update on it (SP4) and wonder of wonders, compare and sync started working perfectly right away. Of course, now before every update I pray a little so that I don't encounter the "Source schema drift detected" message.
I have been unsuccessfully fighting this annoying error for MANY SSDT versions.
Searching for it you will see multiple places where it is claimed to be fixed, WHICH IS FALSE, as it is happening right now with VS 2022 SSDT.
In my case, it ONLY happens when comparing against ONE out of the 5 database servers I regularly use the tool with.
The only workaround I have found that usually works is to REBOOT the destination database server (NOT just cycle the SQL Server Service) and then run the SSDT compare QUICKLY!
As the server that this happens on is an integration server running on a VM in my local network, I can bounce the server, but in other scenarios this would be a show-stopper.
IMO the most onerous things about this issue is that you cannot even generate the script to copy / paste into SSMS, which is how I often use the tool.
This issue has not been fixed for YEARS and is very intermittent, so I have no hope of seeing it actually fixed - I hope this workaround is helpful to someone.
We need to move an enterprise ERP during the upgrade (from 2005 to 2008). I have done some reading regarding the merits of running compatibility mode and I know there are some differences in the SQL estimator running native vs. compatibility mode, but I was wondering if any of you have encountered any performance improvements running a SQL database in compatibility mode on a newer server, i.e. are there any papers or actual experience that suggest that I am going to get better performance running SQL2008 vs. SQL2014 with Compat mode on the database. Do I actually benefit from the new server. We are licensed either way and the ERP is only "guaranteed" on 2008.
I hope to get some feedback for anyone who has run into this problem before. (Compatibility has been around for a long time, so I am sure someone has). Considering that our databases are ~400GB, clustered and SAN'd makes a really real-world test somewhat difficult to really do. Even more-so that the SAN will "prioritise" things - just make the test even more difficult. We all know that SQL 2014 performs better than 2012, but with the poorest of data, it may be the case - hence the general request.
I have never run into any issues with compatibility mode for any version of SQL Server. I also haven't really noticed any performance benefits or drawbacks doing so, but I admit that I haven't done any real timing tests. Usually when I've had to do that, I've upgraded the hardware on the box, so a true comparison is difficult.
Having said that, are you sure that's the way you want to go?
Why not just run a test environment with the database migrated to 2008 and no compatibility? If everything works for you in the test environment, then upgrade directly.
Most SQL Server upgrades are pretty painless, unless you're trying to skip a version or two, which you aren't. Even in failover clusters they aren't that hard as long as you follow the step-by-step procedure from Microsoft.
Some people don't like SSIS for the following reasons,
Need to find and click the express replacement scattered in different place when design a little bit more complex package.
These merge, lookup components don't perform well. I heard a lot of consultants just recommend loading data in the SQL Server tables and use transact-sql.
I've used a powershell in a small project which export the data and create csv files. I've used powershell and like it. Is it a trend to replace some of the tasks traditionally using SSIS with Powershell? Especially in the export only cases?
For very small projects/tasks power shell is an ok tool.
For projects that need to be robust, maintainable, modular, handle errors and auditing, SSIS is vastly superior.
The truth is, too many SSIS implementations are crafted by devs that don't understand the strengths of the program. They simply try to replicate their current T-SQL ETL process into SSIS with minimal effort or leverage of its capabilities. Performance issues almost always go right along with this.
SSIS is Not just a GUI way to get SPs and TSQL to autorun. If you really want to learn more on the subject I suggest picking up a few books - careful listening to narrow-fielded experts; their skillsets can easily fade from relevance and keep others behind with them.
Powershell trend away from SSIS? Not anywhere close to where it counts.
This is an old topic but I find it is well worth discussing. So I'd like to put a few ideas why I think SSIS is a bad choice 99% of the time as a ETL tool.
At this time, the only thing I can think of SSIS better than PowerShell is its performance in handling huge amount of data with multiple sources / targets, this is mainly due to the SSIS internal parallelism and caching capability.
However, SSIS is notorious for its error message, almost unable to debug once SSIS packages are deployed, also from source control perspective, SSIS packages, which are XML files, are difficult to compare between versions, also very fragile once either source or target objects have minor changes (like a column on target is increased by one char).
In my prod environment, there are many SSIS packages deployed and scheduled with sql agent jobs, so when there is a job failure, there is no way for me to figure out the problem until I went to TFS to find the SSIS project and open it in Visual Studio to figure out the logic. It is a nightmare.
With PowerShell, the code you see is the code executed, and you can always get the logic from the PS code and do the trouble-shooting along the way.
With many, many open-sourced PS modules these days, PowerShell's power is increasing exponentially, it is indeed the time to consider using PS as an alternative tool rather than the SSIS.
Could anybody tell me what the current trend for SQL Server Integration Services is? Is it better than other ETL tools available in market like Informatica, Cognos, etc?
I was introduced to SSIS a couple of weeks ago. Executive summary: I am unlikey to consider it for future projects.
I'm pretty sure flow charts (i.e. non-structured) were discredited as an effective programming paradigm a long time, except in a tiny minority of cases.
There's no point replacing a clean textual (source code) interface with a colourful connect-the-dots one if the user still needs to think like a programmer to know where to drag the arrows.
A program design that you can't access (e.g. fulltext search, navigate using alternative methods, effectively version control, ...) except by one prescribed method is a massive productivity killer. And a wonderful source of RSI.
It's possible there is a particular niche where it's just right, but I imagine most ETL tasks would outgrow it pretty quickly.
SSIS isn't great for production applications from my experience for the following reasons:
To call an SSIS package remotely, you have to call a stored procedure, that calls a job, that calls the SSIS
Using the above method, you can't pass in parameters.
Passing parameters means you have to call the SSIS on a local server - meaning code running on a remote server will have to call code running on the SQL server to execute the package.
I would always rather write specific code to handle ETL and use SSIS for one off transforms.
In my opinion it's quite good platform, and I see a good progress on it. Many of the drwabacks that 2005 version had and that the community complained about, have been corrected on 2008.
From my point of view, the best thing is that you can extend and complement it with SQL or .NET code in an organized way as much as you want.
For instance, you can decide if in your solution you want 80% of c# code and 20% of ETL componenets or 5% of c# code and 95% of ETL components.
disclaimer - i work for microsoft
now the answer
SSIS or SQL Server Integration services is a great tool for ETL operations, there is a lot of uptake in the market place. there is no additional cost other than licensing SQL server and you can also use .Net languages to write tasks.
http://www.microsoft.com/sqlserver/2008/en/us/integration.aspx
http://msdn.microsoft.com/en-us/library/ms141026.aspx
I would list as benefits:
you use SSIS for bigger projects, probably/preferably once or in one run, and then use the integration project for many months with minor changes; the tasks, packages and everything in general is easily readable (of course, depends on perspective)
the tool itself handles the scheduled runs, sending you mails with the logs, and - as long as my experience reaches - it communicates very well with all the other tools (such as SSAS, SQL Server Management Studio, Microsoft Office Excel, Access etc., and other, non-Microsoft tools)
the manually, in-detail configured tasks seem to take over the responsibility in all ways, letting only small chance for errors
as also mentioned above, there are many former problems corrected in the new versions
I would recommend it for ETL, especially if you would continue with analytical processes, since the SSIS, SSAS and SSRS tools blend together quite smoothly.
Drawback: debugging/looking for errors is a bit harder until you get used to it.
I have seen the references to VistaDB over the years and with tools like SQLite, Firebird, MS SQL et. al. I have never had a reason to consider it.
What are the benefits of paying for VistaDB vs using another technology? Things I have thought of:
1. Compact Framework Support. SQLite+MSSQL support the CF.
2. Need migration path to a 'more robust' system. Firebird+MSSQL.
3. Need more advanced features such as triggers. Firebird+MSSQL
The VistaDB client runtime is free. The runtime will never "expire at 3am" as you put it. Only the developer tools are licensed in that manner. You need 1 license per developer, simple. We even offer a really inexpensive Lite version with no Visual Studio tools.
Some other benefits
100% managed code - there are no interop or other unmanaged calls in the engine. This is a big deal to some, and others couldn't care less.
No registry access required - Most other in proc databases require registry access to look for parent controls, or permissions. VistaDB only does what you tell it to do, and will even run in Medium Trust.
XCopy deployment for runtime and your database (single file). You can xcopy you application, the runtime, and your database and run. Nothing to install or configure on the machine, no special privileges needed (we can run in Medium Trust or higher).
Isolated storage - You can put your entire database into Isolated Storage and run it from there directly. This makes it very easy to build secure click once applications that write databases in a domain friendly way for corporate environments. There is no need to store the user data on a shared drive or worry about permission mapping.
CLR Triggers / CLR Procs - You can write CLR Code and use them as Triggers or Stored Procs. We have just recently introduced changes to make it even easier to maintain a single CLR Assembly that can run in both VistaDB and SQL Server 2005/2008.
T-SQL Procs - VistaDB T-SQL Procs are compatible with SQL Server 2005/2008. Any procedure that works in our engine will run in SQL Server. That does not mean anything that runs there will port to us. We are a subset of the functionality in SQL Server. But we are also the only way to run T-SQL Procs without SQL Server (SQL CE can't do it).
I personally think one of the biggest features is the ability to upsize to SQL Server later. All of the VistaDB types, syntax, and CLR Procs, T-SQL procs, etc all will run on SQL Server. (You can't take everything from SQL Server down to VistaDB though, it is a subset)
32/64 bit Deployment - VistaDB is a single assembly deployment that runs both 32 and 64 bit without changes. SQL CE requires two different runtimes depending upon the OS, and cannot run under IIS at all. Access has no 64 bit runtime, and the most recent 32 bit runtime can only be deployed through MSI. The 32 bit version of Windows has the runtime, the 64 bit version does not.
Relational Integrity - VistaDB also actually enforces your constraints and Foreign Keys. You can specific cascade update, and delete operations. The person who commented we are like SQLITE is wrong in this regard. They parse constraints, but do not enforce them.
EDIT: They do have support for FK's now in SQLite. But they are not compiled in by default, and do not use the same syntax as SQL Server.
Medium Trust - The ability to run on a medium trust web server is another feature that many will not care about, but it is a big deal. Many third party controls can't even run in Medium Trust. We can run the complete engine within Medium Trust because of our commitment to 100% managed code and least permission required.
- Full disclosure - I am the owner of VistaDB so I may be biased. :)
Well, the main thing is that it is pure managed code - for what that is worth; it works not only on your typical Windows machines running .NET, but works wherever you run the Compact Framework and even works on Mono. Here are some noteworthy bullet points from their homepage:
Small < 1 MB footprint truly embedded ZeroClick
Microsoft SQL Server 2005 compatible data types and T-SQL syntax
None of the SQL CE limits
Single user, multi user local or using shared network.
Partially trusted shared hosting is no problem.
Royalty-free distribution - single CPU deployment of SQL Server costs more than a site license of VistaDB!
One thing worth noting is that Rob Howard's company, telligent, uses it as the default database for their new CMS software, "Graffiti."
I have played with it here and there but have yet to build anything against it.
For me this most interesting feature of VistaDB is that it can be run in Medium Trust environment. Which makes it perfect solution for creating small to medium .NET websites which can be deployed on server by copying and pasting (x-copy deployment).
And almost all windows shared hosting providers (like GoDaddy) won't let you run your websites in Full Trust mode. And also won't install for you any 3rd party binaries into GAC like System.Data.SQLite.dll if you wish to use SQLite for example.
I hadn't seen VistaDB before, it does look pretty cool.
Update: Received a comment from someone from VistaDB - their update model is only for getting new versions. Your old ones won't stop working if your license expires, which is good to know.
Keeping the original post here as IMHO the warning about expiring software licenses is still worth thinking about, even though VistaDB itself is fine.
It definitely seems 'more featureful' than SQLite, but I don't see anything there to justify the cost. The site seems to indicate that you can buy one license for $279, but it implies this is just a 1 year subscription. Would you have to then pay another $279 next year to stop your site falling over?
If so, remember to factor into the 'cost' how much inconvenience it's going to be when you get a call at 3am (murphy's law, it's always 3am) from your panicking customers because their VistaDB license has expired :-(
I've had this experience personally with some expiring software, and it's never good. You can send your customers emails and messages and flash their entire screen blinking red saying "YOU NEED TO GET A NEW LICENSE BEFORE NEXT WEEK" and they'll still never do it, and you'll still get the pain at 3am when it does expire.