I've been searching all over for an example of doing this but I'm not finding it. I know it is possible because we had it working at one point but the resource that developed the process isn't currently available to fix the process which is currently corrupted beyond repair. In fact corrupted so badly we can't even get into take a look at what was there to build a copy of the process over again.
What we have is a 'Production_DB' and a 'Test_DB' which and the two are essentially the same. What was taking place is that a SSIS task was firing at the end of each work day and refreshing 'Test_DB' with the data that is in 'Production_DB'. In this way testing can take place and changed can be made to the test bed without any concern that it will get too far afield of the live data because each evening this data is brought back to exactly what is in production. Meanwhile for testing purposes all testing is begin measure against actual real life data examples so when processes are pointed at the production data set there is less chance of issues.
The problem we have is several months back we didn't realize it but the SSIS package and source files became corrupted beyond readability. So, now we are looking for a way to replace the package to restore the process, but as of yet I have not been able to find an example that I can use to build from.
We are on SQL Server 2008 R2.
If anyone has some references they can point me to it would be greatly appreciated!
Depending on the amount of tables and the SQL Server version you can use the import export wizard to identify prod as the source and test as the destination...use that wizard to create a task and SAVE the ending task (it should save as an SSIS package I believe). This will get you a quick way of making the SSIS package to copy the data over and you can even overwrite the destination data if you would like.
Right click the database > tasks > import data
Related
I have created a SSIS package that exports several rows to Excel, usually less than 200 (including the header). When I run it in VS2015 Debug everything turns green.
I even wait like this question says.
Still, nothing but the header ever gets to the file.
I know it's not much data, but I'm trying to automate it as the data will eventually grow and I don't want to manually run this all the time.
Any ideas? I'm using SQL Server 2012 and wrote this SSIS package with VS2015.
Something that occasionally happens with Excel destinations is that hidden formatting will cause the data flow connector to begin writing data at a row other than 1.
If this happens, you'll need to recreate your template from a fresh Excel document, and reconstruct the header as needed.
It depends on the buffer size that underlying process uses. I monitored the consumption of C: drive while the SSIS package was writing to the Excel destination, and found that the space was getting full, and as soon the whole space is occupied package ended with success without writing any row to excel destination. Therefore I cleared enough space from my C: drive (around 2 GB) and everything started working fine then.
Also found the following useful thread that might be helpful for someone.
Forewarning: I wasn't entirely sure if this question should be in here (SO) or in The Workplace because it isn't so much about programming, as much as it is convincing my co-worker that I think their method is bad. But it's still programming related. So MODs, please feel free to relocate this question to 'the workplace'. Anyway...
At work we have large SSAS cubes that have been split into multiple partitions. The individual who set up these partitions scheduled every partition to be processed everyday. But in hindsight because the data in these partitions is historic there is no need to process each partition everyday. Only the current partition should be processed after the latest data has been added into the cube's data source.
I gave my coworker a task to automate this process. I figured all they need to do is get the current date, and then process the partition corresponding to that date range. Easily scriptable.
My coworker creates an SSIS package for doing this...
Cons:
the ssis package is hard to source control
the ssis package will be hard to test
the ssis package is a pain in the ass to debug
the ssis package requires Visual Studio and Visual Studio Data Tools to even open
lastly, I feel SSIS packages lead to heavy technical-debt
Pros:
it's easier for my coworker to do (maybe)
Correct me if I'm wrong on any of those but just the first reason is enough for me to scrap all of their work.
Needless to say I'm extremely biased against anything done in SSIS. But processing a cube can be scripted out in xmla (ref: link). And then using a SQL Server Agent job you can schedule that script to run a specific times. The only tricky part would be changing out the partition name that is processed within the script. Furthermore, the script/job can be kept in source control and then deployed to the MSSQL server whenever a change is made.
Am I being too critical here? I'm just trying to keep the next developers from ripping their hair out.
What you can do is to have two SQL Jobs:
1) Full processing + repartitioning
2) Incremental processing (processing of only last (current) partition).
You don't need SSIS neither for (1), nor for (2).
For (2) the script will be fixed - you just make a call to process one partition and incremental processing of dimensions (if required). Current partition must have a condition WHERE >= .... (not BETWEEN), so it covers the future dates if a new partition is not created yet.
For (1), you can write TSQL code that creates a new partition for the new period and reprocess the cube. It can be scheduled to run over weekend when the server is idle, or once per month.
(1) does below:
backup existing cube (Prod) via SSAS Command of SQL Agent
restore the backup as TempCube via SSAS Command of SQL Agent with AllowOverwrite (in case if temp cube was not deleted before)
delete all partitions in TempCube via TSQL + LinkedServer to SSAS
re-create partitions and process cube (full) via TSQL +LinkedServer to SSAS
Backup TempCube
Delete TempCube
Restore backup of TempCube as Production Cube.
As you see, the process is crash safe and you don't need SSIS. Even when a job (that creates a new partition) wasn't run for some reason, the cube still have the new data. The data will be split when a new partition structure is created by (1).
I think you are looking at this wrong way. To be honest your list of cons is pretty bad and is just a reflection of your opinion of SSIS. There is always more than one tool in the toolbox for any job. The proper tool to use will vary from shop to shop.
If the skill set of the party responsible for development and maintenance of this automated process is SSIS then you should really have a better reason than personal preference to rewrite the process with a different tool. A couple reasons I can think of are company standards and skill set of the team.
If company standard dictates the tool then follow that. You should have staff that are capable of using the tools the company mandates. Otherwise assess the skill set of your team. If you have a team of SSIS developers don't force them to use something else because its your personal preference.
Your task of dynamic SSAS partitions processing can be automated with or without SSIS. SSIS is just an environment to execute tasks and do data manipulation. On its Pros - it has built-in components which execute XMLA script from variable and capture error messages. In pure .NET you have to do it yourself, but it is not too complex.
Several sample approaches to your task
Create XMLA XML and execute it with SSIS.
Generate XMLA from AMO library and execute it in .NET. You need to look at chapter 4d) Process All Dims. Provided sample does more that that and steps are put into SSIS package as well.
I personally used SSIS in similar situation, probably because the other 99% of ETL logic and data manipulation is on SSIS. As said before, SSIS offers no significant advantage here. The second example shows how to do it in pure .NET.
My issue is that I have a SQL database that is the output of a SQL project in an ASP.NET solution. This output consists of 2 datafiles, lets call them MainDB.mdf and DocContent.mdf, and the necessary log files. DocContent.mdf has one table, doc_content, which, you guessed it, stores the bytes for documents. There are multiple dev's working on this project, and we each have a (localdb) instance of the DB to use.
What I need to make happen is to attach MY MainDB.mdf referring to dev B's DocContent.mdf. The data FK's match 1:1, the schema is identical, the only difference is that Dev B populated the varbinary field with uploaded documents.
SQL seems to be doing everything in its power to prevent this. I get:
I realize I could do a Data Compare and pump the data in that way, but unfortunately Dev B and I are not at the same location and the logistics of him sending his full DB output would be harder than finding a work around with what I've got (I hope, anyway).
Surely there is a way to tell SQL to shut up and just use the datafile provided?
The only info I could find on this subject referred to attaching a DB without its LDF. Anyone know how to make this scenario work?
So, for anyone interested, the short answer is 'no'. There are a number of convoluted steps that you you can take to force the database into SUSPECT mode and possible even get it into single user emergency mode, but it was not worth the effort and the end result is that you will have to hack into the sys tables in order to make it limp along this way.
In the end it was easier to get a full transfer of Dev B's data files.
I have few questions about the memory usage in ssis package.
If i am loading data from server A to Server B and the ssis package is in my desktop System and running through BIDS,Whether the buffer creation(memory usage) will happen in my desktop system?If this is the case,the performance(low memory compare to servers) will be slow right?
How to enable the usage of server resources while developing package in my desktop system?
Please help me, if i have 3 ssis developer and all are developing different packages at a time,What is the best development method?
To expand on #3, the best way I have found to allow teams to work on a single SSIS solution is to decompose a problem (package) down into smaller and smaller chunks and control their invocation through a parent-child/master-slave type relationship.
For example, the solution concerns loading the data warehouse. I'd maybe have 2 Controller packages, FactController.dtsx and DimensionController.dtsx. Their responsibility is to call the various packages that solve the need (loading facts or dimensions). Perhaps my DimensionProductLoader package is dealing with a snowflake (it needs to update the Product and the SubProduct table) so that gets decomposed into 2 packages.
The goal of all of this is to break the development process down into manageable chunks to avoid concurrent access to a single package. Merging the XML will not be a productive use of your time.
The only shared resource for all of this is the SSIS project file (dtproj) which is just an XML document enumerating the packages that compromise the project. Create an upfront skeleton project with well-named, blank packages and you can probably skip some of the initial pain surrounding folks trying to merge the project back into your repository. I find that one-off type of merges go much better, for TFS at least, than everyone checking their XML globs back in.
Yes. A package runs on the same computer as the program that launches it. Even when a program loads a package that is stored remotely on another server, the package runs on the local computer.
If by server resources you mean server CPU, you cant. Is like using resources of any other computer on the network. Of course, if you have an OleDBSource that runs a select on SQL server, the CPU that "runs" the select will be the one on SQL Server, obviously, but once the resultset is retrieved, it is handled by the computer where the package is running.
Like any other development method. If you have a class on a C# project being developed by 3 developer, how do you do it? You can have each developer working on the same file and merge the changes, after all a package is a xml file, but is more complicated. I wouldn't recommend. I've been on situations where more than one developer worked on the same package but not at the exact same time.
Expanding on Diego's and Bill's Answers:
1) Diego has this mostly correct, I would just add: The package runs on the computer that runs it, but even worse, running a package through BIDS is not even close to what you will see on a server since the process BIDS uses to run the package is a 32bit process running local. You will be slower due to limits related to running in the 32bit subsystem, as well as copying all of your data for the buffer across the network to the buffer in memory on your workstation, transforming it as your package flows, and then pushing it again across the network to your destination server. This is fine for testing small subsets of your data in a test environment, but should not be used to estimate the performance on a server system.
2) Diego has this correct. If you want to see server performance, deploy it to a test server and run it there.
3) billinkc has this correct. One of the big drawbacks to SSIS in TFS is that there is not an elegant way to share work on a single package. If you want to use more than one developer in a single process break it into smaller chunks and let only one developer work on each piece. As long as they are not developing the same package at the same time, you should be fine.
We have some large schema changes coming down the pipe and are in needs of some tips in writing upgrade scripts manually. We're using SQL Server 2000 and do not have access to automated tools nor are they an option at this point in time. The only database tool we have is SQL Server Management Studio.
You can import the database to a local machine with has a newer version of SQL, then you can use the 'Generate Scripts' feature to script out a lot of the database objects.
Make sure to set in the Advanced Settings to script for SQL Server 2000.
If you are having problems with the script generated, you can try breaking it up into chunks and run it in small batches. That way if you have any specific generated scripts you can just write the SQL manually to get it to run.
While not quite what you had in mind, you can use Schema comparing tools like SQL Compare, and then just script the changes to a sql file, which you can then edit by hand before running it. I guess that would be as close to writing it manually without writing it manually.
If you -need- to write it all manually i would suggest getting some intellisense-type of tools to speed things up.
Your upgrade strategy is probably going to be somewhat customized for your deployment scenario, but here are a few points that might help.
You're going to want to test early and often (not that you wouldn't do this anyway), so be sure to have a testing DB in your initial schema, with a backup so you can revert back to "start" and test your upgrade any number of times.
Backups & restores can be time-consuming, so it might be helpful to have a DB with no data rows (schema-only) to test your upgrade script. Remember to get a "start" backup so you can go back there on-demand.
Consider stringing a series of scripts together - you can use one per build, or feature, or whatever. This way, once you've got part of the script working, you can leave it alone.
Big data migration can get tricky. If you're doing data transformations, copying or moving rows to new tables, etc., be sure to check row counts before the move and account for all rows afterwards.
Plan for failure. If something goes wrong, have a plan to fix it -- whether that's rolling everything back to a backup taken at the beginning of the deployment, or whatever. Just be sure you've got a plan and you understand where your go / no-go points are.
Good luck!