Can anyone please tell me how to perform a simple batch process in SSIS?
I know how to do such a thing using T-SQL and/or .NET code but what I wish to do is to use the SSIS GUI to do this entirely. I am not sure if it is possible and all I can find on Google are over complicated solutions.
To explain a bit more - I am reading from a flat file and I want to insert its entire contents into a Sql Server table. Pretty simple huh... But I want to do it 50,000 records at a time. I expect this to be as simple as setting a property somewhere or at the most using some kind of loop tool from the toolbox.
Thanks
You are right. It is a property of the Data Flow task.
Right click on the task, click on the "Properties..." command, look for the DefaultBufferMaxRows property and set the desired value.
References #MSDN.
Related
The firm I work in has a lot of data sources entering the firm database using the Informatica ETL tool, stored in maplets and other data models (sorry If I'm not using the exact terminology).
The problem is that all the business logic is stored in the 'graphical interface' and nowhere else - Every time I want to see what field goes into the target field I have to trace the inputs through the maplet and that takes a very long time.
The Question is: Is there a tool that can takes all the relationships in the Informatica maplet and somehow export them to a excel table (so I can see it all without tracing)? that way I could try to make proper documentation....
Thanks in Advance.
It's possible to export mappings or whole workflows to XML. Next, you can use this tool - it will create tables with source to target dependency for every mapping.
Keep in mind it will only map input to output, it won't extract the full logic and transformations done along the way - that would've been to complex for simple visualization.
Informatica supports exporting mapping information to Excel - just search the documentation which tells you how to do it.
However, for anything other than the simplest of mappings, what ends up in Excel is not that easy to understand. If your Informatica installation supports it, then using the lineage capabilities is a much better bet.
I have an SSIS job that is scheduled to run every 5 minutes via SQL Agent. The job imports the contents of an excel file into a SQL table. That all works great, but the files get placed there sporadically and often times when the job runs there is no file there at all. The issue is this is causing the job to fail and send a notification email that the job failed, but I only want to be notified if the job failed while processing a file, not because there was no file there in the first place. From what I have gathered I could fix this with a script task to check if the file is there before the job continues, but I haven't been able to get that to work. Can someone break down how the script task works and what sort of script I need to check if a file exists? Or if there is some better way to accomplish what I am trying to do I am open to that as well!
The errors I get when I tried the Foreach Loop approach are
This can be done easily with a Foreach Loop Container in SSIS.
Put simply, the container will check the directory you point it at and perform the tasks within the container for each file found. If no files are found the contents of the container are never executed. Your job will not fail if no files are found. It will complete reporting success.
Check out this great intro blog post for more info.
In the image attached the question, the specific errors are related to the Excel Source failing validation. When SSIS opens a package for editing or running, the first thing it does is validate all of the artifacts needed for a successful run are available and conform to the expected shape/API. Since the expected file may not be present, right click on the Excel Connection Manager and in the Properties menu, find a setting for DelayValidation and change it to True. This will ensure the connection manager only validates the resource is available if the package is actually going to use it i.e. it passes into the Foreach Loop Container. You will also need to set the same DelayValidation to True on your Data Flow Task.
You did not mention what scripting approach you're applying to search for your file. While using C# or VB.NET are typical scripting languages used in a Scripting control task of this nature, you can also use TSQL that will simply return a boolean value saved to a user variable (Sometimes systems limit the use C# and VB.NET). Then you apply that user variable in the control flow to determine whether to import (boolean = 1) or not (boolean = 0).
Take a look at the following link that shows in detail how to set up the TSQL script that checks for whether or not a file exist.
Check for file exists or not in sql server?
Take a look at the following link that shows how to apply a conditional check based on a boolean user variable. This example also shows how to apply VB.NET in a script task to determine if the file exists (as an alternative to the before mentioned TSQL approach).
http://sql-articles.com/articles/bi/file-exists-check-in-ssis/
Hope this helps.
I will preface this right off the bat by saying that I am new to database design.
I have been working to rewrite some legacy code that controls an import process into one of our pieces of software. Part of this new process includes the modification of the incoming XML files (that come into our system via FTP) to remove certain elements as well as swap values in special cases.
As a part of the new system, we are implementing a way of versioning inside the database so that we can pull the most recent version of the xml directly from that instead modifying the file over and over again. In order to prove that this can be done, I have created a very simple table inside of SQL Server 2016 that stores the XML, then wrote a simple PowerShell script to pull that XML file from the database and store it inside of an object. Now that I know that this is indeed possible, I need to refine how I design the table.
This is where my expertise starts to take a hit. As of right now, the table contains three columns: xml_Version, xml_FileID, and xml_FileContents.
The general idea is to have a GUID (xml_FileID) that is tied to each version of the XML and another column that indicates what version of the XML that is. I would also assume that you need some way of tying each version of the XML to it's original file, too.
I was hoping that someone could point me in the right direction about how I should go about designing the table to accomplish this task. I can provide more information if needed.
Thanks.
Edit: I think what I'm having the most trouble grasping is what I should be referencing when I'm trying to grab data out of the database. Storing the XML in the table with a unique identifier is the easy part - but the unfortunate part is that there's nothing in the XML itself that I can grab out of there that would be able to uniquely identify the correlating data within the database. Does that make sense?
I am using SSIS packages to daily refresh the data. Package logic is as follows,
Delete all rows in destination table
Insert full new data into destination table.
I am trying to find out ways to rollback delete if my insert fails. I tried using SSIS package transaction as below:
But now, after Delete SQL task is run , my package goes stuck for long time and does not respond.
What is the recommended way for doing this?
Any help is much appreciated.
There are quite a few techniques to consider here including some more complex ideas, but if we're looking at simpler ones, you could insert into a table with a different name but the same structure, and only if that works would you then swap it out somehow. One way of doing this is to use views for your access to tables, and then modify the view on success to use the table you've just inserted into.
It might not be the most elegant way, but it is one of the simpler ones to consider.
Change the default of the Transaction Option property of the package to "Required" and make sure each object has that property set to "Supported" which is the default.
Additionally, you can minimize the transaction by doing the same thing with a sequence container around just your Execute SQL task and data flow.
FYI, I can't see pictures at work so I do not know what your package looks like.
I want to export all my queries as individual files for purposes of putting them into mercurial source control, but I don't know how to export the individual queries as individual files without having to open each one, then save to the folder, then add into the project, or some equally convoluted process.
I wouldn't mind having to add each one individually, but how do I get them out of the database as individual files without opening them all and doing each one save as? Ostensibly I would like them named with the name they have in the database right now.
I could easily dump the whole lot into one long file using database tasks, but that's not really super helpful is it?
I have SSMS 2k5 and 2k8 (and VS 2k5, 2k8, 2010 to boot) to work with, any thoughts?
Right click on the database. Select Generate Script. On the last page. Script To file you can choose single file or file per object
When you script a database in SSMS you have the option of one file per objects.
SMO is useful with a small app to iterate through
Third party tools like Red Gate SQL Compare (there are other free tools) can script too
I would write a small C# program which extracts your database object via SMO and stores them in your filesystem the way you want.
It is rather easy to write stored procedures which fetches the definition into the result as text. sp_helptext could be used as start.
Than you can use PowerShell to write the Output to the file system.
It sounds as if this would fit rather good into the Really Simple Data Dictionary codeplex project. link text