I have SSIS package with data flow task and execute SQL task components in For each loop container. Package flow is, Date flow task(flat file--> Conditional split to insert data into SQL server tables)-->Execute SQL task(perform some SQL operations on inserted data and insert the calculated values in one final analysis table) . File name is like name1_name2_yyyymmdd_1234.txt. I want to fetch the date from file name and insert that date value in table in SQL Server as FileDate. I am trying to do it using derived column but unable to figure our where will I save it so that it will be available in Insert statement in Execute SQL Task which is after Data flow task.
This should be done outside the dataflow but within the ForEach loop.
Pass in two parameters (package scope) to a script task. One with the #filename (read only) from the forloop and to store the #fileDate (read/write).
Split will create a 0-based array in which you only care about the third piece.
Dts.Variables["fileDate"].Value = DateTime.ParseExact(Dts.Variables["fileName"].Value.Split('_')[2]
,"yyyyMMdd", System.Globalization.CultureInfo.InvariantCulture);
Now you can use #fileDate anywhere you would like.
Related
I am new to SSIS. I am trying to create an ETL pipeline to automate the updating and deleting process for a database.
I have created a data flow task which reads the Excel file and sends the data to respective staging tables in SQL Server.
For the data to be updated in the main database, it has to go through some transformation in the staging tables. I have created a stored procedure that will enforce these changes.
I want the stored procedure to get called right after data is loaded through the data flow task to the staging tables rather than me going to SSMS to manually execute the stored procedure.
I have tried adding the "Execute SQL Task" on the control flow tab but not getting any results.
I would like to further add many more transformations in this whole process in future steps. Any ideas on how to make this whole process more convenient would also be appreciated.
[Data Flow Task] -> [Execute SQL Task]
Configure the Execute SQL Task with a Direct Input value of
EXECUTE dbo.MasterQuery;
Based on the image of your stored procedure, it would appear you have a logic error in there.
IF EXISTS(SELECT 1 FROM dbo.OutlookDataStg WHERE [Flag] = 'Outlook')
BEGIN
UPDATE dbo.OutlookDataStg
SET [Data Type] = 'Outlook'
WHERE [Flag] = 'Actual'
-- Cut off at this point
END
The logic provided is
If there is at least one row in the table dbo.OutlookDataStg where the value flag is Outlook, then update the same table but set the Data type to Outlook for any rows with a flag of Actual.
Unless you have some unusual condition, it would see you've mixed your Flag and Data Type values
I need to extract data from a DB2 database to a SQL Server. I need to create my query based on a Excel file I have 176 records, which I need to create repeating queries & put in SQL server
So for example;
I have an Excel with a Number, From date, To date, and a Country
So the query should use these information from the records
SELECT *
FROM dbo.Test
WHERE Number = excel.Number1 AND Date BETWEEN excel.fromDate1 AND excel.toDate1 AND Country = excel.country1
And then another query with
SELECT *
FROM dbo.Test
WHERE Number = excel.Number2 AND Date BETWEEN excel.fromDate2 AND excel.toDate2 AND Country = excel.country2
Etc...
How should I do something like this in SSIS?
If needed I can put the DB2 and Excel data in MS SQL
You can proceed with the following approach:
Extract data rows from Excel and put it into SSIS Object Variable
Proceed with a Foreach loop to get each row from the Object Variable, parsing Object Variable to separate variables
Inject variable values into SQL Select command with Expressions
Perform Data Flow task based on SQL command, transform and put it into the target
Overall, your task seems to be feasible, but requires some knowledge on parsing Object Variable in Foreach Loop, and writing Variable Expressions.
I have created a package in ssis in which i use some date-variables inside my SQL statements ( i.e declare #DateIn ="2018-02-22" and declare #DateTo = "2018-03-22"), in order to load the corresponding data inside the tables of the data warehouse.
What I need to do is to create a task or a different package, which will give me the possibility to define externally the values of these variables, every time i run it, in order to fill in the tables of the warehouse with the data that corresponds to the dates i set every time.
From what I've read, I should maybe use a script task or an execute sql task or parameters
Could you help me please? Or could you suggest me a good tutorial/link?
I have found plenty but can't decide if they meet the needs of what i am describing above.
Thank you
Create DTSX package with variables #DateStart and #DateEnd
Create table containing 3 columns DateStart, DateEnd, Active
Create stored procedure that reads DateStart, DateEnd where Active = 1 from your newly created table and does an alter on the SQL Server Job updating your variables value that are inside of your DTSX package with your desired value using sp_update_jobstep
See link
Ex of command:
dtexec /f YourPackage.dtsx
/set \package.variables[DateStart].Value;myvalue
/set \package.variables[DateStart].Value;myvalue
Add sp_start_job inside the stored procedure to start the job with the new variable values.
Create job with 1 step containing the execute of the stored procedure from Step 3
All you need to do is update the values from your table created in Step 2 and then execute job to run the stored procedure to update DTSX job exec command and start it. You can trigger this from a website and control the tables values from textboxes.
Also specific Permissions are required and the SP that updates the SQL Agent job needs to be run by Sysadmin
Good question by the way for the new learner!
There are many ways for this scenario,few of them I have mentioned below.
1-Create variable in variable pane #DateIn and #DateTo for storing the date and data type will be date.
Now put 2 entry in Excel ,text or xml for these two variables and call it by using foreachloop container and assign this to variables.
2-Create a SQl table in which you can store those values either by manually on daily basis or load the table with excel ,text ,xml or csv file and call the table in Execute SQL Task and select the result set and pass the result set values to the variables.
I hope it will solve your problem.
I have an SSIS package with two Execute SQL Tasks, the first one has a simple select statement to select a single column from a table, the result set is an ID column which I have to use as input to the second Execute SQL Task to get records from another table which matching IDs. Basically I have to implement following SQL statement in SSIS:
SELECT * FROM TableB WHERE ID IN
(
SELECT ID FROM TableA
)
How can I configure the second Execute SQL task to use the result set of first Execute SQL task as input to execute an SQL statement like mentioned above?
One solution would be to populate an object variable with the result of the first Execute SQL Task.
Then have a Script Task which loops through the object variable and dynamically builds the SQL String for the second Execute SQL Task, and puts that into a string variable.
Then finally, the second Execute SQL Task uses the string variable for its SQL query.
ETL Script to dynamically map multiple execute sql resultset to multiple tables (table name based on sql file provided)
I have a source folder with sql files ( I can put it up as stored procedures as well ) . I know how to loop and execute sql tasks in a foreach container. Now the part where I'm stuck is I need to use the final result set of each sql queries and shove it into a table with the same name as the sql file.
So, Folder -> script1.sql , script2.sql etc -> ETL -> goes to table script1, table script2 etc.
EDIT : Based on the comment made by Joe, I just want to say that I'm aware of using insert within a script but I need to insert it onto a table in a different server.And Linked servers are not the ideal solutions
Any psuedocode or link to tutorials will be extremely helpful . Thanks!
I would add the table creation to the script. It is probably the simplest way to do this. If your script is Select SomeField From Table1, you could change it to Select SomeField Into Table script1 From Table1. Then there is no need to map in SSIS which is not easy to do from my experience.