I have a folder in Cloud with some files and I have an SSIS Package that upload these file to SQL. I have also two variables on my SSIS package: start_date and end_date.
Basically, I would like to do that:
If start_date is empty then the SSIS should upload files since last uploaded.
If start_date is not empty the SSIS should upload the file from start_date to end_date.
I have a variable that configures the files that SSIS needs to upload. I have a problem here:
I don´t know I can set up this step, because when start_date is empty the SSIS gives an error message
You should use the following syntax:
IF '" + (DT_WSTR,10)REPLACENULL(#[$Package::start_date],"") + "' <> ''
Related
Pretty new to BI and SQL in general, but a few months ago I didn't even know what a model is and now here I am...trying to build a package that runs daily.
Currently running this is Excel via PowerQuery but because the data is so much, I have to manually change the query every month. Decided to move it into SSIS.
Required outcome: Pull the last date in my Database and use it as a variable in the model (as I have millions of rows, I only want to load lines with dates greater than what I have in my table already).
Here is my Execute SQL Task:
I set up a variable for the SQL query
and trying to use it in my OLE DB query like this
Execute SQL Task: results, are fine - returns date as "dd/mm/yyyy hh24:mi:ss"
SELECT MAX (CONVACCT_CREATE_DATE) AS Expr1 FROM GOMSDailySales
Variable for OLE DB SQL Query:
"SELECT fin_booking_code, FIN_DEPT_CODE, FIN_ACCT_NO, FIN_PROD_CODE, FIN_PROG_CODE, FIN_OPEN_CODE, DEBIT_AMT, CREDIT_AMT, CURRENCY_CODE, PART_NO, FIN_DOC_NO, CREATE_DATE
FROM cuown.converted_accounts
WHERE (CREATE_DATE > TO_DATE(#[User::GetMaxDate],'yyyy/mm/dd hh24:mi:ss'))
AND (FIN_ACCT_NO LIKE '1%')"
Currently getting missing expression error, if I add " ' " to my #[User::GetMaxDate], I get a year must be between 0 and xxxx error.
What am I doing wrong / is there a cleaner way to get this done?
In the OLEDB source use the following, change the data access mode to SQL command, and use the following command:
SELECT fin_booking_code, FIN_DEPT_CODE, FIN_ACCT_NO, FIN_PROD_CODE, FIN_PROG_CODE, FIN_OPEN_CODE, DEBIT_AMT, CREDIT_AMT, CURRENCY_CODE, PART_NO, FIN_DOC_NO, CREATE_DATE
FROM cuown.converted_accounts
WHERE (CREATE_DATE > TO_DATE(?,'yyyy/mm/dd hh24:mi:ss'))
AND (FIN_ACCT_NO LIKE '1%')
And click on the parameters button and map #[User::GetMaxDate] to the first parameter.
For more information, check the following answer: Parameterized OLEDB source query
Alternative method
If parameters are not supported in the OLE DB provider you are using, create a variable of type string and evaluate this variable as the following expression:
"SELECT fin_booking_code, FIN_DEPT_CODE, FIN_ACCT_NO, FIN_PROD_CODE, FIN_PROG_CODE, FIN_OPEN_CODE, DEBIT_AMT, CREDIT_AMT, CURRENCY_CODE, PART_NO, FIN_DOC_NO, CREATE_DATE
FROM cuown.converted_accounts
WHERE CREATE_DATE > TO_DATE('" + (DT_WSTR, 50)#[User::GetMaxDate] +
"' ,'yyyy/mm/dd hh24:mi:ss') AND FIN_ACCT_NO LIKE '1%'"
Then from the OLE DB source, change the data access mode the SQL Command from variable and select the string variable you created.
Your trying to use the SSIS variable like a variable in the query. When constructing a SQL query in a string variable you simply need to concatenate the strings together. The expression for your query string variable should look like this.
"SELECT fin_booking_code, FIN_DEPT_CODE, FIN_ACCT_NO, FIN_PROD_CODE, FIN_PROG_CODE, FIN_OPEN_CODE, DEBIT_AMT, CREDIT_AMT, CURRENCY_CODE, PART_NO, FIN_DOC_NO, CREATE_DATE
FROM cuown.converted_accounts
WHERE CREATE_DATE > " + #[User::GetMaxDate] +
"AND (FIN_ACCT_NO LIKE '1%')"
I am trying to create an SSIS package for Stored Procedure Deployment and Backup for our project.
I have some .sql file, each file contains one stored procedure definition and the name of the file is the stored procedure name itself.
I am trying to do the following by using SSIS
Read all files names one by one
Find the definition of each stored procedure if it exists in the database
If exists, then save the definition with the same name in a different folder (In my case it's a ROLLBACK folder)
For all new SP it save in a same file named DropNewSp.sql with DROP STORED PROCEDURE command.
After completing the backup process, execute all files in the destination database.
I am able to generate the desire .sql files, but I am faceing the following problem
1. The package also generated unwanted blank .sql file for all new Stored Procedure
2. The execution process failed if the stored procedure has some dependency on subsequent stored proc
In this answer, I will provide the main steps with some references to get more information on how to achieve each step. Even if I agree with the comments mentioned above that this is not the job of SSIS.
Add a foreach loop container that loop over .sql files and store the file name inside a variable:
Looping over files with the Foreach Loop
Lesson 2-2: Add and configure the Foreach Loop container
Load multiple source files
Add an Expression Task to retrieve the file name from the File Full Path (variable)
#{User::FileNameWithoutExtension] = SUBSTRING (#[User::FullFilePath], LEN( #[User::FullFilePath] ) - FINDSTRING( REVERSE( #[User::FullFilePath] ), "\\", 1) + 2, LEN (RIGHT( #[User::FullFilePath], FINDSTRING( REVERSE( #[User::FullFilePath] ), "\\", 1 ) - 1 ) ) - FINDSTRING( REVERSE( #[User::FullFilePath] ), ".", 1 ) )
Add an Execute SQL Task inside the foreach loop container to check if the stored procedure is found in the database:
SELECT COUNT(*) FROM sys.objects WHERE type = 'P' AND name = ?)
Pass the procedure name as parameter to the execute sql task:
Passing Variables to and from an SSIS task
Store the count result inside a variable of type integer using Resultsets:
SSIS Basics: Using the Execute SQL Task to Generate Result Sets
Using precedence constraints with expressions add 2 paths from the execute sql task
the first using expression #[User::Count] == 0
the second using expression #[User::Count] > 0
Other references:
Working with Precedence Constraints in SQL Server Integration Services
Defining Workflow in SSIS using Precedence Constraints
On the second path add an Execute SQL Task to get the procedure definition using the same approach above:
SELECT OBJECT_DEFINITION (OBJECT_ID(N'<databasename>.<schemaname>.' + CAST(? as VARCHAR(100))));
And store the result inside a variable using a result set.
Add a Script Task to write the procedure definition into the destination file
On the first path add a File system task to move the file into the directory specified
Add another foreach loop to read new files and execute the content.
I'm scheduling a SSIS package for exporting data to flat file.
But i want to generate file names with some date information, such as foo_20140606.csv
Is it possible?
Thanks
With the help of expressions you can make connection dynamic.
Select your flat file connection from Connection Managers pane. In Properties pane, click on Expression(...). Then choose ConnectionString Property from drop down list and in Expression(...) put your expression and evaluate it.
Expression build -
For day : (DT_STR,4,1252)DAY( DATEADD( "dd", -1, getdate() ))
For month: (DT_STR,4,1252)MONTH( DATEADD( "dd", -1, getdate() ))
For Year: (DT_STR,4,1252)YEAR( DATEADD( "dd", -1, getdate() ))
Example expression(you need to tweak as per your requirement) -
"E:\\Backup\\EmployeeCount_"+(DT_STR,4,1252)DATEPART( "yyyy" , getdate() ) + RIGHT("0" + (DT_STR,4,1252)DATEPART( "mm" , getdate() ), 2) + RIGHT("0" + (DT_STR,4,1252)DATEPART( "dd" , getdate() ), 2) +".txt"
which is giving E:\Backup\EmployeeCount_20140627.txt as value.
Please note - You need a working flat file connection so first create flat file connection whose connectionString property is then going to be replaced automatically by expression.
You can follow these step by step articles as well.
ssis dynamically naming destination
SSIS Dynamic Flat File Connection
Select your file connection in the Connection Managers, go to the Properties and click on the (...) beside expressions.
In the editor select ConnectionString from the Property column.
In the Expression text box, you can enter something like "rootNameOfFile" + (DT_WSTR, 50)(DT_DBDATE)GETDATE() + ".csv"
Evaluate your expression to make sure you're getting what you expect, and voila!
If you don't have SSDT and thus can't edit this with a GUI here is how you edit the SSIS package directly:
Before:
<DTS:ConnectionManager
DTS:refId="Package.ConnectionManagers[DestinationConnectionFlatFile]"
DTS:ObjectName="DestinationConnectionFlatFile"
DTS:DTSID="{C69365C4-EF12-4606-980B-E8862EE997A4}"
DTS:CreationName="FLATFILE">
<DTS:ObjectData>
After:
<DTS:ConnectionManager
DTS:refId="Package.ConnectionManagers[DestinationConnectionFlatFile]"
DTS:CreationName="FLATFILE"
DTS:DTSID="{C69365C4-EF12-4606-980B-E8862EE997A4}"
DTS:ObjectName="DestinationConnectionFlatFile">
<DTS:PropertyExpression
DTS:Name="ConnectionString">"C:\\Exportdir\\Filename_"
+ (DT_WSTR,4)DATEPART("yyyy",GetDate()) +
RIGHT("0" + (DT_WSTR,2)DATEPART("mm",GetDate()) ,2) +
RIGHT("0" + (DT_WSTR,2)DATEPART("dd",GetDate()),2) + "_" +
RIGHT("0" + (DT_WSTR,2)DATEPART("hh",GetDate()),2)+
RIGHT("0" + (DT_WSTR,2)DATEPART("mi",GetDate()),2) + ".csv"
</DTS:PropertyExpression>
<DTS:ObjectData>
you can easily give the dynamic file name for flat or csv file in SSIS package like DataFeed_{yyyyMMdd} .txt eg. DataFeed_20181212.txt. simply go to flat file connection manager property and define the expression value and in another way you can define the package level variable and this variable you can use in expression and as per environment you can able to define folder location by using configuration file .dtsconfig file
more details # ssis dynamic file name for flat file
I have an SSIS package that runs a stored procedure on a server (scheduled weekly) and spits out the result, from a query in the stp, to a file to be picked up by another program.
I need the name of this output file to be the week of the year and the year in WW YY format.
My current package is basic, using a DestinationConnectionFlatFile Manager, outputing to a static name.
Any clues as to how to do this would be great.
Thanks
Select the Flat file connection manager and edit the expression to something like:
"C:\\" + DatePart("wk",getdate()) + " " + DatePart("year",getdate()) + ".txt"
These kind of expression are based on VB.
I Think That My Question Is Simple.
How Can I Find That My Query Is Running From Where
( Where is The Location of the Script File itself ) ?
Edit :
Thank You For Your Answer.
I Need To Import a XML File Using my TSQL Script File And i want to Keep Them Together,
so Wherever Someone try to run the TSQL script file, it must knows the current directory of itself to know where is the XML file and then import it. Thank Again !
You need a well known location where you can place XML files for the server to load. This could be a share on the SQL Server machine, or on a file server which the SQL Server service account has permissions to read from.
You then need a comment like this at the top of your script:
--Make sure you've placed the accompanying XML file on \\RemoteMachine\UploadShare
--Otherwise, expect this script to produce errors
Change \\RemoteMachine\UploadShare to match the well known location you've selected. Optionally, have the comment followed by 30-40 blank lines (or more comments), so that it's obvious to anyone running it that they might need to read what's there.
Then, write the rest of your script based on that presumption.
I Found A Solution to my problem that's simpler !
You Know I Just Import My XML File To A Temp Table for once.
Then I Write a Select Query for That Temp Table That Contains my imported Data Like This :
" SELECT 'INSERT INTO MyTable VALUES (' + Col1 + ', ' + Col2 + ')' FROM MyImportedTable "
And Now I Have Many Insert Commands For Each One Of My Imported Records.
And I Save All of the Insert Commands in My Script. And So I Just Need My Script File Everywhere I Go.