I'm developing a SSIS package to perform extractions from SQL Server 2008 to an Excel file.
This is my data flow:
"Extraction of concepts" is an OLE DB Source. It executes this SQL statement:
SELECT
Id,
Name,
Surname,
(
SELECT
CI.Interest + '; '
FROM
CustomerInterests CI
WHERE
CI.CustomerId = C.ID
FOR XML PATH ('')
) AS Interest
FROM
Customer C
WHERE Id = ?
When I try to save my query I get this error:
If I modify my SQL Statement like following, the error does not appear:
SELECT
Id,
Name,
Surname,
NULL AS Interest
FROM
Customer C
WHERE Id = ?
Can you help me?
Thanks
In order to achieve what you want, follow the below steps:
1) Add variable in SSIS called ID
2) In your WHERE Statement
Use : WHERE #Id = ?
3) Assign the #ID varible in Data flow with ID SSIS varibale.
Updated:
Related
Pretty new to BI and SQL in general, but a few months ago I didn't even know what a model is and now here I am...trying to build a package that runs daily.
Currently running this is Excel via PowerQuery but because the data is so much, I have to manually change the query every month. Decided to move it into SSIS.
Required outcome: Pull the last date in my Database and use it as a variable in the model (as I have millions of rows, I only want to load lines with dates greater than what I have in my table already).
Here is my Execute SQL Task:
I set up a variable for the SQL query
and trying to use it in my OLE DB query like this
Execute SQL Task: results, are fine - returns date as "dd/mm/yyyy hh24:mi:ss"
SELECT MAX (CONVACCT_CREATE_DATE) AS Expr1 FROM GOMSDailySales
Variable for OLE DB SQL Query:
"SELECT fin_booking_code, FIN_DEPT_CODE, FIN_ACCT_NO, FIN_PROD_CODE, FIN_PROG_CODE, FIN_OPEN_CODE, DEBIT_AMT, CREDIT_AMT, CURRENCY_CODE, PART_NO, FIN_DOC_NO, CREATE_DATE
FROM cuown.converted_accounts
WHERE (CREATE_DATE > TO_DATE(#[User::GetMaxDate],'yyyy/mm/dd hh24:mi:ss'))
AND (FIN_ACCT_NO LIKE '1%')"
Currently getting missing expression error, if I add " ' " to my #[User::GetMaxDate], I get a year must be between 0 and xxxx error.
What am I doing wrong / is there a cleaner way to get this done?
In the OLEDB source use the following, change the data access mode to SQL command, and use the following command:
SELECT fin_booking_code, FIN_DEPT_CODE, FIN_ACCT_NO, FIN_PROD_CODE, FIN_PROG_CODE, FIN_OPEN_CODE, DEBIT_AMT, CREDIT_AMT, CURRENCY_CODE, PART_NO, FIN_DOC_NO, CREATE_DATE
FROM cuown.converted_accounts
WHERE (CREATE_DATE > TO_DATE(?,'yyyy/mm/dd hh24:mi:ss'))
AND (FIN_ACCT_NO LIKE '1%')
And click on the parameters button and map #[User::GetMaxDate] to the first parameter.
For more information, check the following answer: Parameterized OLEDB source query
Alternative method
If parameters are not supported in the OLE DB provider you are using, create a variable of type string and evaluate this variable as the following expression:
"SELECT fin_booking_code, FIN_DEPT_CODE, FIN_ACCT_NO, FIN_PROD_CODE, FIN_PROG_CODE, FIN_OPEN_CODE, DEBIT_AMT, CREDIT_AMT, CURRENCY_CODE, PART_NO, FIN_DOC_NO, CREATE_DATE
FROM cuown.converted_accounts
WHERE CREATE_DATE > TO_DATE('" + (DT_WSTR, 50)#[User::GetMaxDate] +
"' ,'yyyy/mm/dd hh24:mi:ss') AND FIN_ACCT_NO LIKE '1%'"
Then from the OLE DB source, change the data access mode the SQL Command from variable and select the string variable you created.
Your trying to use the SSIS variable like a variable in the query. When constructing a SQL query in a string variable you simply need to concatenate the strings together. The expression for your query string variable should look like this.
"SELECT fin_booking_code, FIN_DEPT_CODE, FIN_ACCT_NO, FIN_PROD_CODE, FIN_PROG_CODE, FIN_OPEN_CODE, DEBIT_AMT, CREDIT_AMT, CURRENCY_CODE, PART_NO, FIN_DOC_NO, CREATE_DATE
FROM cuown.converted_accounts
WHERE CREATE_DATE > " + #[User::GetMaxDate] +
"AND (FIN_ACCT_NO LIKE '1%')"
I have a view in SQL Server that detects all the Products Ids from a table that does not have a Product Description.
In vertica we have all the Product IDs with their description.
My idea is to build a SSIS package that can pull all the Products Ids with missing description and use it as a where clause in the Vertica query.
I already tried with user variables but I failed. I am beginner on SSIS
This is what I am looking for.
SELECT [Product ID], [Product Desc]
FROM VerticaTable
WHERE [Product ID] IN (#Variable?)
The variable should like like this.
SELECT DISTINCT [Product ID] FROM SQLSerrverViewThatHasMissingDesc
Basically you have two ways to do that.
First of all, I would like to let you know the SELECT in the SSIS package will not show you anything, but you can make the changes to the query, such as insert the results into a table.
Create a Object type variable1 and use Foreach Loop Container to iterate that variable1, assign each of the value to another variable2 (created in advance, maybe it is a String type variable). Then in the container, put you Execute SQL task with the query:
SELECT [Product ID], [Product Desc]
FROM VerticaTable
WHERE [Product ID] = ?
Map variable1 as Index 0 in the parameter mapping page. (if you are using OLE DB as Connection type)
The drawback using this method is you cannot view all the productID at the same time because this is a loop process,unless you would like to load each of the result to a table.
If you prefer to see all the candidate results at the same time, the most common way is using Dynamic SQL because variables are not supported anywhere in your query.
So, the same query as you wrote, but need some add-ups.
1) First, Create three variables, variable1, variable2, variable3, variable1 is a Object type and variable2 and variable3 are String type.
2) Next, you need a Foreach Loop Container to build up your variable. In the Foreach Loop Container, drag and drop Script Task. Before the container, connected by a Execute SQL task, and after the container, connecting another Execute SQL task
So physical Order: Execute SQL task 1 -> Foreach Loop Container, inside: Script task -> Execute SQL task 2
3) Open the Execute SQL Task 1, put SELECT DISTINCT [Product ID] FROM SQLSerrverViewThatHasMissingDesc in the query, and go to the Result set page, assign Variable1
4) Move on to the Foreach loop Container, choose Foreach ADO Enumerator, then choose Variable1 as ADO object source variable, mode is Rows in the first table, then go to Variable mappings page, choose Variable2, and default the index.
5) Go to the Script task, open it, choose Variable as ReadOnlyVariable, in the ReadWrite variable, choose Variable3, edit the script, paste the follow code inside the Main()
if (Dts.Variables["User::Variable3"].Value.ToString() == "")
{
Dts.Variables["User::Variable3"].Value = Dts.Variables["User::Variable2"].Value.ToString();
}
else
{
Dts.Variables["User::Variable3"].Value = Dts.Variables["User::Variable2"].Value.ToString() + "," + Dts.Variables["User::Variable2"].Value.ToString();
}
Dts.TaskResult = (int)ScriptResults.Success;
6) After above step, ideally, your Variable3 will have a long string separated by ,, like A,B,C,D,E. The reason you want to get this string is because this Variable3 will be used later in the Execute SQL task as part of the Dynamic SQL
7) Here comes the final step, in the Execute SQL Task 2, use following query:
DECLARE #SQL VARCHAR(MAX)
SET #SQL = 'SELECT [Product ID], [Product Desc]
FROM VerticaTable
WHERE [Product ID] IN (''' + ? + ''')'
In the parameter mapping, choose Vareiable3, index as 0.
8) Execute the package.
But it will not show you anything, but if you try to simulate the dynamic process in SSMS, you will see the result.
LONG's answer is a good one. I am going to add a little tweak to get your IN clause. Please accept his answer and if you find this helpful just upvote.
Use a Execute SQL task to get the IN clause directly using a stuff function in SQL Server and result set to a single line directly into a string variable.
The coalesce just handles null (no result) and sets variable to empty string.
declare #t table (ProdID int)
insert into #t
values(1),(1),(2),(3)
select coalesce(
stuff(
(select ','+cast(ProdID as varchar(5))
from #t
group by ProdID
for XML path(''))
,1,1,'')
,'') as delimProdIds
Results:
delimProdIds
1,2,3
Simply change #t to your view
In the control flow (After this exec SQL step) put an expression that the variable !='' (Empty String) If it does, it will end the process right here.
Long's script to create the variable has a slight error:
Dts.Variables["User::Variable3"].Value =
Dts.Variables["User::Variable2"].Value.ToString() + "," +
Dts.Variables["User::Variable2"].Value.ToString();
When Variable3 has values, they are overlaid with Variable2 every time. It should probably be:
Dts.Variables["User::Variable3"].Value =
Dts.Variables["User::Variable3"].Value.ToString() + "," +
Dts.Variables["User::Variable2"].Value.ToString();
I'm trying to load incremental data from ODBC server to SQL server using common table expression.
When running the query in the Dbeabver application, is executed correctly:
with test as
(
SELECT userid,sum(goldbalance)
FROM Server.events_live
where eventTimestamp>=DATE '2016-01-01' + INTERVAL '-100 day'
group by userid
order by sum(goldbalance) desc)
)
select * from test
when running it from an sql command expression of the ODBC source, it fails due to wrong syntax. It looks as follow:
with test as
(
SELECT userid,sum(goldbalance)
FROM deltadna.events_live
where eventTimestamp>=DATE '"+#[User::datestring]+"' + INTERVAL '-100 day'
group by userid
order by sum(goldbalance) desc)
)
select * from test"
the datestring variable is getting the server date and convert it to string in the format yyyy-mm-dd. I'm usually use this method to pull data from ADO.NET and it works properly.
Is there any other way to pull incremental data from ODBC server using ssis variables?
With OLE DB
Try this code, it works for me with my own tables with SQL Server :
SELECT userid,sum(goldbalance) AS SUMGOLD
FROM deltadna.events_live
WHERE eventTimestamp >= DATEADD(DAY, -100,CONVERT(DATE,?))
GROUP BY userid
ORDER BY SUMGOLD desc
You have to click on Parameters in the OLEDB Source Editor to configure what you need. Use the '?' to represent a variable in your query.
If you query if too complicated, stored it in a stored procedure and call it like this:
EXEC shema.storedProcedureName ?
And map the '?' to your variable #user::DateString
With ODBC
The expressions are outside the data flow in Data Flow Properties.
Select the expression property and add your dynamic query.
And your expression will be
"SELECT userid,sum(goldbalance) AS SumGold
FROM deltadna.events_live
where eventTimestamp>=DATE "+#[User::datestring]+" +INTERVAL '-100 day'
group by userid
order by SumGold desc"
OK this seems like it should be insanely easy, but I cannot figure it out. Every where I look online says to create temp tables and VB scripts and I cannot believe I have to do that. My goal is to insert all the records in a table with a date later than the max date in that destination table.
UPDATE The 2 tables are in two different non linked SQL databases
So:
Select #[User::Dated] = MAX(Dateof) from Table2
Insert into Table2
Select *
From Table1
Where DateOf > #[User::Dated]
I am trying to do this in SSIS. I declared a variable, the SQL execution step looks like it is assigning the single row output to it. But when I got go into the data flow it give me no parameters to choose, when I force the known parameter which is in the project scope it says no parameter exists
Create two OLE DB data sources each pointing at you two databases.
Create a variable called max_date and make its data type String.
Place an Execute SQL Task on the Control Flow, change its connection type to OLE DB and for the connection select the name of the data source that contains Table2. Set the ResultSet to Single Row. Add the following for the SQLStatement:
SELECT CAST(MAX(Dateof) AS VARCHAR) AS max_date FROM Table2
Go to the Result Set pane, click Add and enter the following:
Result Name: max_date
Variable Name: User::max_date
You can now use the max_date variable in an expression to create a SQL statement, for example you could use it in another Execute SQL Task which would use the second Data Connection like so:
"INSERT INTO Table2
SELECT *
FROM Table1
WHERE DateOf > '" + #[User::max_date] + "'"
Or in an OLE DB Source in a data flow like so:
"SELECT *
FROM Table1
WHERE DateOf > '" + #[User::max_date] + "'"
You can do this in a single SQL Task if you want:
Insert into Table2
Select *
From Table1
Where DateOf > (Select MAX(Dateof) from Table2)
If you want to use multiple Execute SQL Task items in the control flow, or want to make use of the parameter in a data flow instead, you have to change the General > Result Set option for your MAX() query to Single Row, then move from General to Result Set and Add a new variable for your result set to occupy.
To use that variable in your INSERT INTO.... query via Execute SQL Task, you'll construct your query with a ? for each parameter and map them in the parameter mapping section. If a variable is used multiple times in a query it's easiest to use a stored procedure, so you can simply pass the relevant parameters in SSIS.
We have one database existing on server A. Server A also hosts our program code which will be calling the SQL statement.
We have another database VMIntranetTest existing on server B VMC-MMS
Server A and server B have different logon user credentials. Server A and server B both exist on our internal network.
Using PHP, I have the following SQL statement defined.
$strSql = 'INSERT INTO VMC-MMS.VMIntranetTest.dbo.TestTable (FirstName, LastName, Age) ' .
'SELECT FNAME, LNAME, AGE ' .
'FROM BSLIB.SQLTSTF ';
FROM -> BSLIB.SQLTSTF <- is on our local server (A), so my connection string used to execute the statement will have the user credentials to connect to server A.
INSERT INTO -> VMC-MMS.VMIntranetTest.dbo.TestTable <- is the different server.database.dbo.table (Server B).
How do I specify the user credentials to be used for the INSERT INTO portion of the statement? The secondary portion containing the SELECT FROM statement should already be covered by my initial connection string.
Thank you,
Edit 1 in regards to Paul's answer.
I've attempted to use the OPENROWSET as mentioned, and have the following SQL statement.
INSERT INTO VMIntranetTest.TestTable (FirstName, LastName, Age)
OPENROWSET('vmas400',
'Server=192.168.1.2;Trusted_Connection=yes;user_id=INTRAIS;password=****',
'SELECT FNAME, LNAME, AGE FROM BSLIB.SQLTSTF' ) as a
As you can see, I changed things around a little bit. My connection string through the code opens the connection to Server B "VMC-MMS". My SQL statement "select" portion, uses the OPENROWSET to open a connection to Server A "192.168.1.2".
However, I am getting this error message:
SQLSTATE[HY000]: General error: 1 near "OPENROWSET": syntax error
Edit 2 i needed to put the entire OPENROWSET portion inside a VALUES ( ) clause. Now I'm getting a message:
SQLSTATE[HY000]: General error: 1 no such table: VMIntranetTest.TestTable
Edit 3
I've now got the following SQL
INSERT INTO VMIntranetTest.TestTable (FirstName, LastName, Age)
select a.FNAME, a.LNAME, a.AGE FROM
OPENROWSET('vmas400',
'Server=192.168.1.2;Trusted_Connection=yes;user_id=INTRAIS;password=****',
'SELECT FNAME, LNAME, AGE FROM BSLIB.SQLTSTF' ) as a
And am getting this error:
SQLSTATE[HY000]: General error: 1 near "(": syntax error
You just need the SELECT statement in there. Try this:
INSERT INTO VMIntranetTest.dbo.TestTable (a.FirstName, a.LastName, a.Age)
SELECT a.FirstName, a.LastName, a.Age FROM
OPENROWSET('vmas400',
'Server=192.168.1.2;Trusted_Connection=yes;user_id=INTRAIS;password=****',
'SELECT FNAME, LNAME, AGE FROM BSLIB.dbo.SQLTSTF' ) as a
EDIT: Try the query now. You had the table specified from the database without specifying the schema. If you have appropriate permissions, the above query will now work. Otherwise, you will need to specify the schema. Sorry for not catching that!
You could use OPENROWSET to select data from the different database
http://msdn.microsoft.com/en-us/library/ms190312.aspx