OK this seems like it should be insanely easy, but I cannot figure it out. Every where I look online says to create temp tables and VB scripts and I cannot believe I have to do that. My goal is to insert all the records in a table with a date later than the max date in that destination table.
UPDATE The 2 tables are in two different non linked SQL databases
So:
Select #[User::Dated] = MAX(Dateof) from Table2
Insert into Table2
Select *
From Table1
Where DateOf > #[User::Dated]
I am trying to do this in SSIS. I declared a variable, the SQL execution step looks like it is assigning the single row output to it. But when I got go into the data flow it give me no parameters to choose, when I force the known parameter which is in the project scope it says no parameter exists
Create two OLE DB data sources each pointing at you two databases.
Create a variable called max_date and make its data type String.
Place an Execute SQL Task on the Control Flow, change its connection type to OLE DB and for the connection select the name of the data source that contains Table2. Set the ResultSet to Single Row. Add the following for the SQLStatement:
SELECT CAST(MAX(Dateof) AS VARCHAR) AS max_date FROM Table2
Go to the Result Set pane, click Add and enter the following:
Result Name: max_date
Variable Name: User::max_date
You can now use the max_date variable in an expression to create a SQL statement, for example you could use it in another Execute SQL Task which would use the second Data Connection like so:
"INSERT INTO Table2
SELECT *
FROM Table1
WHERE DateOf > '" + #[User::max_date] + "'"
Or in an OLE DB Source in a data flow like so:
"SELECT *
FROM Table1
WHERE DateOf > '" + #[User::max_date] + "'"
You can do this in a single SQL Task if you want:
Insert into Table2
Select *
From Table1
Where DateOf > (Select MAX(Dateof) from Table2)
If you want to use multiple Execute SQL Task items in the control flow, or want to make use of the parameter in a data flow instead, you have to change the General > Result Set option for your MAX() query to Single Row, then move from General to Result Set and Add a new variable for your result set to occupy.
To use that variable in your INSERT INTO.... query via Execute SQL Task, you'll construct your query with a ? for each parameter and map them in the parameter mapping section. If a variable is used multiple times in a query it's easiest to use a stored procedure, so you can simply pass the relevant parameters in SSIS.
Related
Pretty new to BI and SQL in general, but a few months ago I didn't even know what a model is and now here I am...trying to build a package that runs daily.
Currently running this is Excel via PowerQuery but because the data is so much, I have to manually change the query every month. Decided to move it into SSIS.
Required outcome: Pull the last date in my Database and use it as a variable in the model (as I have millions of rows, I only want to load lines with dates greater than what I have in my table already).
Here is my Execute SQL Task:
I set up a variable for the SQL query
and trying to use it in my OLE DB query like this
Execute SQL Task: results, are fine - returns date as "dd/mm/yyyy hh24:mi:ss"
SELECT MAX (CONVACCT_CREATE_DATE) AS Expr1 FROM GOMSDailySales
Variable for OLE DB SQL Query:
"SELECT fin_booking_code, FIN_DEPT_CODE, FIN_ACCT_NO, FIN_PROD_CODE, FIN_PROG_CODE, FIN_OPEN_CODE, DEBIT_AMT, CREDIT_AMT, CURRENCY_CODE, PART_NO, FIN_DOC_NO, CREATE_DATE
FROM cuown.converted_accounts
WHERE (CREATE_DATE > TO_DATE(#[User::GetMaxDate],'yyyy/mm/dd hh24:mi:ss'))
AND (FIN_ACCT_NO LIKE '1%')"
Currently getting missing expression error, if I add " ' " to my #[User::GetMaxDate], I get a year must be between 0 and xxxx error.
What am I doing wrong / is there a cleaner way to get this done?
In the OLEDB source use the following, change the data access mode to SQL command, and use the following command:
SELECT fin_booking_code, FIN_DEPT_CODE, FIN_ACCT_NO, FIN_PROD_CODE, FIN_PROG_CODE, FIN_OPEN_CODE, DEBIT_AMT, CREDIT_AMT, CURRENCY_CODE, PART_NO, FIN_DOC_NO, CREATE_DATE
FROM cuown.converted_accounts
WHERE (CREATE_DATE > TO_DATE(?,'yyyy/mm/dd hh24:mi:ss'))
AND (FIN_ACCT_NO LIKE '1%')
And click on the parameters button and map #[User::GetMaxDate] to the first parameter.
For more information, check the following answer: Parameterized OLEDB source query
Alternative method
If parameters are not supported in the OLE DB provider you are using, create a variable of type string and evaluate this variable as the following expression:
"SELECT fin_booking_code, FIN_DEPT_CODE, FIN_ACCT_NO, FIN_PROD_CODE, FIN_PROG_CODE, FIN_OPEN_CODE, DEBIT_AMT, CREDIT_AMT, CURRENCY_CODE, PART_NO, FIN_DOC_NO, CREATE_DATE
FROM cuown.converted_accounts
WHERE CREATE_DATE > TO_DATE('" + (DT_WSTR, 50)#[User::GetMaxDate] +
"' ,'yyyy/mm/dd hh24:mi:ss') AND FIN_ACCT_NO LIKE '1%'"
Then from the OLE DB source, change the data access mode the SQL Command from variable and select the string variable you created.
Your trying to use the SSIS variable like a variable in the query. When constructing a SQL query in a string variable you simply need to concatenate the strings together. The expression for your query string variable should look like this.
"SELECT fin_booking_code, FIN_DEPT_CODE, FIN_ACCT_NO, FIN_PROD_CODE, FIN_PROG_CODE, FIN_OPEN_CODE, DEBIT_AMT, CREDIT_AMT, CURRENCY_CODE, PART_NO, FIN_DOC_NO, CREATE_DATE
FROM cuown.converted_accounts
WHERE CREATE_DATE > " + #[User::GetMaxDate] +
"AND (FIN_ACCT_NO LIKE '1%')"
I have ssis package in that I'm taking values from flat file and insert it into table.
I have taken one Execute SQL Task in that creating one temptable
CREATE TABLE [tempdb].dbo.##temptable
(
date datetime,
companyname nvarchar(50),
price decimal(10,0),
PortfolioId int,
stype nvarchar(50)
)
Insert into [tempdb].dbo.##temptable (date,companyname,price,PortfolioId,stype)
SELECT date,companyname,price,PortfolioId,stype
FROM ProgressNAV
WHERE (Date = '2011-09-30') AND (PortfolioId = 5) AND (stype in ('Index'))
ORDER BY CompanyName
Now in above query I need to pass (Date = '2011-09-30') AND (PortfolioId = 5) AND (stype in ('Index'))
these 3 parameter using variable name I have created variables in package so that I become dynamic.
In your Execute SQL Task, make sure SQLSourceType is set to Direct Input, then your SQL Statement is the name of the stored proc, with questionmarks for each paramter of the proc, like so:
Click the parameter mapping in the left column and add each paramter from your stored proc and map it to your SSIS variable:
Now when this task runs it will pass the SSIS variables to the stored proc.
The EXCEL and OLED DB connection managers use the parameter names 0 and 1.
I was using a oledb connection and wasted couple of hours trying to figure out the reason why the query was not working or taking the parameters. the above explanation helped a lot
Thanks a lot.
Along with #PaulStock's answer, Depending on your connection type, your variable names and SQLStatement/SQLStatementSource Changes
https://learn.microsoft.com/en-us/sql/integration-services/control-flow/execute-sql-task
SELECT, INSERT, UPDATE, and DELETE commands frequently include WHERE clauses to specify filters that define the conditions each row in the source tables must meet to qualify for an SQL command. Parameters provide the filter values in the WHERE clauses.
You can use parameter markers to dynamically provide parameter values. The rules for which parameter markers and parameter names can be used in the SQL statement depend on the type of connection manager that the Execute SQL uses.
The following table lists examples of the SELECT command by connection manager type. The INSERT, UPDATE, and DELETE statements are similar. The examples use SELECT to return products from the Product table in AdventureWorks2012 that have a ProductID greater than and less than the values specified by two parameters.
EXCEL, ODBC, and OLEDB
SELECT* FROM Production.Product WHERE ProductId > ? AND ProductID < ?
ADO
SELECT * FROM Production.Product WHERE ProductId > ? AND ProductID < ?
ADO.NET
SELECT* FROM Production.Product WHERE ProductId > #parmMinProductID
AND ProductID < #parmMaxProductID
The examples would require parameters that have the following names:
The EXCEL and OLED DB connection managers use the parameter names 0 and 1. The ODBC connection type uses 1 and 2.
The ADO connection type could use any two parameter names, such as Param1 and Param2, but the parameters must be mapped by their ordinal position in the parameter list.
The ADO.NET connection type uses the parameter names #parmMinProductID and #parmMaxProductID.
A little late to the party, but this is how I did it for an insert:
DECLARE #ManagerID AS Varchar (25) = 'NA'
DECLARE #ManagerEmail AS Varchar (50) = 'NA'
Declare #RecordCount AS int = 0
SET #ManagerID = ?
SET #ManagerEmail = ?
SET #RecordCount = ?
INSERT INTO...
I have a view in SQL Server that detects all the Products Ids from a table that does not have a Product Description.
In vertica we have all the Product IDs with their description.
My idea is to build a SSIS package that can pull all the Products Ids with missing description and use it as a where clause in the Vertica query.
I already tried with user variables but I failed. I am beginner on SSIS
This is what I am looking for.
SELECT [Product ID], [Product Desc]
FROM VerticaTable
WHERE [Product ID] IN (#Variable?)
The variable should like like this.
SELECT DISTINCT [Product ID] FROM SQLSerrverViewThatHasMissingDesc
Basically you have two ways to do that.
First of all, I would like to let you know the SELECT in the SSIS package will not show you anything, but you can make the changes to the query, such as insert the results into a table.
Create a Object type variable1 and use Foreach Loop Container to iterate that variable1, assign each of the value to another variable2 (created in advance, maybe it is a String type variable). Then in the container, put you Execute SQL task with the query:
SELECT [Product ID], [Product Desc]
FROM VerticaTable
WHERE [Product ID] = ?
Map variable1 as Index 0 in the parameter mapping page. (if you are using OLE DB as Connection type)
The drawback using this method is you cannot view all the productID at the same time because this is a loop process,unless you would like to load each of the result to a table.
If you prefer to see all the candidate results at the same time, the most common way is using Dynamic SQL because variables are not supported anywhere in your query.
So, the same query as you wrote, but need some add-ups.
1) First, Create three variables, variable1, variable2, variable3, variable1 is a Object type and variable2 and variable3 are String type.
2) Next, you need a Foreach Loop Container to build up your variable. In the Foreach Loop Container, drag and drop Script Task. Before the container, connected by a Execute SQL task, and after the container, connecting another Execute SQL task
So physical Order: Execute SQL task 1 -> Foreach Loop Container, inside: Script task -> Execute SQL task 2
3) Open the Execute SQL Task 1, put SELECT DISTINCT [Product ID] FROM SQLSerrverViewThatHasMissingDesc in the query, and go to the Result set page, assign Variable1
4) Move on to the Foreach loop Container, choose Foreach ADO Enumerator, then choose Variable1 as ADO object source variable, mode is Rows in the first table, then go to Variable mappings page, choose Variable2, and default the index.
5) Go to the Script task, open it, choose Variable as ReadOnlyVariable, in the ReadWrite variable, choose Variable3, edit the script, paste the follow code inside the Main()
if (Dts.Variables["User::Variable3"].Value.ToString() == "")
{
Dts.Variables["User::Variable3"].Value = Dts.Variables["User::Variable2"].Value.ToString();
}
else
{
Dts.Variables["User::Variable3"].Value = Dts.Variables["User::Variable2"].Value.ToString() + "," + Dts.Variables["User::Variable2"].Value.ToString();
}
Dts.TaskResult = (int)ScriptResults.Success;
6) After above step, ideally, your Variable3 will have a long string separated by ,, like A,B,C,D,E. The reason you want to get this string is because this Variable3 will be used later in the Execute SQL task as part of the Dynamic SQL
7) Here comes the final step, in the Execute SQL Task 2, use following query:
DECLARE #SQL VARCHAR(MAX)
SET #SQL = 'SELECT [Product ID], [Product Desc]
FROM VerticaTable
WHERE [Product ID] IN (''' + ? + ''')'
In the parameter mapping, choose Vareiable3, index as 0.
8) Execute the package.
But it will not show you anything, but if you try to simulate the dynamic process in SSMS, you will see the result.
LONG's answer is a good one. I am going to add a little tweak to get your IN clause. Please accept his answer and if you find this helpful just upvote.
Use a Execute SQL task to get the IN clause directly using a stuff function in SQL Server and result set to a single line directly into a string variable.
The coalesce just handles null (no result) and sets variable to empty string.
declare #t table (ProdID int)
insert into #t
values(1),(1),(2),(3)
select coalesce(
stuff(
(select ','+cast(ProdID as varchar(5))
from #t
group by ProdID
for XML path(''))
,1,1,'')
,'') as delimProdIds
Results:
delimProdIds
1,2,3
Simply change #t to your view
In the control flow (After this exec SQL step) put an expression that the variable !='' (Empty String) If it does, it will end the process right here.
Long's script to create the variable has a slight error:
Dts.Variables["User::Variable3"].Value =
Dts.Variables["User::Variable2"].Value.ToString() + "," +
Dts.Variables["User::Variable2"].Value.ToString();
When Variable3 has values, they are overlaid with Variable2 every time. It should probably be:
Dts.Variables["User::Variable3"].Value =
Dts.Variables["User::Variable3"].Value.ToString() + "," +
Dts.Variables["User::Variable2"].Value.ToString();
I am trying to create a global temp table using the results from one query, which can then be selected as a table and manipulated further several times without having to reprocess the data over and over.
This works perfectly in SQL management studio, but when I try to add the table through an Excel query, the table can be referenced at that time, but it is not created in Temporary Tables in the tempdb database.
I have broken it down into a simple example.
If I run this in SQL management studio, the result of 1 is returned as expected, and the table ##testtable1 is created in Temporary Tables
set nocount on;
select 1 as 'Val1', 2 as 'Val2' into ##testtable1
select Val1 from ##testtable1
I can then run another select on this table, even in a different session, as you'd expect. E.g.
Select Val2 from ##testtable1
If I don't drop ##testtable1, running the below in a query in Excel returns the result of 2 as you'd expect.
Select Val2 from ##testtable1
However, if I run the same Select... into ##testtable1 query directly in Excel, that correctly returns the result of 1, but the temptable is not created.
If I then try to run
Select Val2 from ##testtable1
As a separate query, it errors saying "Invalid object name '##testtable1'
The table is not listed within Temporary Tables in SQL management studio.
It is as if it is performing a drop on the table after the query has finished executing, even though I am not calling a drop.
How can I resolve this?
Read up on global temp tables(GTT). They persist as long as there is a session referencing it. In SSMS, if you close the session that created the GTT prior to using it in another session, the GTT would be discarded. This is what is happening in Excel. Excel creates a connection, executes and disconnects. Since there are no sessions using the GTT when Excel disconnects, the GTT is discarded.
I would highly recommend you create a normal table rather than use a GTT. Because of their temporary nature and dependence on an active session, you may get inconsistent results when using a GTT. If you create a normal table instead, you can be certain it will still exist when you try to use it later.
The code to create/clean the table is pretty simple.
IF OBJECT_ID('db.schema.tablename') IS NOT NULL
TRUNCATE TABLE [tablename]
ELSE
CREATE [tablename]...
GO
You can change the truncate to a delete to clean up a specific set of data and place it at the start of each one of your queries.
is it possible you could use a view? assuming that you are connecting to 5 DBs on the same server can you union the data together in a view:
CREATE VIEW [dbo].[testView]
AS
SELECT *
FROM database1.dbo.myTable
UNION
SELECT *
FROM database2.dbo.myTable
Then in excel:
Data> New Query > From Database > FromSQL Server Database
enter DB server
Select the view from the appropriate DB - done :)
OR call the view however you are doing it (e.g. vba etc.)
equally you could use a stored procedure and call that from VBA .. basically anything that moves more of the complexity to the server side to make your life easier :D
You can absolutely do this. Notice how I'm building a temp table from SQL called 'TmpSql' ...this could be any query you want. Then I set it to recordset 1. Then I create another recordset 2, that goes and gets the temp table data.
Imagine if you were looping on the first cn.Execute where TmpSql is changing.. This allows you to build a Temporary table coming from many sources or changing variables. This is a powerful solution.
cn.open "Provider= ..."
sql = "Select t.* Into #TTable From (" & TmpSql & ") t "
Set rs1 = cn.Execute(sql)
GetTmp = "Select * From #TTable"
rs2.Open GetTmp, cn, adOpenDynamic, adLockBatchOptimistic
If Not rs2.EOF Then Call Sheets("Data").Range("A2").CopyFromRecordset(rs2)
rs2.Close
rs1.Close
cn.Close
I need to create a "ghost" table in SQL Server, which doesn't actually exist but is a result set of a SQL Query. Pseudo code is below:
SELECT genTbl_col1, genTblcol2
FROM genTbl;
However, "genTbl" is actually:
SELECT table1.col AS genTbl_col1,
table2.col AS genTbl_col2
FROM table1 INNER JOIN table2 ON (...)
In other words, I need that every time a query is run on the server trying to select from "genTbl", it simply creates a result set from the query and treats it like a real table.
The situation is that I have a software that runs queries on a database. I need to modify it, but I cannot change the software itself, so I need to trick it into thinking it can actually query "genTbl", when it actually doesn't exist but is simply a query of other tables.
To clarify, the query would have to be a sort of procedure, available by default in the database (i.e. every time there is a query for "genTbl").
Use #TMP
SELECT genTbl_col1, genTblcol2
INTO #TMP FROM genTbl;
It exists only in current session. You can also use ##TMP for all sessions.