I am creating an SSIS package to transfer tables between databases on different servers. The timestamp on the source database is in UTC and I would like to convert it to my local time using [CallConnectedTime] AT TIME ZONE 'UTC' AT TIME ZONE 'Pacific Standard Time' AS CallConnectedTime.
How is it possible to do this transformation in the Transfer SQL Objects Task? I thought about using a Data Flow Task but then I need to create one for each table I am bringing over.
The Transfer SQL Server Objects Task does not allow you to define any data transformations.
If you don't want to go the route of creating multiple Data Flow Tasks, you could create an ADO recordset of the table names after the Transfer task runs, then create a Foreach Loop to iterate over the ADO recordset. To do this you might use the following query in an Execute SQL Task, with ResultSet set to Full result set:
SELECT Name
FROM sys.tables
WHERE type_desc = 'USER_TABLE'
Map the result set to an Object variable type. Then you can create the Foreach Loop with an ADO enumerator, point it at your Object variable, and then create another variable to hold the value of the Name field.
Using the variable that now holds the Name field, create another Execute SQL Task inside your Foreach Loop. This Execute SQL task will build and execute a dynamic SQL statement that will UPDATE the table by setting the CallConnectedTime.
The Execute SQL Task inside your loop would then look something like this:
DECLARE #query NVARCHAR(MAX)
DECLARE #Table VARCHAR(1000) = ?
SET #query = N'
UPDATE ' + #Table + N'
SET CallConnectedTime = CallConnectedTime AT TIME ZONE ''UTC'' AT TIME ZONE ''Pacific Standard Time'''
EXEC(#query)
And under Parameter Mapping on the left pane of the editor, you'd add your table name variable as Parameter Name 0 like this:
Your Control Flow should then look something like this when you're done with the above steps (ignore the error icons).
Related
How to use an sql with a cursor inside to create an SSIS package? As per the requirement we can't use a stored procedure call inside ssis. My sqlquery is to select data from multiple table using union and update certain fields according to the fetched data using cursor 'for' loop.
How to avoid a cursor in SSIS using SSIS tools:
Execute your cursor query in execute SQL and save result into a variable of type object.
Add a foreach and enumerate on ADO object. Set enumeration on the variable from #1.
Set a variable (probably string) to each loop.
Use that variable as you would in your original cursor. (probably done with a function to create the SQL and an execute SQL task.
As part of a SSIS package I have a SQL table containing the staging table name and corresponding 'real' table name. The staging table names will change based on the date but there is a previous step that works out what the Real_Table is.
How do I loop through each one in SQL and insert all the data from the staging tables (columns are identical in both) into the real table and update the flag from 0 to 1 to mark it was done. This is my table:
Staging_Table Real_Table UpdateFlag
Customers_01012018 Customers 0
Order_01012018 Order 0
Suppliers_02022018 Suppliers 0
You can use while loop to load the data into Real tables,
DECLARE #total INT
DECLARE #start INT
DECLARE #query NVARCHAR(MAX)
DECLARE #staging_table NVARCHAR(MAX)
DECLARE #real_table NVARCHAR(MAX)
SET #start = 1
SET #total = (SELECT COUNT(*) FROM Stg_tables)
WHILE(#start <= #total)
BEGIN
SELECT TOP 1 #staging_table = Staging_Table, #real_table = Real_Table FROM Stg_tables WHERE UpdateFlag = 0
SET #query = 'INSERT INTO ' + #real_table + ' SELECT * FROM ' + #staging_table
EXEC(#query)
UPDATE Stg_tables SET UpdateFlag = 1 WHERE Staging_Table = #staging_table AND Real_Table = #real_table
SET #start = #start + 1
END
An overview of how to go about this is below. Of course make sure to match the correct configurations to your environment and set the metadata appropriately where applicable.
Create an Execute SQL Task that selects the staging and real table
names. Choose the "Full result set" ResultSet. On the Result Set
page, add an object variable and set the Result Name to 0 to use the
immediate results.
Add a Foreach Loop that is of the Foreach ADO Enumerator type. Use the object variable from the Execute SQL Task as the ADO Object Source Variable. On the Variable Mappings page, add a string variable at index 1 and 0. One of these will hold the staging table name and the other will hold the real table name. These will align with the order they were selected in the Execute SQL Task, so if you selected the staging table first there use a the variable that will hold this name at index 0.
Create another string variable that contains an expression selecting the necessary columns that will be loaded from the staging table with the variable holding this table name concatenated within in. An Example of this follows with the #[User::StagingTableVariable] variable representing the variable holding the staging table name.
Within the Foreach Loop, add Data Flow Task then add an OLE DB Source inside this. This will use the variable with the SQL selecting from the staging table, but to correctly set the metadata choose the SQL Command option and use a SQL statement that selects the same columns from an existing staging table name. Once this is set, change the ValidateExternalMetadata property to false, choose "SQL command from variable" for the Data Access Mode, and pick the variable holding the SQL statement that uses the staging table name.
Add an OLE DB Destination and connect the source to this. Like before, use an existing real table and map the columns. After this, again set ValidateExternalMetadata to false, change Data Access Mode to the use a table or view name from variable (I would recommend the fast load option), and add the variable holding the real table name.
After the Data Flow Task add another Execute SQL Task that's linked to the DFT. Create a string variable with an update statement for the mapping table where the table names originate from. Set the SQLSourceType value to variable and select this variable for the SourceVariable property. If you're using a text column in the WHERE clause of the update statement, make sure that the expression contains single quotes (') as a typical SQL update statement would.
Example OLE DB Source Variable Expression:
"SELECT ColA, ColB, ColC from YourSchema." + #[User::StagingTableVariable]
Example Variable Expression Update Command:
"UPDATE YourSchema.MappingTable SET UpdateFlag = 1 where Real_Table = '" + #[User::RealTableVariable] + "'"
I'm creating some views with a lot of references to tables in another database.
At some point the other database needs to change.
I want to make it easy for the next developer to change the scripts to use another database.
This obviously work like it should:
CREATE VIEW ViewName
AS
SELECT *
FROM AnotherDatabase.SchemaName.TableName;
But when I do:
DECLARE #DB CHAR(100)
SET #DB = 'AnotherDatabase'
GO
CREATE VIEW ViewName
AS
SELECT *
FROM #DB.SchemaName.TableName;
I get the error:
Msg 137, Level 15, State 2, Procedure ViewName, Line 3
Must declare the scalar variable "#DB".
I could do something like:
DECLARE #SQL ...
SET #SQL = ' ... FROM ' + #DB + ' ... '
EXEC (#SQL)
But that goes against the purpose of making it easier for the next developer - because this dynamic SQL approach removed the formatting in SSMS.
So my question is: how do I make it easy for the next developer to maintain T-SQL code where he needs to swap out the database reference?
Notes:
I'm using SQL Server 2008 R2
The other database is on the same server.
Consider using SQLCMD variables. This will allow you to specify the actual database name at deployment time. SQL Server tools (SSMS, SQLCMD, SSDT) will replace the SQLCMD variable names with the assigned string values when the script is run. SQLCMD mode can be turned on for the current query windows from the menu option Query-->SQLCMD mode option.
:SETVAR OtherDatabaseName "AnotherDatabaseName"
CREATE VIEW ViewName AS
SELECT *
FROM $(OtherDatabaseName).SchemaName.TableName;
GO
This approach works best when SQL objects are kept under source control.
When you declare variables, they only live during the execution of the statement. You can not have a variable as part of your DDL. You could create a bunch of synonyms, but I consider that over doing it a bit.
The idea that your database names are going to change over time seems a bit out of the ordinary and conceivably one-time events. However, if you do still require to have the ability to quickly change over to point to a new database, you could consider creating a light utility directly in SQL to automatically generate the views to point to the new database.
An implementation may look something like this.
Assumptions
Assuming we have the below databases.
Assuming that you prefer to have the utility in SQL instead of building an application to manage it.
Code:
create database This;
create database That;
go
Configuration
Here I'm setting up some configuration tables. They will do two simple things:
Allow you to indicate the target database name for a particular configuration.
Allow you to define the DDL of the view. The idea is similar to Dan Guzman's idea, where the DDL is dynamically resolved using variables. However, this approach does not use the native SQLCMD mode and instead relies on dynamic SQL.
Here are the configuration tables.
use This;
create table dbo.SomeToolConfig (
ConfigId int identity(1, 1) primary key clustered,
TargetDatabaseName varchar(128) not null);
create table dbo.SomeToolConfigView (
ConfigId int not null
references SomeToolConfig(ConfigId),
ViewName varchar(128) not null,
Sql varchar(max) not null,
unique(ConfigId, ViewName));
Setting the Configuration
Next you set the configuration. In this case I'm setting the TargetDatabaseName to be That. The SQL that is being inserted into SomeToolConfigView is the DDL for the view. I'm using two variables, one {{ViewName}} and {{TargetDatabaseName}}. These variables are replaced with the configuration values.
insert SomeToolConfig (TargetDatabaseName)
values ('That');
insert SomeToolConfigView (ConfigId, ViewName, Sql)
values
(scope_identity(), 'dbo.my_objects', '
create view {{ViewName}}
as
select *
from {{TargetDatabaseName}}.sys.objects;'),
(scope_identity(), 'dbo.my_columns', '
create view {{ViewName}}
as
select *
from {{TargetDatabaseName}}.sys.columns;');
go
The tool
The tool is a stored procedure that takes a configuration identifier. Then based on that identifier if drops and recreates the views in the configuration.
The signature for the stored procedure may look something like this:
exec SomeTool #ConfigId;
Sorry -- I left out the implementation, because I have to scoot, but figured I would respond sooner than later.
Hope this helps.
I using ssis package.I want insert flat file source (text file) to sql.
Addres of text file is dynamic so i define variable for path.I have the sp like this:
CREATE PROCEDURE [dbo].[Insert_FileMaster]
#FILE_PATH nVARCHAR(MAX)
,#id int OUTPUT
AS
BEGIN
insert into [dbo].[FileMaster] ([FM_Name])
values(#FILE_PATH))
set #id = ##IDENTITY
END
I want exec this sp With variable parameter.
this is my package:
Which ssis tool should i use?and how to get output from sp (return parametert must be use in another sp in package)?
You will want to add an Execute SQL Task before your Data Flow Task (this will be at the Control Flow level).
You will need to configure the Execute SQL task as described in this answer.
Insert a single row and return its primary key
How do I set the database name dynamically in a SQL Server stored procedure?
Sometimes, the use of SYNONYMs is a good strategy:
CREATE SYNONYM [schema.]name FOR [[[linkedserver.]database.]schema.]name
Then, refer to the object by its synonym in your stored procedure.
Altering where the synonym points IS a matter of dynamic SQL, but then your main stored procedures can be totally dynamic SQL-free. Create a table to manage all the objects you need to reference, and a stored procedure that switches all the desired synonyms to the right context.
This functionality is only available in SQL Server 2005 and up.
This method will NOT be suitable for frequent switching or for situations where different connections need to use different databases. I use it for a database that occasionally moves around between servers (it can run in the prod database or on the replication database and they have different names). After restoring the database to its new home, I run my switcheroo SP on it and everything is working in about 8 seconds.
Stored Procedures are database specific. If you want to access data from another database dynamically, you are going to have to create dynamic SQL and execute it.
Declare #strSQL VarChar (MAX)
Declare #DatabaseNameParameter VarChar (100) = 'MyOtherDB'
SET #strSQL = 'SELECT * FROM ' + #DatabaseNameParameter + '.Schema.TableName'
You can use if clauses to set the #DatabaseNameParameter to the DB of your liking.
Execute the statement to get your results.
This is not dynamic SQL and works for stored procs
Declare #ThreePartName varchar (1000)
Declare #DatabaseNameParameter varchar (100)
SET #DatabaseNameParameter = 'MyOtherDB'
SET #ThreePartName = #DatabaseNameParameter + '.Schema.MyOtherSP'
EXEC #ThreePartName #p1, #p2... --Look! No brackets